Well, if your goal is trying out for education, but on Windows, you could start with DrRacket. http://racket-lang.org/
It is a reasonable IDE, it has some GUI libraries included, open-source, cross-platform, works fine on Windows.
Racket is based on Scheme language (which is a part of Lisp language family). It has a mode for Scheme as described in R6RS or R5RS standard, and it has a few not-fully-compatible dialects.
I use Common Lisp, but not under Windows. Common Lisp has more cross-implementation libraries, it could be useful sometimes. Probably, EQL is the easiest to set up under Windows (it is ECL, a Common Lisp implementation, merged with Qt for GUI; I remember there being a bundled download). Maybe CommonQt or Cells-GTK would work. I remember that some of the Common Lisp package management systems have significant problems under Windows or require either Cygwin or MSys (so they can use tar, gzip, mkdir etc. as if they were on a Unix-like system)
My goals are: 1) to get the "Lisp experience" with minimum overhead; and 2) to use the best available tools.
And I hope these two goals are not completely contradictory. I want to be able to write my own application on my computer conveniently after a few minutes, and to fluently progress to more complex applications. On the other hand, if I happen to later decide that Lisp is not for me, I want to be sure it was not only because I chose the wrong tools.
Thanks for all the answers! I will probably start with Racket.
If it's worth saying, but not worth its own post, even in Discussion, it goes here.