It's taken for granted that bad guys want to rule worlds, but why? Wouldn't it be a lot of work?

Seems Quirrell has a good motive for conquest that you didn't mention, or at least a motive for attempting the unification of wizards, whether under his leadership or not.

"Those fool Muggles will kill us all someday!" Professor Quirrell's voice had grown louder. "They will end it! End all of it!"

Harry was feeling a bit lost here. "What are we talking about here, nuclear weapons?"

"Yes, nuclear weapons!"

And remember his magical fascism speech, where he said to imagine an outside enemy attacking the divided armies?

It could be that Quirrell wanted a believable fake secret motive for his interest in Harry, and all of the above was so that Harry wouldn't question this:

"Sso," Harry hissed, "what iss your plan for me, precissely?"

"You ssaid no time," came the snake's hiss, "but plan iss for you to rule country, obvioussly, even your young noble friend hass undersstood that by now, assk him on return if you wissh.

Or, his desire to unite wizards against the muggle threat could be sincere.

EDIT: A proposal of this theory by 75th, and a little bit of discussion, here: http://lesswrong.com/lw/ams/harry_potter_and_the_methods_of_rationality/60di