if there's a short answer, what happened to Lucas between the end of the first trilogy and the beginning of the second?

The short answer is that A New Hope had nothing whatsoever to do with any grand story about Darth Vader, who was merely a mid-level flunky of the Empire. Empire Strikes Back and Return of the Jedi frantically retcon this, but Lucas simply didn't have a decent backstory centered around Darth Vader and couldn't come up with one.

The longer answer goes into the above, and also points out that Lucas's support network (of beta readers, if you will) which edited and improved his trilogy had largely fragmented or vanished by this point. For example, his wife, a very skilled film editor, had divorced him by the time of the prequels.