The only one of the books you mention that I've read is Wrinkle in Time, so I'll address that one. It isn't world-driven! It's a strongly character-driven story. The planets she invents, the species she imagines, the settings she dreams up - these do not supply the thrust of the story. The people populating the book do that, and pretty, emotionally-charged prose does most of the rest. Further, L'Engle's worldbuilding isn't awful, and moreover, its weaknesses aren't distracting. It has an element of whimsy to it and it's colored by her background values, but there's nothing much in there that is outrageous and important and unexplained.
Eliezer's stories, meanwhile - I'd have to dislike them even more if I were interpreting them as being character-driven. His characters tend to be ciphers with flat voices, clothed in cliché and propped up by premise. And it's often okay to populate your stories with such characters if they aren't the point - if the point is world or premise/conceit or plot or even just raw beautiful writing. I actually think that Eliezer's fiction tends to be premise/conceit driven, not setting driven, but he backs up his premises with setting, and his settings do not appear to be up to the task. So to summarize:
A bad story element (such as setting, characterization, plot, or writing quality) may be forgivable, and not preclude the work it's found in from being good, if:
Eliezer's subpar worldbuilding slips by according to the first criterion. I don't think his stories are truly setting-driven. But it fails the second two. His settings are indispensably necessary to back up his premises. ("Three Worlds Collide" could not have been plausibly set during some encounter between three boats full of humans on Earth.) And - this one is a matter of taste to some extent, I'll grant - the settings are poor enough to be distracting. (The non-consensual sex thing is just a particularly easy target. It's hardly the only bizarre, unexplained thing he's ever dropped in.)
As promised, here is the "Q" part of the Less Wrong Video Q&A with Eliezer Yudkowsky.
The Rules
1) One question per comment (to allow voting to carry more information about people's preferences).
2) Try to be as clear and concise as possible. If your question can't be condensed to a few paragraphs, you should probably ask in a separate post. Make sure you have an actual question somewhere in there (you can bold it to make it easier to scan).
3) Eliezer hasn't been subpoenaed. He will simply ignore the questions he doesn't want to answer, even if they somehow received 3^^^3 votes.
4) If you reference certain things that are online in your question, provide a link.
5) This thread will be open to questions and votes for at least 7 days. After that, it is up to Eliezer to decide when the best time to film his answers will be. [Update: Today, November 18, marks the 7th day since this thread was posted. If you haven't already done so, now would be a good time to review the questions and vote for your favorites.]
Suggestions
Don't limit yourself to things that have been mentioned on OB/LW. I expect that this will be the majority of questions, but you shouldn't feel limited to these topics. I've always found that a wide variety of topics makes a Q&A more interesting. If you're uncertain, ask anyway and let the voting sort out the wheat from the chaff.
It's okay to attempt humor (but good luck, it's a tough crowd).
If a discussion breaks out about a question (f.ex. to ask for clarifications) and the original poster decides to modify the question, the top level comment should be updated with the modified question (make it easy to find your question, don't have the latest version buried in a long thread).
Update: Eliezer's video answers to 30 questions from this thread can be found here.