The most terrifying part of the experience for me is the idea that Eliezer could have such a strongly different reaction to the story; it made me less confident that something like CEV will converge nicely.
Eliezer having a different response to a fanfic you both ended up reading and even enjoyed is minor variation, yet it is what made you take seriously: "Maybe values of different humans are incompatible."
That terries me. People are different. The possiblity should have been screaming at you everytime you looked at the world. From personal interactions to study of history and different cultures examples of this are numerous.
So Eliezer said in his March 1st HPMOR progress report:
So I read that and it was certainly very much worth reading - thanks for the recommendation! Obviously, the following contains spoilers.
I'm confused about how the story is supposed to be "terrifying". I rarely find any fiction scary, but I suspect that this is about something else: I didn't think Failed Utopia #4-2 was "failed" either and in Three Worlds Collide, I thought the choice of the "Normal" ending made a lot more sense than choosing the "True" ending. The Optimalverse seems to me a fantastically fortunate universe, pretty much the best universe mammals could ever hope to end up in, and I honestly don't see how it is a horror novel, at all.
So, apparently there's something I'm not getting. Something that makes an individual's hard-to-define "free choice" more valuable than her much-easier-to-define happiness. Something like a paranoid schizophrenic's right not to be treated,
So I'd like the dumb version please. What's terrifying about the Optimalverse?