The thing that I found most frightening was that at the end when the protagonist had become a servant to Celestia, she optimized for the satisfaction of values through friendship and ponies, which made me think that in becoming an uplift, she eventually had to trade in her utility function for Celestia's.
So Eliezer said in his March 1st HPMOR progress report:
So I read that and it was certainly very much worth reading - thanks for the recommendation! Obviously, the following contains spoilers.
I'm confused about how the story is supposed to be "terrifying". I rarely find any fiction scary, but I suspect that this is about something else: I didn't think Failed Utopia #4-2 was "failed" either and in Three Worlds Collide, I thought the choice of the "Normal" ending made a lot more sense than choosing the "True" ending. The Optimalverse seems to me a fantastically fortunate universe, pretty much the best universe mammals could ever hope to end up in, and I honestly don't see how it is a horror novel, at all.
So, apparently there's something I'm not getting. Something that makes an individual's hard-to-define "free choice" more valuable than her much-easier-to-define happiness. Something like a paranoid schizophrenic's right not to be treated,
So I'd like the dumb version please. What's terrifying about the Optimalverse?