Actually, I have pretty much your same misgivings/objections; it didn't feel particularly scary to me either :-/
maybe it's the fact that uploading/etc. is basically a foregone conclusion when facing a superintelligence? Although I thought that was obvious from the concept itself :-/
So Eliezer said in his March 1st HPMOR progress report:
So I read that and it was certainly very much worth reading - thanks for the recommendation! Obviously, the following contains spoilers.
I'm confused about how the story is supposed to be "terrifying". I rarely find any fiction scary, but I suspect that this is about something else: I didn't think Failed Utopia #4-2 was "failed" either and in Three Worlds Collide, I thought the choice of the "Normal" ending made a lot more sense than choosing the "True" ending. The Optimalverse seems to me a fantastically fortunate universe, pretty much the best universe mammals could ever hope to end up in, and I honestly don't see how it is a horror novel, at all.
So, apparently there's something I'm not getting. Something that makes an individual's hard-to-define "free choice" more valuable than her much-easier-to-define happiness. Something like a paranoid schizophrenic's right not to be treated,
So I'd like the dumb version please. What's terrifying about the Optimalverse?