Eliezer_Yudkowsky comments on Open Thread, November 1 - 7, 2013 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (299)
(Comment cosmetically edited in response to Kaj_Sotala, and again to replace a chunk of text that fell in a hole somewhere)
OK, I'll have a go (will be incomplete).
People in general will find the Optimalverse unpleasant for a lot of reasons I'll ignore; major changes to status quo, perceived incompatibility with non-reductionist worldviews, believing that a utopia is necessarily unpleasant or Omelas-like (a variant of this fallacy?), and lots of even messier things.
People on LessWrong may be thinking about portions of the Fun Theory Sequence that the Optimalverse conflicts with, and in some cases they may think that these conflicts destroy all of the value of the future, hence horror.
(rot13 some bits that might consitute spoilers)
Humans want things to go well, but they also want things to have been able to go badly, such that they made the difference. Relevant: Living By Your Own Strength, Free to Optimize.
The existence of a superintelligence makes human involvement superfluous, and humans do not want this to happen. Relevant: Amputation of Destiny.
Gur snpg gung gur NV vf pbafgenvarq gb fngvfsl uhzna inyhrf gur cbal jnl zrnaf gung n uhtr nzbhag bs cbffvoyr uhzna rkcrevrapr vf abj vzcbffvoyr gb rire ernyvfr. Eryrinag: Hzz... znlor Value is Fragile? Abg dhvgr. Uryc zr bhg urer, thlf! (nyfb, vafreg lbhe bja cersreerq snaqbz wbxr nobhg cbbe Ylen arire trggvat gb unir unaqf rgp.)
Nf lbh zragvbarq, gur jnl va juvpu gur NV'f cnegvphyne qrsvavgvba bs "uhzna" jnf abg evtug naq pna arire or zbqvsvrq, urapr nyvra ncbpnylcfrf. Eryrinag: The Hidden Complexity of Wishes
Themes that are more explicit after the extra worldbuilding in Caelum est Conterrens:
Gjb cbffvoyr svany nggenpgbef sbe uhzna tebjgu ner cerfragrq (Ybbc naq Enl Vzzbegnyf):
Fbzr urer ner abg fngvfsvrq jvgu rvgure naq ernyyl, ernyyl ubcr gurer vf n guveq jnl sbe uhznaf gb npuvrir haobhaqrq tebjgu gung erznvaf zrnavatshy (ol gurve yvtugf).
Notes:
I'm sympathetic to your position; this is the substance of my comment here that I think I understand what's supposed to horrify me.
That comment of mine is no doubt wrong; there will be things that don't horrify me that I didn't even realise were supposed to.
There are quick and obvious comebacks to nearly all the above points. In a lot of cases, those quick comebacks are dealt with in the linked articles. Read the Fun Theory Sequence; it's my favorite sequence, despite the fact that I disagree with more of it than any of the others.
Endorsed as a good summary.