Warning: A sensitive person should not read this, to avoid nightmares.
BX. Yrg'f tvir hc pbzcnegzragnyvmngvba sbe n zbzrag, naq ybbx ng guvatf va n arne zbqr. Jung'f gehr, vf nyernql gehr. Jung rknpgyl vf vg yvxr, ybbxvat vagb Pguhyuh'f znq rlrf?
Svefg nffhzcgvba, Grtznex zhygvirefr. Vs lbh qba'g funer guvf nffhzcgvba, lbh ner fcnerq bs hapbhagnoyr hafcrnxnoyr ubeebef. Bar bs gurfr nqwrpgvirf vf abg zrnaf yvgrenyyl: zngurzngvpnyyl fcrnxvat, n frg bs ovanel cebtenzf bs svavgr yratgu vf pbhagnoyr. Ba gur bgure unaq, gur "hafcrnxnoyr ubeebef" cneg vf yvgrenyyl gehr: gurer ner guvatf jr pbafvqre ubeevoyr, naq fbzr bs gurz ner fb ubeevoyr gung rira gur fubegrfg pbzcerffrq fgevat qrfpevovat gurve ubeebe vf gbb ybat gb or cebabhaprq va na nirentr uhzna yvsrgvzr. Gur bayl tbbq guvat vf gung Fbybzbabss cevbe erqhprf gur cebonovyvgl bs gur hafcrnxnoyr ubeebef, nygubhtu znal fcrnxnoyr ubeebef trg fvzvyne cevbe cebonovyvgl guna gur havirefr jr pbairagvbanyyl vaunovg abj. Oevat vg gb gur arne zbqr naq vzntvar nyy gur cnva npebbff gur Grtznex zhygvirefr. Lbh yvgrenyyl pna'g: orpnhfr sbe nalguvat lbhe oenva pna vzntvar, 3^^^3 gvzrf jbefr guvat vf unccravat fbzrjurer. Oevat vg gb gur arne zbqr: vg'f gehr; vg'f gur erny cnva bs rirel xvaq.
Frpbaq nffhzcgvba, ngbzf unir ab vqragvgl. Abg bayl ner hafcrnxnoyr ubeebef unccravat npebff gur Grtznex havirefr; jurarire gurer vf n pbasvthengvba bs ngbzf fnzr nf lbhe pheerag obql vaibyirq, gurl ner unccravat yvgrenyyl gb lbh. Nyfb, sbe rirel zbzrag va guvf havirefr, gurer vf n havirefr gung orunirq vqragvpnyyl hc gb guvf zbzrag, ohg ng guvf zbzrag (ol n arj culfvpny ynj, fcrpvsvp sbe gung havirefr) hafcrnxnoyr ubeebef fgneg unccravat. Ng rirel zbzrag, yvgrenyyl. Oevat vg gb gur arne zbqr. Gurer vf n pbcl bs lbh gung ernqf gurfr jbeqf, naq evtug va gur sbyybjvat zbzrag... jungrire lbhe vzntvangvba pna vafreg urer, n 3^^^3 gvzrf jbefr guvat unf unccrarq. Gb lbh, va gur bgure havirefr. Vg'f erny.
Pbzcnerq jvgu guvf, Cnfpny zhttvat vf whfg n fvyyl unezyrff guerng. Vg qvffbyirf va gur cher ubeebe bs gur Grtznex zhygvirefr. Vg'f yvxr jura lbhe ubhfr oheaf qbja naq lbhe pne rkcybqrf, naq gurer vf guvf Cnfpny thl pbzvat naq jneavat lbh gung hayrff lbh qribgr lbhe jubyr yvsr gb evghnyf naq oynpxznvy, lbh zvtug ybfr lrg nabgure craal! Jung bgure nccebcevngr nafjre vf gurer orfvqrf n ulfgrevpny ynhtugre?
Gur gubhtug gung vf fhccbfrq gb yrg hf fyrrc pnyzyl, vf gung nyy gurfr ubeebef, rfcrpvnyyl gur hafcrnxnoyr barf, unir ernyyl ybj cevbe cebonovyvgl. Cebonovyvgl rcfvyba vf abg yvgrenyyl mreb, ohg gur qvssrerapr orgjrra mreb naq rcfvyba vf bayl rcfvyba, qba'g pner nobhg vg -- gurfr guvatf ner bayl erny sbe crbcyr hasbeghangr rabhtu gb yvir va gubfr havirefrf. Gb guvf yrg zr ercyl gung gur cevbe cebonovyvgl bs bhe pbairagvbany havirefr vf nyfb rcfvyba. Fb creuncf jr fubhyq abg bayl pner nobhg nofbyhgr cevbe cebonovyvgvrf, ohg nyfb nobhg eryngvir cevbe cebonovyvgvrf pbzcnerq jvgu cevbe cebonovyvgl bs bhe havirefr. (Naljnl, V thrff zbfg havirefrf jvgu aba-rcfvyba cevbe cebonovyvgvrf ner hanoyr gb pbagnva vagryyvtrag yvsr.) Ba gur yriryf pbzcnenoyr jvgu bhe havirefr, znal ubeebef ner unccravat gbb, ohg ng yrnfg gurl ner abg unccravat gb lbh. (V qvqa'g znxr na rknpg pnyphyngvba, ohg V fhccbfr gung n eryngviryl fvzcyr havirefr, pbzcnenoyr jvgu bhe havirefr, unf n irel fznyy punapr bs pbagnvavat lbhe rknpg ercyvpn.)
Gurer vf nabgure fpnel gubhtug, juvpu vf abg onfrq ba bssvpvny YJ grnpuvatf, fb vg pna cbffvoyl or jebat: Fbzr havirefrf unir n yvzvgrq yvsrgvzr. Nsgre tvira gvzr gurl eha bhg bs raretl naq rira vs gurl pbagvahr gb rkvfg va fbzr frafr, gurl fgbc rkvfgvat naguebcvpnyyl; gurl ner abg noyr gb pbagnva vagryyvtrag yvsr. V fhccbfr gung ba nirentr guvf unccraf fbbare gb gur fvzcyr havirefrf -- gur barf jvgu uvture cevbe cebonovyvgl. (Vaghvgviryl, fvzcyr ehyrf cebivqr yrff bccbeghavgl sbe ybat eha. Sbe rnpu fvzcyr havirefr, gurer vf n pbzcyrk havirefr jvgu n fhcrefrg bs vgf ehyrf, noyr gb eha ybatre.) Fb rira erylvat ba uvtu cevbe cebonovyvgvrf vf abg rabhtu va gur ybat eha. Nf gur gvzr tbrf ba, gur fvzcyr havirefrf qvr bhg, naq bhe yvivat pbcvrf jvyy zber bsgra svaq gurzfryirf va gur zber pbzcyrk havirefrf. Ubjrire ybj vf n cevbe cebonovyvgl bs n gehyl ubeevsvp havirefr, gurer jvyy or n gvzr jurer havirefrf jvgu guvf yriry bs cevbe cebonovyvgl jvyy or qbzvanag va n fhowrpgvir rkcrevrapr. Va lbhe fhowrpgvir rkcrevrapr.
Ng gur raq, urer vf bar unccl gubhtug gung V qvq abg guvax jura jevgvat gur cerivbhf pbzzrag: Rira jura gur pbzcyrk havirefrf noyr gb pbagnva hafcrnxnoyr ubeebef orpbzr qbzvanag va n fhowrpgvir rkcrevrapr, bayl n cneg bs gurz jvyy pbagnva gurfr ubeebef; cbffvoyl n irel fznyy cneg. Fb n qbzvanag cneg bs fhowrpgvir rkcrevrapr znl fgvyy or cerggl... abezny.
Fgvyy, vg vf abg tbbq sbe n frafvgvir zvaq gb guvax gbb zhpu nobhg ubeebef gung ner erny ba gur rcfvyba yriry, ohg xvaq bs erny abarguryrff. Nccylvat guvf gb eryvtvba, rnpu eryvtvba vf xvaq bs gehr (vapyhqvat nal eryvtvba lbh jbhyq vairag ng guvf zbzrag, vapyhqvat nal cbffvoyr zhttvat, vapyhqvat gur zhttvatf gung jrer arire qrpynerq rkcyvpvgyl), ohg gerngvat gurz frevbhfyl zrnaf cevivyrtvat n ulcbgurfvf; gurl ner gehr bayl va na rcfvyba fhofrg bs gur zhygvirefr. Znlor guvf jubyr pbzzrag vf thvygl bs cevivyrtvat n ulcbgurfvf, gbb.
RQVG: Npghnyyl, jura V guvax nobhg vg zber, zl nffhzcgvba bs yrff cebonoyr havirefrf orpbzvat qbzvanag yngre va gvzr vf cebonoyl jebat. Gvzr vf whfg nabgure cnenzrgre va gur gvzryrff culfvpf. Vs gur havirefrf jvgu uvture cebonovyvgvrf qvr bhg fbbare, vg qbrf abg zrna gung gur fhowrpgvir rkcrevrapr jvyy fuvsg gb yrff cebonoyr havirefrf; vg whfg zrnaf gung gur fhowrpgvir rkcrevrapr jvyy svaq vgfrys va gur rneyl cunfrf bs gur havirefr.
Warning: A sensitive person should not read this, to avoid nightmares.
Apparently, I am not a sensitive person. I scoff at Tegmark multiverses and Pascalian Muggers: these themselves are the nightmares of thought gone wrong. And I won't be Pascal-Mugged into spending any significant effort in working out exactly how these thoughts have gone wrong. I have other things to do than hare off on someone else's trip.
There seems to be some continuing debate about whether or not it is rational to appease a Pascal Mugger. Some are saying that due to scope insensitivity and other biases, we really should just trust what decision theory + Solomonoff induction tells us. I have been thinking about this a lot and I'm at the point where I think I have something to contribute to the discussion.
Consider the Pascal Mugging "Immediately begin to work only on increasing my utility, according to my utility function 'X', from now on, or my powers from outside the matrix will make minus 3^^^^3 utilons happen to you and yours."
Any agent can commit this Pascal's mugging (PM) against any other agent, at any time. A naive decision-theoretic expected-utility optimizer will always appease the mugger. Consider what the world would be like if all intelligent beings were this kind of agent.
When you see an agent, any agent, your only strategy would be to try to PM it before it PMs you. More likely, you will PM each other simultaneously, in which case the agent which finishes the mugging first 'wins'. If you finish mugging at the same time, the mugger that uses a larger integer in its threat 'wins'. (So you'll use the most compact notation possible and things like, "minus the Busy Beaver function of Graham's number utilons".)
This may continue until every agent in the community/world/universe has been PMed. Or maybe there could be one agent, a Pascal Highlander, who manages to escape being mugged and has his utility function come to dominate...
Except, there is nothing stipulating that the mugging has to be delivered in person. With a powerful radio source, you can PM everyone in your future light-cone unfortunate enough to decode your message, potentially highjacking entire distant civilizations of decision-theory users.
Pascal's mugging doesn't have to be targeted. You can claim to be a Herald of Omega and address your mugging "to whoever receives this transmission"
Another strategy might be to build a self-replicating robot (itself too dumb to be mugged) which has a radio which broadcasts a continuous fully general PM, and send it out into space. Then you commit suicide to avoid the fate of being mugged.
Now consider a hypothetical agent which completely ignores muggers. And mugs them back.
Consider what could happen if we build an AI which is friendly in every possible respect except that it appeases PMers.
To avoid this, you might implement a heuristic that ignores PMs on account of the prior improbability of being able to decide the fate of so many utilons, as Robin Hanson suggested. But an AI using naïve expected utility + SI may well have other failure modes roughly analagous to PM that we won't think of until its too late. You might get agents to agree to pre-commit to ignore muggers, or to kill them, but to me this seems unstable. A bandaid that's not addressing the heart of the issue. I think an AI which can envision itself being PMed repeatedly by every other agent on the planet and still evaluate appeasement as the lesser evil cannot possibly be a Friendly AI, even if it has some heuristic or ad hoc patch that says it can ignore the PM.
Of course there's the possibility that we are in a simulation which is occasionally visited by agents from the mother universe, which really does contain 3^^^^3 utilons/people/dustspecks. I'm not convinced acknowledging this possibility changes anything. There's nothing of value that we, as simulated people, could give our Pascal Mugging simulation overlords. Their only motivation would be as absolute sadistic sociopaths, but if that's the reality of the multiverse, in the long term we're screwed no matter what we do, even with friendly AI. And we certainly wouldn't be in any way morally responsible for their actions.
Edit 1: fixed typos