Gunnar_Zarncke comments on Memory is Everything - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (37)
You seem to be objecting that this is an unfair thought experiment because humans were not designed to contemplate these extreme cases.
But that's precisely the point! These extreme cases might not have been present in our ancestral environment. They might not be present now. But there is a decent chance that they are coming...that, someday, we will be literally offered this choice or something analogous to it by a superintelligence AI who, even if friendly, honestly just wants to ascertain our preferences. Perhaps the superintelligent AI can create a utopia for us, but during the week in which it is being constructed by nano-robots, the Earth's surface will be scoured to bits and resemble a living hell. Would we still want it?
That's why this post poses a good, relevant question. And I see that most people seem to just want to squirm in their seats and complain about the tough question rather than answer it.
Me, I would take option 2, assuming that the billion dollars I would get afterwards would enable more than a week of bliss of a magnitude in the positive direction equal to or greater than the magnitude of suffering I would experience for that horrible week.
Plus, no matter how back that first week of torture is, I will know in the back of my head during all of that that I can look forward to a billion dollars at the end of it. Now, if part of the torture involves temporarily deleting my memory of having made the deal and making me confused about why I am being tortured and how long it will last (possibly forever), it would make me think a bit harder about the deal, but I would still take option 2.
You got me wrong. I'm not objecting. The though experiment is a valid and interesting one. It's just that the answers it elicidates fall into a certain class of problems which I pointed out.