"Would you rather experience the mild distraction of a dust speck in your eye, or allow someone else to be tortured for fifty years?" "Would you rather be tortured for fifty years, or have someone else experience the mild discomfort of a dust speck in their eye?"
Asking this question to (let's say) humans will cause them to believe that only one person is getting the dust speck in the eye. Of course they're going to come up with the wrong answer if they have incomplete information.
Now you have to make your decisions knowing that the entire universe totally wouldn't mind having dust specks in exchange for preventing suffering for one other person. If that doesn't change your decision ... something is wrong.
There are two problems with this. The first is that if you take a number of people as big as 3^^^3 and ask them all this question, an incomprehensibly huge number will prefer to torture the other guy. These people will be insane, demented, cruel, or dreaming or whatever, but according to your ethics they must be taken into account. (And according to mine, as well, actually). The number of people saying to torture the guy will be greater than the number of Planck lengths in the observable universe. That alone is enough disutility to say "Torture away!"
The other problem is that you assert that "something is wrong" when my decision remains the same after a wrong question is asked with incomplete information 3^^^3 times does not change my decision. What is wrong? I can tell you that your intuition that "something must be wrong" is just incorrect. Nothing is wrong with the decision. (And this paragraph is for the LCPW where everyone answered selflessly to the question, which is of course not even remotely plausible).
The device allows for certain issues like slavery and income distribution to be determined beforehand. Would one vote for a society in which there is a chance of severe misfortune, but greater total utility? e.g, a world where 1% earn $1 a day and 99% earn $1,000,000 vs. a world where everyone earns $900,000 a day. Assume that dollars are utilons and they are linear (2 dollars indeed gives twice as much utility). What is the obvious answer? Bob chooses $900,000 a day for everyone.
But Bob is clever and he does not trust himself that his choice is the rational choice, so he goes into self-dialogue to investigate:
In the insurance industry, purchasing insurance comes with a price. Perhaps 1.5% premium of the cost of reimbursing you for your house that may burn down. The actuaries have run the probabilities and determine that you have a 1% chance that your house will burn down. Assume that all dollar amounts are utilons across all assets. Bob once again is a rational man. Every year Bob is chooses to pay 1.5% in premium even though his average risk is technically a 1% loss, because Bob is risk adverse. So risk adverse that he prefers a world in which he has less wealth, the .5% went to the insurance companies making a profit. Once again Bob questions his rationality on purchasing insurance:
Eliezer's post on Torture vs. Dust Specks [Here] has generated lots of discussion as well as what Eliezer describes as interesting [ways] of [avoiding] the question. We will do no sort of thing in this post, we will answer the question as intended; I will interpret that eye specks is cumulatively greater suffering than the suffering of 50 years.
My PVG tells me that I would rather have a speck in my eye, as well as the eye's of 3^^^3 people, than to risk to have one (perhaps me) suffer torture for 50 years, even though my risk is only 1/(3^^^3) which is a lot less than 50 years (Veil of ignorance). My PVG is what I will maximize, and doing so is the definition of instrumental rationality.
In short, the rational answer is not TORTURE or SPECKS, but depends on what your preference, values and goals are. You may be one of those whose preference is to let that one person feel torture for 50 years, as long as your actions that steer the future toward outcomes ranked higher in your preferences, you are right too.
Correct me if I am wrong but I thought rationality did not imply that there were absolute rational preferences, but rather rational ways to achieve your preferences...
I want to emphasize that in no way did I intend for this post to declare anything. And want to thank everyone in advance for picking apart every single word I have written. Being wrong is like winning the lottery. I do not claim to know anything, the assertive manner in which I wrote this post was merely a way to convey my ideas, of which, I am not sure off.