Hmm. I think, actually, that I'm figuring out the whole problem with Torture vs. Dust Specks.
The fundamental problem with the "shut up and do the math" argument is the assumption of the type of math underlying it. If accurate modeling of utility requires assigning a mere real number to accurately model the disutility of both dust and torture, "3^^^3 is really big" gives us lots of good guidance as to the actual results, because we know what happens if you put a real number through an equation where 3^^^3 increases things. And even if 3^^^3 isn't big enough, you can sub in 5^^^^^5 or 9^^^^^^^^^9 and change the result without changing the logic.
But, if accurate modeling of utility requires assigning a matrix (or hypercomplex number) to accurately model the disutility of both dust and torture, instead of a mere real number, and a nonlinear utility function is applied, well, exponentiation of matrices is difficult, and "3^^^3 is really big" does not give us good guidance as to the actual results.
Now, does utility need to be expressed as a matrix? It wouldn't if our sole value was, for example, maximizing inclusive genetic fitness. But since Thou Art Godshatter, we have multiple different internal systems voting on the utility of any particular action or state. Arrow's Impossibility theorem then proves we can't reliably turn the results of the voting into something that would fit on the real number line.
It seems likely that when you throw around a big enough number (3^^^3 or 9^^^^^^^^^9 or whatever) it will overwhelm the initial difference . . . but the difference has moved from "shut up and acknowledge the inevitable result of the math" to "dammit, we don't have enough computing resources for this, we have to guess."
Perhaps a related concept is this: Are utility functions necessarily scalar, or can they be vector functions? If they are scalar, then there is always a large enough value of dust specks to outweigh the disutility of one person being tortured. If utility can be a vector function, then it's possible that the disutility of torture outweighs the disutility of any number of dust specks.
The device allows for certain issues like slavery and income distribution to be determined beforehand. Would one vote for a society in which there is a chance of severe misfortune, but greater total utility? e.g, a world where 1% earn $1 a day and 99% earn $1,000,000 vs. a world where everyone earns $900,000 a day. Assume that dollars are utilons and they are linear (2 dollars indeed gives twice as much utility). What is the obvious answer? Bob chooses $900,000 a day for everyone.
But Bob is clever and he does not trust himself that his choice is the rational choice, so he goes into self-dialogue to investigate:
In the insurance industry, purchasing insurance comes with a price. Perhaps 1.5% premium of the cost of reimbursing you for your house that may burn down. The actuaries have run the probabilities and determine that you have a 1% chance that your house will burn down. Assume that all dollar amounts are utilons across all assets. Bob once again is a rational man. Every year Bob is chooses to pay 1.5% in premium even though his average risk is technically a 1% loss, because Bob is risk adverse. So risk adverse that he prefers a world in which he has less wealth, the .5% went to the insurance companies making a profit. Once again Bob questions his rationality on purchasing insurance:
Eliezer's post on Torture vs. Dust Specks [Here] has generated lots of discussion as well as what Eliezer describes as interesting [ways] of [avoiding] the question. We will do no sort of thing in this post, we will answer the question as intended; I will interpret that eye specks is cumulatively greater suffering than the suffering of 50 years.
My PVG tells me that I would rather have a speck in my eye, as well as the eye's of 3^^^3 people, than to risk to have one (perhaps me) suffer torture for 50 years, even though my risk is only 1/(3^^^3) which is a lot less than 50 years (Veil of ignorance). My PVG is what I will maximize, and doing so is the definition of instrumental rationality.
In short, the rational answer is not TORTURE or SPECKS, but depends on what your preference, values and goals are. You may be one of those whose preference is to let that one person feel torture for 50 years, as long as your actions that steer the future toward outcomes ranked higher in your preferences, you are right too.
Correct me if I am wrong but I thought rationality did not imply that there were absolute rational preferences, but rather rational ways to achieve your preferences...
I want to emphasize that in no way did I intend for this post to declare anything. And want to thank everyone in advance for picking apart every single word I have written. Being wrong is like winning the lottery. I do not claim to know anything, the assertive manner in which I wrote this post was merely a way to convey my ideas, of which, I am not sure off.