Hmm. I think, actually, that I'm figuring out the whole problem with Torture vs. Dust Specks.

The fundamental problem with the "shut up and do the math" argument is the assumption of the type of math underlying it. If accurate modeling of utility requires assigning a mere real number to accurately model the disutility of both dust and torture, "3^^^3 is really big" gives us lots of good guidance as to the actual results, because we know what happens if you put a real number through an equation where 3^^^3 increases things. And even if 3^^^3 isn't big enough, you can sub in 5^^^^^5 or 9^^^^^^^^^9 and change the result without changing the logic.

But, if accurate modeling of utility requires assigning a matrix (or hypercomplex number) to accurately model the disutility of both dust and torture, instead of a mere real number, and a nonlinear utility function is applied, well, exponentiation of matrices is difficult, and "3^^^3 is really big" does not give us good guidance as to the actual results.

Now, does utility need to be expressed as a matrix? It wouldn't if our sole value was, for example, maximizing inclusive genetic fitness. But since Thou Art Godshatter, we have multiple different internal systems voting on the utility of any particular action or state. Arrow's Impossibility theorem then proves we can't reliably turn the results of the voting into something that would fit on the real number line.

It seems likely that when you throw around a big enough number (3^^^3 or 9^^^^^^^^^9 or whatever) it will overwhelm the initial difference . . . but the difference has moved from "shut up and acknowledge the inevitable result of the math" to "dammit, we don't have enough computing resources for this, we have to guess."