Well, we have better reasons to believe that arithmetic is consistent than we have to believe that human beings' strong moral impulses are coherent in cases outside of everyday experience. I think much of the point of the SPECKS vs. TORTURE debate was to emphasize that our moral intuitions aren't perceptions of a consistent world of values, but instead a thousand shards of moral desire which originated in a thousand different aspects of primate social life.
For one thing, our moral intuitions don't shut up and multiply. When we start making decisions that affect large numbers of people (3^^^3 isn't necessary; a million is enough to take us far outside of our usual domain), it's important to be aware that the actual best action might sometimes trigger a wave of moral disgust, if the harm to a few seems more salient than the benefit to the many, etc.
Keep in mind that this isn't arguing for implementing Utilitarianism of the "kill a healthy traveler and harvest his organs to save 10 other people" variety; among its faults, that kind of Utilitarianism fails to consider its probable consequences on human behavior if people know it's being implemented. The circularity of "SPECKS" just serves to point out one more domain in which Eliezer's Maxim applies:
You want to scream, "Just give up already! Intuition isn't always right!"
This came to mind: What you intuitively believe about a certain statement may as well be described as an "emotion" of "truthiness", triggered by the focus of attention holding the model just like any other emotion that values situations. Emotion isn't always right, estimate of plausibility isn't always right, but these are basically the same thing. I somehow used to separate them, along the line of probability-utility distinction, but this is probably more confusing then helpful a distinction, with truthiness on its own and the concept of emotions containing everything but it.
In line with my fine tradition of beating old horses, in this post I'll try to summarize some arguments that people proposed in the ancient puzzle of Torture vs. Dust Specks and add some of my own. Not intended as an endorsement of either side. (I do have a preferred side, but don't know exactly why.)
Oh what a tangle. I guess Eliezer is too altruistic to give up torture no matter what we throw at him; others will adopt excuses to choose specks; still others will stay gut-convinced but logically puzzled, like me. The right answer, or the right theory to guide you to the answer, no longer seems so inevitable and mathematically certain.
Edit: I submitted this post to LW by mistake, then deleted it which turned out to be the real mistake. Seeing the folks merrily discussing away in the comments long after the deletion, I tried to undelete the post somehow, but nothing worked. All right; let this be a sekrit area. A shame, really, because I just thought of a scenario that might have given even Eliezer cause for self-doubt: