Tuning one's preference function is a constrained optimization problem. What I want is a preference function simple enough for my very finite brain to be able to compute it in real time, and that does a good job (whatever exactly that means) on-some-kind-of-average over some plausible probability distribution of scenarios it's actually going to have to deal with.
Choosing between torturing one person for 50 years and giving 3^^^3 people minimally-disturbing dust specks is a long, long way outside the range of scenarios that have non-negligible probability of actually coming up. It's a long, long way outside the range of scenarios that my decision-theoretic intuition has been tuned on by a few million years of evolution and a few decades of experience.
My preference function returns values with (something a bit like) error bars on them. In this case, the error bars are much larger than the values: there's much more noise than signal. That's a defect, no doubt about it: a perfect preference function would never do that. A perfect preference function is probably also unattainable, given the limitations of my brain.
What possible reason is there for supposing that my preference function would be improved, for the actual problems it actually gets used for, by nailing down its behaviour far outside the useful range?
If there were good reason to think that decision theory is like (a Platonist's view of) logic, with a Right Answer to every question and no limits to its validity, then there would be reason to expect that nailing down my preference function's values out in la-la-land would be useful. But is there? Not that I know of. Decision theory is an abstraction of actual human preferences. Applying it to problems like Eliezer's might be like extrapolating quantum mechanics down to a scale of 10^-(10^100) m.
I am not wholly unsympathetic to the many commenters in Torture vs. Dust Specks who argued that it is preferable to inflict dust specks upon the eyes of 3^^^3 (amazingly huge but finite number of) people, rather than torture one person for 50 years. If you think that a dust speck is simply of no account unless it has other side effects - if you literally do not prefer zero dust specks to one dust speck - then your position is consistent. (Though I suspect that many speckers would have expressed a preference if they hadn't known about the dilemma's sting.)
So I'm on board with the commenters who chose TORTURE, and I can understand the commenters who chose SPECKS.
But some of you said the question was meaningless; or that all morality was arbitrary and subjective; or that you needed more information before you could decide; or you talked about some other confusing aspect of the problem; and then you didn't go on to state a preference.
Sorry. I can't back you on that one.
If you actually answer the dilemma, then no matter which option you choose, you're giving something up. If you say SPECKS, you're giving up your claim on a certain kind of utilitarianism; you may worry that you're not being rational enough, or that others will accuse you of failing to comprehend large numbers. If you say TORTURE, you're accepting an outcome that has torture in it.
I falsifiably predict that of the commenters who dodged, most of them saw some specific answer - either TORTURE or SPECKS - that they flinched away from giving. Maybe for just a fraction of a second before the question-confusing operation took over, but I predict the flinch was there. (To be specific: I'm not predicting that you knew, and selected, and have in mind right now, some particular answer you're deliberately not giving. I'm predicting that your thinking trended toward a particular uncomfortable answer, for at least one fraction of a second before you started finding reasons to question the dilemma itself.)
In "bioethics" debates, you very often see experts on bioethics discussing what they see as the pros and cons of, say, stem-cell research; and then, at the conclusion of their talk, they gravely declare that more debate is urgently needed, with participation from all stakeholders. If you actually come to a conclusion, if you actually argue for banning stem cells, then people with relatives dying of Parkinson's will scream at you. If you come to a conclusion and actually endorse stem cells, religious fundamentalists will scream at you. But who can argue with a call to debate?
Uncomfortable with the way the evidence is trending on Darwinism versus creationism? Consider the issue soberly, and decide that you need more evidence; you want archaeologists to dig up another billion fossils before you come to a conclusion. That way you neither say something sacrilegious, nor relinquish your self-image as a rationalist. Keep on doing this with all issues that look like they might be trending in an uncomfortable direction, and you can maintain a whole religion in your mind.
Real life is often confusing, and we have to choose anyway, because refusing to choose is also a choice. The null plan is still a plan. We always do something, even if it's nothing. As Russell and Norvig put it, "Refusing to choose is like refusing to allow time to pass."
Ducking uncomfortable choices is a dangerous habit of mind. There are certain times when it's wise to suspend judgment (for an hour, not a year). When you're facing a dilemma all of whose answers seem uncomfortable, is not one of those times! Pick one of the uncomfortable answers as the best of an unsatisfactory lot. If there's missing information, fill in the blanks with plausible assumptions or probability distributions. Whatever it takes to overcome the basic flinch away from discomfort. Then you can search for an escape route.
Until you pick one interim best guess, the discomfort will consume your attention, distract you from the search, tempt you to confuse the issue whenever your analysis seems to trend in a particular direction.
In real life, when people flinch away from uncomfortable choices, they often hurt others as well as themselves. Refusing to choose is often one of the worst choices you can make. Motivated continuation is not a habit of thought anyone can afford, egoist or altruist. The cost of comfort is too high. It's important to acquire that habit of gritting your teeth and choosing - just as important as looking for escape routes afterward.