A way to make this argument would be to claim human values about how to interpret human values are themselves complex. As an illustration of this one could point out that the naive utilitarian position in torture-vs.-specks totally disregards the preferences of the 3^^^3 people that pertain to what answer to the dilemma the answerer should give --- we'll assume those 3^^^3 people mostly do not hold naive utilitarian ethics. Then the answerer's problem is much tougher, because to disregard those people's preferences he has to be confident that he understands morality better than they do, which, for a naive preference utilitarian, is a self-defeating position.
(And my knowledge of implicit utilitarian meta-ethics gets iffy here, but the naive utilitarian also has no sense in which he could say that choosing specks was wrong, because wrongness is only determined by preferences. He could only say he himself didn't prefer to do what his ethics told him to do -- but his ethics are his preferences, so his claim to not prefer specks would be mostly wrong, otherwise self-contradicting.)
I wrote a post about this, and also about non-obvious and important considerations for the trolley problem. Hopefully sound arguments in this vein will cause people to recognize moral uncertainty and especially meta-ethical uncertainty as a serious problem. The neglect of the subject increases the chance that an FAI team will see a meta-ethical consensus around them when there isn't one -- consider that Eliezer has (purposefully-exaggeratedly?) claimed that meta-ethics is a solved problem, even though folk like Wei Dai disagree.
Actually, re implicit utilitarian meta-ethics, I have some confusions. Assume preference utilitarianism. We'll say most people think utilitarianism is wrong. They'd prefer you used virtue ethics. They think morality is hella important, moreso than their other preferences -- that's feasible. In such a world, would a preference utilitarian thus be obliged to forget utilitarianism and use virtue ethics? And is he obliged to think about ethics and meta-ethics in the ways preferred by the set of people whose preferences he's tracking? If so, isn't utilitarianis...
Reasoning using a representation of human utility that's a simple continuum from pain to pleasure, as torture vs dust specks does, is a shattering blow to the complexity of value.
Making moral decisions of such vast scope without understanding the full multidimensionality of human experience and utility is completely irresponsible. An AI using the kind of reasoning found in Torture vs Specks would probably just wirehead everyone for huge-integer-pleasure for eternity.
I don't pretend to know the correct answer to Torture vs Specks because I don't have a full understanding of human value, and because I don't understand how to do calculations with hypercomplex numbers. A friendly AI *has* to take into account the full complexity of our value and not just a one-dimensional continuum whenever it makes any moral decision. So only a friendly AI which has correctly extrapolated our values can know to high confidence the best answer to torture vs specks.
(edit 1) re:Oscar Cunningham
Why does complexity of value apply here specifically and not a curiosity stopper? Well consequentialist problems come in different difficulty levels - Torture for 5 years vs Torture for 50 years is easy - torture is bad, so less torture is less bad. You are comparing amounts of the same thing. You don't have to understand complexity of value to do that. To compare the value of two very different things, like Torture and Specks, requires you to understand the complexity of value. You can't simplify experiences to integers, because complex value isn't simply an integer.
The intuition that torture must be outweighed by a large enough number of specks, is just that: an intuition. You don't know the dynamics involved in a formal comparison based on a technical understanding of complex value.