these approaches lead to the counterintuitive conclusion
Counter-intuitiveness is not a property of the conclusion alone. Surprise does have an important role in learning, because otherwise one may rationalize and modify a theory being tested to fit the facts discovered, so a good way to test a theory's informativeness is to set predictions beforehand and notice one's surprise whenever they don't pan out.
However, experiments are never tests of a single hypothesis but of complexes of hypotheses. A problem with testing a theory's predictions to evaluate the theory is that one may make terrible predictions; the strength of the predictions is not a test of the theory alone but of the tester's ability at extrapolating from a theory.
E.g.: I have several times encountered the argument, "If evolution is true, then why do scientists wear clothes?!" The idea is that by wearing clothes scientists admit to weaknesses of the bare skin design, and evolutionary theory would never predict less furry creatures, creatures who would sometimes be well served by clothing, to evolve from furry creatures. That's a really stupid thing to think, and the conclusion one may draw from noticing surprise that scientists believe in evolution and wear clothes is not that there is a problem with the theory, but with the predictor's extrapolation from the theory.
Evolution is not disproved by scientists wearing clothes. That scientists wear clothes is not even the most infinitesimally small theoretical evidence against evolution.
Obviously, I post in hindsight. But "Participants who indicated greater endorsement of utilitarian solutions had higher scores on measures of Psychopathy, machiavellianism, and life meaninglessness...a set of psychological characteristics that many would consider prototypically immoral," i.e. "Participants who were abnormally likely to endorse unpopular moral calculations were aneurotypical in ways society labels as bad, and they were more willing to defy societal conventions and trust their own moral judgement rather than society's, and/or ignore society's mores for personal gain." Surprise?
I vaguely recall reading about a study showing that most people were better at estimating their grocery bill when they abstractly estimated as they went along than when they tried to keep track of dollars and cents in their heads. Suppose this is true, pretend that it was well know and that the workings of cash registers were a total mystery.
One might find that those more adept at calculating would be less likely to take approximations as representing objective truths, and less likely to respect seeing $2.95 or $2.99 and $3.00 as being qualitatively the same, even though all three numbers are of the same type - amounts that are "threeish dollars".
Would the usually superior results of the approximaters' models show that the true operation of the registers most resembles converting prices into approximations and arriving at a larger number in a formula no one understood but was approximately addition, and then adding a bit for tax and translating the result into a dollar amount? Would the counters who laboriously try to add the prices and carry the numbers and then calculate the tax by multiplying (rather than "adding a bit" as the good approximating models do - they never accidentally forget to move a decimal point) be obviously wrong, by having the only models that fail catastrophically, in addition to their doing worse on average?
The counters might claim that counting best represents the type of thing cash registers do, and present contrived thought experiments like a person buying two items, one for $1.60 and the other for $2.80, with a 7.5% tax rate. Obviously, no experiment can be run with a cash register as such prices are never found in practice; counters say ($1.60+$2.80)*1.075=$4.73, while approximaters say $1.60 is the type of thing that is one-and-a-halfish-dollars (a fringe position, still noteworthy, is that it is the type of thing that is twoish dollars), $2.80 is the type of thing that is threeish dollars, add them together to get four-and-a-halfish dollars, and a little bit more than four-and-a-halfish dollars is the price of that basket of goods: "$4.75".
The consonance of "$4.75" and not "$4.73" to apes counting in base ten is not relevant to unraveling the secrets of the cash register, or much else, as interesting as it is in its own right. Likewise a strong emotional aversion to pushing the fat man tells us about the psychology of someone, not the rightness of pushing the fat man (which is dependent on all facts, including of course the psychology of each person).
So says the title of an interesting recent paper I stumbled on yesterday (ungated link; h/t Chris Bertram). Here's the abstract:
This conclusion is very much along the lines of some of my recent LW comments (for example, those I left in this thread). To me it seems quite obvious that in the space of possible human minds, those that produce on the whole reasonably cooperative and reliably non-threatening behavior are overwhelmingly unlikely to produce utilitarian decisions in trolley-footbridge and similar "sacrificial" problems.
Of course, what people say they would do in situations of this sort is usually determined by signaling rather than a realistic appraisal. Kind and philosophical utilitarians of the sort one meets on LW would be extremely unlikely to act in practice according to the implications of their favored theories in real-life "sacrificial" situations, so their views are by themselves not strong evidence of antisocial personality traits. However, actually acting in such ways would be, in my opinion, very strong evidence for such traits, which is correctly reflected in the typical person's fear and revulsion of someone who is known to have acted like that. I would venture to guess that it is in fact the signaling-driven disconnect between people's endorsement of utilitarian actions and the actual decisions they would make that makes the found correlations fairly low. (Assuming also that these tests really are strong indicators of antisocial personalities, of course, which I lack the knowledge to judge.)
(Also, endorsement of utilitarianism even just for signaling value causes its own problems, since it leads to political and ideological support for all sorts of crazy ideas backed by plausible-sounding utilitarian arguments, but that's a whole different issue.)
Here is also a full citation for reference: “The mismeasure of morals: Antisocial personality traits predict utilitarian responses to moral dilemmas”, by Daniel M. Bartels and David A. Pizarro, Cognition 121 (2011), pp. 154-161.
Edit: As Wei Dai points out in a comment, I should also add that some of the previous literature cited by Bartels and Pizarro has concluded that, in their words, "individuals with higher working memory capacity and those who are more deliberative thinkers are... more likely to approve of utilitarian solution." One the face of it, taken together with the conclusions of this paper, this would mean that propensity for utilitarian responses may stem from different causes in different individuals (i.e. deliberative thinking versus antisocial traits).
My own hypothesis, however, is that deliberative thinking leads to verbal utilitarian responses that are likely due to signaling, and that propensity for actual utilitarian "sacrificial" acts would have a much weaker link to deliberative thinking and a much stronger link to antisocial traits than mere utilitarian statements. Unfortunately, I don't know how this could be tested empirically in an ethical manner.