I generally agree, but I challenge the claim that the (mostly social) failures of conscious consequentialist reasoning are just a matter of speed-of-calculation versus a cached rule. In most social situations, one or several such rules feel particularly salient to our decisionmaking at any moment, but the process by which these particular rules seem salient is the essence of our real (unconscious) calculations.

We already have a well-developed neural framework for social situations, and a conscious calculation of utility is unlikely to outperform that framework across that domain (though it can lead to occasional insights that the heuristics of the framework miss). Compare it to our native 'physics engine' that allows us to track objects, move around and even learn to catch a ball, versus the much slower and mistake-prone calculations that can still give the better answer when faced with truly novel situations (our intuitive physics is wrong about what happens to a helium balloon in the car when you slam on the brakes, but a physics student can get the right answer with conscious thought).

I suggest that attempting to live one's entire life by either conscious expected-utility maximization or even by execution of consciously-chosen low-level heuristics is going to work out badly. What works better for human beings is to generally trust the unconscious in familiar social domains, but to observe and analyze ourselves periodically in order to identify (and hopefully patch) some deleterious biases. We should also try to rely more on conscious Bayesian reasoning in domains (like scientific controversies or national politics) that were unlikely to be optimized for in the ancestral environment.

This leaves aside, of course, the question of what to do when one's conscious priorities seem to oppose one's unconscious priorities (which they do, not in most things, but in some crucial matters).

I agree that, if you want to experience the outcomes typical of other people in situations similar to yours, there's not much to be gained by thinking about your utility function. In general it's foolish to rely on inside-view thinking if you have more than enough outside-view information. But if you don't like your peer group's outcomes — maybe you got into an unfortunate situation, or you have some wacky preferences like wanting to build a friendly AI, or you just want to show off your rationalist chops — then using expected utility could still be valuable.

Our inability to model all possible histories of the universe does not make the concept of expected utility useless. We can apply it at whatever level of detail is tractable. We can't do real-time expected utility calculations, but we can use generalizations and implications of utility theory to discover when our rules of thumb are contradictory or self-defeating.

We don't need to throw away intuition either, we can consider it from an outside view. In situations where following intuition gives the best expected outcome, then follow intuition.

Your argument boils down to "Calculating expected utilities is hard, therefore its rarely worth trying." I agree with the premise, but the conclusion goes too far.

There are many situations in which I have done better by considering possible outcomes, the associated likelihoods, and payoffs. I have used this reasoning in my (short) career, in decisions about investing/insurance, in my relationships, and in considering what charities are worthwhile.

Yes, in many situations good habits and heuristics are more useful than thinking about probabilities, but you get mighty close to reifying our "intuitive decision procedure" aka our stone-age brain which was programmed to maximize inclusive fitness (which does not weigh heavily in my utility function) in an environment which was very different from the one in which we now find ourselves.

You are looking at application of decision theory in this context from a wrong angle. You see a decision procedure as constructed bottom up, a complete toy model, that can't possibly match the challenge of the real thing. Instead, decision procedure here is a restriction, a principle that allows to catch inconsistencies in the messy human decision-making process.

If you believe that choosing Y requires X to be true, you don't believe X to be true, but you choose Y, something fishy is going on. You believe in the correctness of the rules, you observe that your stated opinions don't match the rules, and so you are forced to revise your opinions. The complexities of the physical world are not the issue, this is a device for the sanity of mind, and it can be applied at any level of granularity, with concepts however fuzzy and imprecise.

So some outcomes must be conflated with each other and the set of outcomes must be described in rough heuristic terms (e.g. lives where I in order to cut the set down in size.

Lives where you what?

I don't agree with your arguments. First, nobody is proposing infinitely accurate utility function, just that a rough utility function is a good approximation of human behaviour descriptively and prescriptively.

As for your particular examples:

• I don't see how being stupid about reality of relationships is adding much value. You should be aware what are the chances of infidelity etc. If knowing these chances you decide not to monitor your lover, that's a purely consequentialist decision.
• Moderate deontology can be emulated in consequentialism by simply assigning values to following and breaking rules. I don't think anybody's values are truly absolute.
• "Delusions" can be handled by taking an outside view and adding extra terms to the function, for example using all research of what influences our happiness.

None of them is terribly convincing.

And arguing from consequences - the entire field of economics in pretty much all its forms is based on assumption that utility maximization is a good approximation of human behaviour. If utility funcitons aren't even that much, then the entire economics is worthless almost automatically. It doesn't seem to be entirely worthless, so utility functions seem to have some meaning.

I'm sympathetic to much of what you say here, and probably agree with it as a first approximation. But I dislike the implication (not sure whether intended or not) that a single way of making decisions (calculation vs. something closer to our intuitive decision procedure) is right in general. It seems far more likely to me that there are situations where one or the other, or some combination of procedures, is likely to be most useful; and the interesting questions are about what to use when, rather than what to use always and everywhere.

A lot of this probably comes down to:

Don’t assume – that you have a rich enough picture of yourself, a rich enough picture of the rest of reality, or that your ability to mentally trace through the consequences of actions comes anywhere near the richness of reality’s ability to do so.

Why does everybody think that utilitarianism necessarily means consciously crunching real numbers?

Surely the only point you're making in this long post is not that naïve consequentialism is a bad idea?

consider brainstorming for other goals that you might have ignored, and then attach priorities.

And how exactly does one attach priorities?