Short version: beyond a certain (very coarse) precision you can't usefully model humans as logical, goal-directed, decision-making agents contaminated by pesky "biases". Goals, decisions and agency are very leaky abstractions, illusions that arise from the mechanical interplay of our many ad-hoc features. Rather than heading off for the sunset, the 99% typical behavior of humans is going around in circles day after day; if this is goal-directed, the goal must be weird indeed. If you want to make predictions about actual human beings, don't talk about their goals, talk about their tendencies.

Far from distressing me, this situation makes me happy. It's great we have so few optimizers around. Real-world strong optimizers, from natural selection to public corporations to paperclippers, look psychopathic and monstrous when viewed through the lens of our tendency-based morality.

For more details see thread above. Or should I compile this stuff into a toplevel post?