I wonder if Yvain is just making a descriptive, rather than normative, conclusion, i.e., that "preferences" is not a good way to model how humans actually behave.

(If the conclusion is meant to be descriptive, I would reply that once we have powerful tools for self modification, at least some humans will actually self modify into being expected utility maximizers, or whatever the correct decision theory is, so "preferences" will be a good way to model how (some) humans actually behave. And if mind-copying becomes possible and evolution by natural selection continues, there will be strong selection pressure away from reinforcement learning agents, because they do not do well under a mind-copying-enabled environment, compared to, say, UDT agents. Reinforcement learning agents only care about their own future rewards, but evolution favors agents that care about their copy-siblings equally.)

I wish Yvain had telegraphed his overall conclusions for the sequence ahead of time, because that would help to immediately clarify this and any other ambiguities that might arise. If it weren't for your comment, I probably wouldn't have replied to this post, due to the ambiguity. (I could ask for clarification, but that involves delaying the reward of knowing that I've made a substantive and relevant point. Or I could respond to both possible meanings, but I'd have to expend twice the effort for the same reward.)