This post has its flaws, as has been pointed out, but to add the required nuance would make a book (or at least a LW sequence) out of what is currently a good and provocative post.

The nuances I think are vital:

1. The explicit consequence-representing/world-modeling parts of our minds exist alongside direct "function calls" like reflexes.
2. The decision-theory framework only imperfectly describes even that CR/WM part of our minds.
3. It's quite possible (and indeed virtually certain) that changing the payoff matrix has been done by evolution in certain cases, in addition to implementing a SAMEL-friendly decision theory.
4. It is clear that SAMELs are not generally consciously accepted by average human beings as reasons for action, to virtually any extent. The SAMEL seems really strange at first, and is accepted as valid (let alone as pervasive) only after long reflection. It's not that we just don't have the understanding of what we do, it seems that we have a mental block against that part of the understanding.