I am aware of what it means to be rational.

On the other hand, if I want to eliminate poverty in Obscureistan, but I find out that achieving this won't make me happy, that doesn't make me change my goal at all.

But how do you know that this line of reasoning is not culturally induced and the result of abstract high-order contemplations about rational conduct? My problem is that I perceive rationality to change and introduce terminal goals. The toolkit that is called 'rationality', the rules and heuristics developed to help us to achieve our terminal goals are also altering and deleting them. A stone age hunter-gatherer seems to possess very different values than I do. If he learns about rationality and moral ontology his values will be altered considerably. Rationality was meant to help him achieve his goals, e.g. become a better hunter. Rationality was designed to tell him what he ought to do (instrumental goals) to achieve what he wants to do (terminal goals). Yet what actually happens is that he is told, that he will learn what he ought to want. If an agent becomes more knowledgeable and smarter then this does not leave its goal-reward-system intact if it is not especially designed to be stable. An agent who originally wanted to become a better hunter and feed his tribe would end up wanting to eliminate poverty in Obscureistan. The question is, how much of this new "wanting" is the result of using rationality to achieve terminal goals and how much is a side-effect of using rationality, how much is left of the original values versus the values induced by a feedback loop between the toolkit and its user? Here I think it would be important to ask how humans assign utility, if there exist some sort of intrinsic property that makes agents assign more utility to some experiences and outcomes versus others. We have to discern what we actually want from what we think we ought to want. This might sound contradictory, but I don't think it is. If an agent is facing the Prisoner's dilemma that agent might originally tend to cooperate and only after learning about game theory decide to defect and gain a greater payoff. Was it rational for the agent to learn about game theory, in the sense that it helped the agent to achieve its goal or in the sense that it deleted one of its goals in exchange for a more "valuable" goal? It seems to me that becoming more knowledgeable and smarter is gradually altering our utility functions. But what is it that we are approaching if rationality becomes a purpose in and of itself? If we can be biased, if our map of the territory can be distorted, why can't we be wrong about what we value as well? If that is possible, how can we discover better values? What rationality is doing is to extrapolate our volition to calculate the expected utility of different outcomes. But this might distort or alter what we really value by installing new cognitive toolkits designed to achieve an equilibrium between us and other agents with the same toolkit. This is why I think it might be important to figure out what all high-utility goals have in common. Here happiness is just an example, I am not claiming that happiness is strongly correlated with utility, that happiness is the highest order goal. One might argue that we would choose a world state in which all sentient agents are maximally happy over one where all sentient agents achieved arbitrary goals but are on average not happy about it. But is this true? I don't know. I am just saying that we might want to reconsider what we mean by "utility" and objectify its definition. Otherwise the claim that we don't want to "overcome" a lot of what millions of years of brain evolution have formed is not even wrong because if we are unable to prove some sort of human-goal-stability then what we want is a fact about our cultural and intellectual evolution more than a fact about us, about human nature. Are we using our tools or are the tools using us, are we creating models or are we modeled, are we extrapolating our volition or following our extrapolations?

Incidentally, I also don't believe that equating "procrastination" with "our evolutionary desire to rest" is even remotely correct.

I got that from here. For everything else see this comment, would love to hear your opinion. I am not sure what exactly it is that I am confused about. Thank you.