Although instrumental rationality is an interesting category, I tend to view it as ultimately boiling down to epistemic rationality. For example, I reason that A leads to B. I wanted B but I didn't want A, and now my motivations start traveling up and down the A -> B causal chain until I reach equilibrium. Or for another example, I notice that if I choose C for reason R, my rational game-partner will likewise choose C for reason R, because of some symmetry in our properties as agents. Now I need to compare the outcome of choices {C, C} to other possibilities, but I can rule out {C, D}, say.
My attraction to various options will change in response to learning these facts. But the role of rationality seems to end with arriving at and facing the facts.
No? Or, beside the point? (But if beside the point, still an interesting new point, I reckon.)
The trouble is that there is nothing in epistemic rationality that corresponds to "motivations" or "goals" or anything like that. Epistemic rationality can tell you that pushing a button will lead to puppies not being tortured, and not pushing it will lead to puppies being tortured, but unless you have an additional system that incorporates desires for puppies to not be tortured, as well as a system for achieving those desires, that's all you can do with epistemic rationality.
Today's post, What Do We Mean By "Rationality"? was originally published on 16 March 2009. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was 3 Levels of Rationality Verification, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.