Huh— a Google search returns muddled results. I had understood first-order (instrumental) rationality to mean something like causal decision theory: that given a utility function, you extrapolate out the probable consequences of your immediate options and maximize the expected utility. The problem with this is that it doesn't take into account the problems with being modeled by others, and thus leaves you open to being exploited (Newcomblike problems, Chicken) or losing out in other ways (known-duration Prisoner's Dilemma).

I was also taking for granted what I assumed to be the setup with the revenge scenario: that the act of revenge would be a significant net loss to you (by your utility function) as well as to your target. (E.g. you're the President, and the Russians just nuked New York but promised to stop there if you don't retaliate; do you launch your nukes at Russia?)

Phil's right that a known irrational disposition towards revenge (which evolved in us for this reason) could have deterred the Russians from nuking NYC in the first place, whereas they knew they could get away with it if they knew you're a causal decision theorist. But the form of decision process I'm considering (optimizing over strategies, not actions, while taking into account others' likely decision algorithms given a known strategy for me) also knowably avenges New York, and thus deters the Russians.

EDIT: First paragraph was a reply to Vladimir's un-edited comment, in which he also asked what definition of first-order rationality I meant.