I'm a big fan of Ken Binmore, and this quote captures a lot of my dissatisfaction with LW's directions of inquiry. For example, it's more or less taken for granted here that future superintelligent AIs should cooperate on the Prisoner's Dilemma, so some of us set out to create a general theory of "superintelligent AIs" (including ones built by aliens, etc.) that would give us the answer we like.
Would it be correct to say you mean "should" in the wishful thinking sense of "we really want this outcome," rather than something normative or probabilistic?
This is our monthly thread for collecting these little gems and pearls of wisdom, rationality-related quotes you've seen recently, or had stored in your quotesfile for ages, and which might be handy to link to in one of our discussions.