Recently there has been a couple of articles in the discussion page asking whether rationalists should do action A. Now such questions are not uninteresting, but by saying "rationalist" they are poorly phrased.
The rational decision at any time is the decision, given a human with a specific utility function B, and information C, should make to maximise B, given their knowledge (and knowledge about their knowledge) of C. It's not a decision a rationalist should make, it's a decision any human should make. If Omega popped into existence and carefully explained why action A is the best thing for this human to do given their function B, and their information C, then said human should agree.
The important question is not what a rationalist should do, but what your utility function and current information is. This is a more difficult question. Humans are often wrong about what they want in the long term, and it's questionable how much we should value happiness now over happiness in the future (in particular, I suspect current and future me might disagree on this point). Quantifying our current information is also rather hard- we are going to make bad probability estimates, if we can make them at all, which lead us into incorrect decisions just because we haven't considered the evidence carefully enough.
Why is this an important semantic difference? Well it's important for the cause of refining rationality that we don't get caught with associating the notion of rationality with certain goals. Some rationalists believe that they want to save the world, and the best way to do it is by creating friendly AI. This is because they have certain utility functions, and certain beliefs about the probabilities of the singularity. Not all rationalists have these utility functions. Some just want to have a happy home life, meet someone nice, and raise a family. These are different goals, and they can be helped by rationality, because rationality IS the art of winning. Being able to clearly state ones goals and work out the best way to acheieve them is useful pretty much no matter what those goals are. (pretty much to prevent silly examples here!)
When I was in high school, I wrote and ran a prisoner's dilemma simulation where strategies reproduced themselves via this mechanism. After every cell played several rounds against its neighbors, each examined itself and its neighbors to see how many points were accumulated, then either mutated randomly or copied its most successful neighbor.
I was trying to experiment in the fashion of vaguely described other simulations I'd read of, and maybe replicate their interesting result: reportedly initial random strategies were soon beaten by always-defect, which would then eventually be beaten out by tit-for-tat, which would then be beaten by always-cooperate, which would in turn be beaten when always-defect reappeared. Psychological/sociological/historical analogies are an interesting exercise for the reader.
But what did I get, instead? Overwhelming victory of a stategy I eventually called "gangster". IIRC it was something like "start with a random high probability of cooperating, then if your opponent cooperates, you start always defecting, but if your opponent defects you start always cooperating".
Sounds like a pretty awful strategy, right? And overall, it was: its resulting scores at each iteration were a sea of always-cooperated-against-defectors losers, punctuated by dots of lucky always-defected-against-many-cooperators winners. But the losers and the winners were using the same strategy! And because each new generation looked at that strategy's great peak performance rather than it's lousy average performance, there was no likelihood of switching to anything better.
Here I'll make some of the sociological analogies explicit: looking at people who live the kind of life you want to live is a lousy way to pick a life path. It's how gangsters are born, because every little hoodlum imagines themselves as one of the rich dealers' dealers' dealers at the top of the pyramid, not as one of the bottom rung dealers risking jail and death for near minimum wage. It's how kids waste their time aiming at star entertainer and athlete careers, because they all imagine themselves as part of the 99.9th percentile millionaire superstars rather than as one of the mere 99th percentile B-listers or the 90th percentile waitstaff. It's how people waste their salaries gambling, even - who doesn't want to live a life as a multimillionaire who didn't have to work for any of it? Other people did it, and all we have to do is follow their path...
This is a nitpicking digression, but I think it's an important nitpick. "Pick a life path whose average results you prefer" is a great metastrategy, but following it means examining the entire lives of the 50th percentile schlubs. Instead emulating your "heroes" as chosen based on the peak of their fame is just common sense, which commonly fails.
Post this as a top level post.