If you're interested in learning rationality, where should you start? Remember, instrumental rationality is about making decisions that get you what you want -- surely there are some lessons that will help you more than others.
You might start with the most famous ones, which tend to be the ones popularized by Kahneman and Tversky. But K&T were academics. They weren't trying to help people be more rational, they were trying to prove to other academics that people were irrational. The result is that they focused not on the most important biases, but the ones that were easiest to prove.
Take their famous anchoring experiment, in which they showed the spin of a roulette wheel affected people's estimates about African countries. The idea wasn't that roulette wheels causing biased estimates was a huge social problem; it was that no academic could possibly argue that this behavior was somehow rational. They thereby scored a decisive blow for psychology against economists claiming we're just rational maximizers.
Most academic work on irrationality has followed in K&T's footsteps. And, in turn, much of the stuff done by LW and CFAR has followed in the footsteps of this academic work. So it's not hard to believe that LW types are good at avoiding these biases and thus do well on the psychology tests for them. (Indeed, many of the questions on these tests for rationality come straight from K&T experiments!)
But if you look at the average person and ask why they aren't getting what they want, very rarely do you conclude their biggest problem is that they're suffering from anchoring, framing effects, the planning fallacy, commitment bias, or any of the other stuff in the sequences. Usually their biggest problems are far more quotidian and commonsensical.
Take Eliezer. Surely he wanted SIAI to be a well-functioning organization. And he's admitted that lukeprog has done more to achieve that goal of his than he has. Why is lukeprog so much better at getting what Eliezer wants than Eliezer is? It's surely not because lukeprog is so much better at avoiding Sequence-style cognitive biases! lukeprog readily admits that he's constantly learning new rationality techniques from Eliezer.
No, it's because lukeprog did what seems like common sense: he bought a copy of Nonprofits for Dummies and did what it recommends. As lukeprog himself says, it wasn't lack of intelligence or resources or akrasia that kept Eliezer from doing these things, "it was a gap in general rationality."
So if you're interested in closing the gap, it seems like the skills to prioritize aren't things like commitment effect and the sunk cost fallacy, but stuff like "figure out what your goals really are", "look at your situation objectively and list the biggest problems", "when you're trying something new and risky, read the For Dummies book about it first", etc. For lack of better terminology, let's call the K&T stuff "cognitive biases" and this stuff "practical biases" (even though it's all obviously both practical and cognitive and biases is kind of a negative way of looking at it).
What are the best things you've found on tackling these "practical biases"? Post your suggestions in the comments.
Eliezer's comment doesn't say he tried to apply the lessons in Nonprofit Kit for Dummies, though some of it he clearly did — e.g. filing the necessary paperwork to launch a 501c3!
Anyway, reading a how-to book doesn't help much unless you actually do what the book recommends. That's why it's such an important intervention to figure out How To Actually Do The Stuff You Know You Should Be Doing — also known as How to Beat Procrastination.
But the anti-akrasia techniques we've uncovered so far don't work for everyone, and there are other factors at play. For example, since a young age Eliezer has become cognitively exhausted rather quickly. He has spent years trying different things (diet, exercise, context changes, vitamins, etc.) but still hasn't found an intervention that lets him do cognitive work for as long as I can. (Luckily, the value of an hour of cognitive work from Eliezer is much higher than the value of an hour of cognitive work from me.)
Also, there was no time in history when it made sense for Eliezer Yudkowsky to spend his time doing Nonprofit Kit for Dummies stuff. (But it would have made sense, I think, for Eliezer to try harder to find someone who could do non-profit management better, or to try harder to find someone who could execute that search more effectively. This is the kind of thing I meant by mentioning a potential "gap in general rationality.")
P.S. Eliezer's memory of reading Nonprofit Kit for Dummies "before starting the Singularity Institute in 2000" must be mistaken. The first edition of Nonprofit Kit for Dummies wasn't published until 2001.
I'm confused. You seem to be suggesting that procrastination is one of the main "biases" we need to overcome (or, as I would put it, that the ability to beat procrastination is one of the main "practical skills" we need to develop). But aaronsw disagrees that this is what you yourself... (read more)