Partial re-interpretation of: The Curse of Identity
Also related to: Humans Are Not Automatically Strategic, The Affect Heuristic, The Planning Fallacy, The Availability Heuristic, The Conjunction Fallacy, Urges vs. Goals, Your Inner Google, signaling, etc...

What are the best careers for making a lot of money?

Maybe you've thought about this question a lot, and have researched it enough to have a well-formed opinion. But the chances are that even if you hadn't, some sort of an answer popped into your mind right away. Doctors make a lot of money, maybe, or lawyers, or bankers. Rock stars, perhaps.

You probably realize that this is a difficult question. For one, there's the question of who we're talking about. One person's strengths and weaknesses might make them more suited for a particular career path, while for another person, another career is better. Second, the question is not clearly defined. Is a career with a small chance of making it rich and a large chance of remaining poor a better option than a career with a large chance of becoming wealthy but no chance of becoming rich? Third, whoever is asking this question probably does so because they are thinking about what to do with their lives. So you probably don't want to answer on the basis of what career lets you make a lot of money today, but on the basis of which one will do so in the near future. That requires tricky technological and social forecasting, which is quite difficult. And so on.

Yet, despite all of these uncertainties, some sort of an answer probably came to your mind as soon as you heard the question. And if you hadn't considered the question before, your answer probably didn't take any of the above complications into account. It's as if your brain, while generating an answer, never even considered them.

The thing is, it probably didn't.

Daniel Kahneman, in Thinking, Fast and Slow, extensively discusses what I call the Substitution Principle:

If a satisfactory answer to a hard question is not found quickly, System 1 will find a related question that is easier and will answer it. (Kahneman, p. 97)

System 1, if you recall, is the quick, dirty and parallel part of our brains that renders instant judgements, without thinking about them in too much detail. In this case, the actual question that was asked was ”what are the best careers for making a lot of money”. The question that was actually answered was ”what careers have I come to associate with wealth”.

Here are some other examples of substitution that Kahneman gives:

Last Friday, the NYC Less Wrong community held their first Winter Solstice Celebration. Approximately twenty of us gathered for dinner and a night of ritual. We sang songs, told stories, and recited litanies. The night celebrated ancient astronomers, and the work that humanity has done for the past 5000 years. It paid tribute to the harshness of the universe, respecting it as worthy opponent. We explored Lovecraftian mythology, which intersects with our beliefs in interesting ways.

And finally, we looked to the future, vowing to give a gift to tomorrow.

This is the first of 2-3 posts on this subject. In this one, I'm telling a story about what we did and why I wanted to. In the followup(s), I’ll explain the design principles that went into planning such an event, and what we learned from our first execution of it. I’ll also be posting a PDF of a ritual book, similar to the one we read from but with a few changes based on initial, obvious observations.

Why exactly did we do this? Doesn’t this smack of organized religion? Who the hell is Lovecraft and why do we care?

Depending on your background, this may require the bridging of some inferential distance, as well as emotional distance. Bear with me.

I suspect there's a Pons Asinorum of probability between the bettor who thinks that you make money on horse races by betting on the horse you think will win, and the bettor who realizes that you can only make money on horse races if you find horses whose odds seem poorly calibrated relative to superior probabilistic guesses.

There is, I think, a second Pons Asinorum associated with more advanced finance, and it is the concept that markets are an anti-inductive environment.

Let's say you see me flipping a coin.  It is not necessarily a fair coin.  It's a biased coin, and you don't know the bias.  I flip the coin nine times, and the coin comes up "heads" each time.  I flip the coin a tenth time.  What is the probability that it comes up heads?

If you answered "ten-elevenths, by Laplace's Rule of Succession", you are a fine scientist in ordinary environments, but you will lose money in finance.

In finance the correct reply is, "Well... if everyone else also saw the coin coming up heads... then by now the odds are probably back to fifty-fifty."

Recently on Hacker News I saw a commenter insisting that stock prices had nowhere to go but down, because the economy was in such awful shape.  If stock prices have nowhere to go but down, and everyone knows it, then trades won't clear - remember, for every seller there must be a buyer - until prices have gone down far enough that there is once again a possibility of prices going up.

So you can see the bizarreness of someone saying, "Real estate prices have gone up by 10% a year for the last N years, and we've never seen a drop."  This treats the market like it was the mass of an electron or something.  Markets are anti-inductive.  If, historically, real estate prices have always gone up, they will keep rising until they can go down.

Part of the Sequence: The Science of Winning at Life. Co-authored with Minda Myers and Hugh Ristik. Also see: Polyhacking.

When things fell apart between me (Luke) and my first girlfriend, I decided that kind of relationship wasn't ideal for me.

I didn't like the jealous feelings that had arisen within me. I didn't like the desperate, codependent 'madness' that popular love songs celebrate. I had moral objections to the idea of owning somebody else's sexuality, and to the idea of somebody else owning mine. Some of my culture's scripts for what a man-woman relationship should look like didn't fit my own goals very well.

I needed to design romantic relationships that made sense (decision-theoretically) for me, rather than simply falling into whatever relationship model my culture happened to offer. (The ladies of Sex and the City weren't too good with decision theory, but they certainly invested time figuring out which relationship styles worked for them.) For a while, this new approach led me into a series of short-lived flings. After that, I chose 4 months of contented celibacy. After that, polyamory. After that...

Anyway, the results have been wonderful. Rationality and decision theory work for relationships, too!

We humans compartmentalize by default. Brains don't automatically enforce belief propagation, and aren't configured to do so. Cached thoughts and cached selves can remain even after one has applied the lessons of the core sequences to particular parts of one's life. That's why it helps to explicitly examine what happens when you apply rationality to new areas of your life — from disease to goodness to morality. Today, we apply rationality to relationships.

Honest disagreement is often a good sign of progress.

- Gandhi

Now that most communication is remote rather than face-to-face, people are comfortable disagreeing more often. How, then, can we disagree well? If the goal is intellectual progress, those who disagree should aim not for name-calling but for honest counterargument.

To be more specific, we might use a disagreement hierarchy. Below is the hierarchy proposed by Paul Graham (with DH7 added by Black Belt Bayesian).¹

DH0: Name-Calling. The lowest form of disagreement, this ranges from "u r fag!!!" to "He’s just a troll" to "The author is a self-important dilettante."

DH1: Ad Hominem. An ad hominem ('against the man') argument won’t refute the original claim, but it might at least be relevant. If a senator says we should raise the salary of senators, you might reply: "Of course he’d say that; he’s a senator." That might be relevant, but it doesn’t refute the original claim: "If there’s something wrong with the senator’s argument, you should say what it is; and if there isn’t, what difference does it make that he’s a senator?"

DH2: Responding to Tone. At this level we actually respond to the writing rather than the writer, but we're responding to tone rather than substance. For example: "It’s terrible how flippantly the author dimisses theology."

DH3: Contradiction. Graham writes: "In this stage we finally get responses to what was said, rather than how or by whom. The lowest form of response to an argument is simply to state the opposing case, with little or no supporting evidence." For example: "It’s terrible how flippantly the author dismisses theology. Theology is a legitimate inquiry into truth."

DH4: Counterargument. Finally, a form of disagreement that might persuade! Counterargument is "contradiction plus reasoning and/or evidence." Still, counterargument is often directed at a minor point, or turns out to be an example of two people talking past each other, as in the parable about a tree falling in the forest.

DH5: Refutation. In refutation, you quote (or paraphrase) a precise claim or argument by the author and explain why the claim is false, or why the argument doesn’t work. With refutation, you're sure to engage exactly what the author said, and offer a direct counterargument with evidence and reason.

DH6: Refuting the Central Point. Graham writes: "The force of a refutation depends on what you refute. The most powerful form of disagreement is to refute someone’s central point." A refutation of the central point may look like this: "The author’s central point appears to be X. For example, he writes 'blah blah blah.' He also writes 'blah blah.' But this is wrong, because (1) argument one, (2) argument two, and (3) argument three."

DH7: Improve the Argument, then Refute Its Central Point. Black Belt Bayesian writes: "If you’re interested in being on the right side of disputes, you will refute your opponents' arguments. But if you're interested in producing truth, you will fix your opponents' arguments for them. To win, you must fight not only the creature you encounter; you [also] must fight the most horrible thing that can be constructed from its corpse."² Also see: The Least Convenient Possible World.

Having names for biases and fallacies can help us notice and correct them, and having labels for different kinds of disagreement can help us zoom in on the parts of a disagreement that matter.

Warning: sappy personal anecdotes ahead! See also Eliezer's Coming of Age story, SarahC's Reflections on rationality a year out, and Alicorn's Polyhacking.

On January 11, 2007, at age 21, I finally whispered to myself: There is no God.

I felt the world collapse beneath me. I'd been raised to believe that God was necessary for meaning, morality, and purpose. My skin felt cold and my tongue felt like cardboard. This was the beginning of the darkest part of my life, but the seed of my later happiness.

I grew up in Cambridge, Minnesota — a town of 5,000 people and 22 Christian churches (at the time). My father was (and still is) pastor of a small church. My mother volunteered to support Christian missionaries around the world.

I went to church and Bible study every week. I prayed often and earnestly. For 12 years I attended a Christian school that taught Bible classes and creationism. I played in worship bands. As a teenager I made trips to China and England to tell the godless heathens there about Jesus. I witnessed miraculous healings unexplained by medical science.

And I felt the presence of God. Sometimes I would tingle and sweat with the Holy Spirit. Other times I felt led by God to give money to a certain cause, or to pay someone a specific compliment, or to walk to the cross at the front of my church and bow before it during a worship service.

Around age 19 I got depressed. But then I read Dallas Willard’s The Divine Conspiracy, a manual for how to fall in love with God so that following his ways is not a burden but a natural and painless product of loving God. And one day I saw a leaf twirling in the wind and it was so beautiful — like the twirling plastic bag in American Beauty — that I had an epiphany. I realized that everything in nature was a gift from God to me. Grass, lakes, trees, sunsets — all these were gifts of beauty from my Savior to me. That's how I fell in love with God, and he delivered me from my depression.

I moved to Minneapolis for college and was attracted to a Christian group led by Mark van Steenwyk. Mark’s small group of well-educated Jesus-followers are 'missional' Christians: they think that loving and serving others in the way of Jesus is more important than doctrinal truth. That resonated with me, and we lived it out with the poor immigrants of Minneapolis.

About two weeks ago I announced an open competition for LessWrong readers inspired by Robert Axelrod's famous tournaments. The competitors had to submit a strategy which would play an iterated prisoner's dilemma of fixed length: first in the round-robin tournament where the strategy plays a hundred-turn match against each of its competitors exactly once, and second in the evolutionary tournament where the strategies are randomly paired against each other and their gain is translated in number of their copies present in next generation; the strategy with the highest number of copies after generation 100 wins. More details about the rules were described in the announcement. This post summarises the results.

Sometime during the 1740s, the Reverend Thomas Bayes made the ingenious discovery that bears his name but then mysteriously abandoned it. It was rediscovered independently by a different and far more renowned man, Pierre Simon Laplace, who gave it its modern mathematical form and scientific application — and then moved on to other methods. Although Bayes’ rule drew the attention of the greatest statisticians of the twentieth century, some of them vilified both the method and its adherents, crushed it, and declared it dead. Yet at the same time, it solved practical questions that were unanswerable by any other means: the defenders of Captain Dreyfus used it to demonstrate his innocence; insurance actuaries used it to set rates; Alan Turing used it to decode the German Enigma cipher and arguably save the Allies from losing the Second World War; the U.S. Navy used it to search for a missing H-bomb and to locate Soviet subs; RAND Corporation used it to assess the likelihood of a nuclear accident; and Harvard and Chicago researchers used it to verify the authorship of the Federalist Papers. In discovering its value for science, many supporters underwent a near-religious conversion yet had to conceal their use of Bayes’ rule and pretend they employed something else. It was not until the twenty-first century that the method lost its stigma and was widely and enthusiastically embraced.

So begins Sharon McGrayne's fun new book, The Theory That Would Not Die, a popular history of Bayes' Theorem. Instead of reviewing the book, I'll summarize some of its content below. I skip the details and many great stories from the book, for example the (Bayesian) search for a lost submarine that inspired Hunt for Red October. Also see McGrayne's Google Talk here. She will be speaking at the upcoming Singularity Summit, too, which you can register for here (price goes up after August 31st).