Swimming in Reasons
To a rationalist, certain phrases smell bad. Rotten. A bit fishy. It's not that they're actively dangerous, or that they don't occur when all is well; but they're relatively prone to emerging from certain kinds of thought processes that have gone bad.
One such phrase is for many reasons. For example, many reasons all saying you should eat some food, or vote for some candidate.
To see why, let's first recapitulate how rational updating works. Beliefs (in the sense of probabilities for propositions) ought to bob around in the stream of evidence as a random walk without trend. When, in contrast, you can see a belief try to swim somewhere, right under your nose, that's fishy. (Rotten fish don't really swim, so here the analogy breaks down. Sorry.) As a Less Wrong reader, you're smarter than a fish. If the fish is going where it's going in order to flee some past error, you can jump ahead of it. If the fish is itself in error, you can refuse to follow. The mathematical formulation of these claims is clearer than the ichthyological formulation, and can be found under conservation of expected evidence.
More generally, according to the law of iterated expectations, it's not just your probabilities that should be free of trends, but your expectation of any variable. Conservation of expected evidence is just the special case where a variable can be 1 (if some proposition is true) or 0 (if it's false); the expectation of such a variable is just the probability that the proposition is true.
So let's look at the case where the variable you're estimating is an action's utility. We'll define a reason to take the action as any info that raises your expectation, and the strength of the reason as the amount by which it does so. The strength of the next reason, conditional on all previous reasons, should be distributed with expectation zero.
Maybe the distribution of reasons is symmetrical: for example, if somehow you know all reasons are equally strong in absolute value, reasons for and against must be equally common, or they'd cause a predictable trend. Under this assumption, the number of reasons in favor will follow a binomial distribution with p=.5. Mostly, the values here will not be too extreme, especially for large numbers of reasons. When there are ten reasons in favor, there are usually at least a few against.
But what if that doesn't happen? What if ten pieces of info in a row all favor the action you're considering?
= 783df68a0f980790206b9ea87794c5b6)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)