= 27d567bc2d48d9f40e9ccc20ea370b15)
Eugine_Nier comments on Rationality Quotes November 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (388)
Nassim Taleb
What's the difference between "based on computation of the odds" and "based on some model"?
Taleb is doing some handwaving here.
"Some model" in this context is just the assumption of a specific probability distribution. So if, for example, you believe that the observation values are normally distributed with the mean of 0 and the standard deviation of 1, the chance of seeing a value greater than 3 (a "three-sigma value") is 0.13%. The chance of seeing a value greater than 6 (a "six-sigma value") is 9.87e-10. E.g. if your observations are financial daily returns, you effectively should never ever see a six-sigma value. The issue is that in practice you do see such values, pretty often, too.
The problem with Taleb's statement is that to estimate the probabilities of seeing certain values in the future necessarily requires some model, even if implicit. Without one you can not do the "computation of the odds" unless you are happy with the conclusion that the probability to see a value you've never seen before is zero.
Taleb's criticism of the default assumption of normality in much of financial analysis is well-founded. But when he starts to rail against models and assumptions in general, he's being silly.
So, this.
Well, yeah, sure. Yvain wrote it up nicely, but the main point -- that what the model says and how much do you trust the model itself are quite different things -- is not complicated.
To get back to Taleb, he is correct in pointing out that estimating what the tails of an empirical distribution look like is very hard because you don't see a lot of (or, sometimes, any) data from these tails. But if you need an estimate you need an estimate and saying "no model is good enough" isn't very useful.
But surely Taleb isn't saying "no model is good enough." He explicitly advocates greater care in model-building and greater awareness of the risks of error, not people throwing up their hands and giving up. He says at the end:
Actually, yes, he is. He is not terribly consistent, but when he goes into his "philosopher" mode he rants against all models.
In fact, his trademark concept of a black swan is precisely what no model can predict.
Maybe it isn't the clearest way of describing it, but it seems that by "computation of odds" he means using at least some observation of frequencies, and is contrasting this with computing the probability of events for which there have as yet been no occurrences, so no observation of frequency has been possible.
No two real world events are exactly identical. You always need some model to generalize and say the ones you observed are like the ones you predict in some relevant way to reuse the observed frequency in your prediction. Without a model all you can say is that if the circumstances were to repeat exactly, then so would the outcome. And that just isn't very useful.
Hmm. But, if you multiply "once in every ten thousand years" by all the different kinds of things that could be said to happen once every ten thousand years, don't you get something closer to "many times a day"?
The computation is not relevant, because when you make a prediction that, say, some excursion in the stock market will happen only once in ten thousand years, you are making a prediction about that specific thing, not ten thousand things. It will be a thing you have never seen, because if you had seen it happen, you could not claim it would only happen once in ten thousand years—the observation would be a refutation of that claim. Since you have not seen it, you are deriving it from a theory, and moreover a theory applied at an extreme it has never been tested at. For such a prediction to be reliable, you need to know that your theory actually grasps the basic mechanism of the phenomenon, so that the observations that you have been able to make justify placing confidence in its extremes. This is a very high bar to reach. Here are a few examples of theories where extremes turned out to differ from reality:
Newtonian gravity --> precession of Mercury
Ideal gas laws --> non-ideal gases
Daltonian atomic theory --> multiple isotopes of the same element
The computation is directly relevant, given that Taleb is talking about how often he sees "should only happen every N years" in newspapers and faculty news. Doesn't he realise how many things newspapers report on? Astronomy faculties are pretty good for this too, since they watch ridiculous numbers of stars at once.
You can't just ignore the multiple comparisons problem by saying you're only making a prediction about "one specific thing". What about all the other predictions about the stock market you made, that you didn't notice because they turned out to be boringly correct?
Intuition pump: my theory says that the sequence of coinflips HHHTHHTHTT-THHTHHHTT-TTHTHTTTTH-HTTTHTHHHTT, which I just observed, should happen about once every 7 million years.
Intuition pump: if I choose an interesting sequence of coinflips in advance, I will never see it actually happen if the coinflips are honest. There aren't enough interesting sequences of 40 coinflips to ever see one. Most of them look completely random, and in terms of Kolmogorov complexity, most of them are: they cannot be described much more compactly than by just writing them out.
Now, we have a good enough understanding of the dynamics of tossed coins to be fairly confident that only deliberate artifice would produce a sequence of, say, 40 consecutive heads. We do not have such an understanding of the sort of things that appear in the news as "should only happen every N years".
Feynman on the same theme.
Every sequence of 40 coin flips is interesting. Proof: Make a 1 to 1 relation on the sequence of 40 coin flips and a subset of the natural numbers, by making H=1 and T=0 and reading the sequence as a binary representation. Proceed by showing that every natural number is interesting.
If you made ten million predictions of things that happen once every ten thousand years, and about three times a day one of them happens, then it would be sensible to conclude that a given one happens about once every ten thousand years. Most people don't do this, however. If someone managed to make ten million such predictions, they'd likely end up with a lot more than three of them happening a day.