Baruta07 comments on Beautiful Probability - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (109)
Caledonian,
I agree that the two cases are not precisely the same. I also agree that they are not, as a matter of degree, very close. But it seems to me that stopping at a desired result is implicitly the same as "throwing out" other possible results, if the desired result is one of the several results possible in the range of all feasible "n"s. In other words, what I meant by my "more concrete" example is that researcher 2's experiment is properly a member of the set of all possible type-2 experiments (all of which will produce 60%+), while researcher 1's experiment is one of the set of all possible type-1 experiments which may well produce a different number.
The method of selecting "n" matters. Look at Bertrand's paradox for a mathematical example: http://en.wikipedia.org/wiki/Bertrand%27s_paradox_%28probability%29 P(60% cured|n=100) is not the same thing as P(n=100|60% cured).
On the other hand, it might be that Mr. Yudkowsky mean us to read r=70 as implying that n=100 was the minimum n. In that case, provided we know researcher 2 didn't throw out another n=100 data set that he didn't like, I can see why the examples are equivalent; only once researcher 2 expands the data set are we to be suspicious. A good heuristic, then, might be in such cases to only look at the first n = "minimum number of data points" cases. This could save otherwise useless data.
I had assumed from the description of researcher 2's motives (as I think others did) that he performed the minimum number of tests to get the desired result, and that r=70/100 was offset by the control group.