gwern comments on 2012 Survey Results - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (640)
Some Bayesian analysis using the BEST MCMC library for normal two-group comparisons:
(Full size image.)
The results are interesting and not quite the same as a t-test:
the difference in means estimate is sharper than the t-test: Yvain's t-test gave a p-value of 0.26 if the null hypothesis were true (he makes the classic error when he says "there is a 26% probability this occurs by chance" - no, there's a 26% chance this happened by chance if one assumes the null hypothesis is true, which says absolutely nothing about whether this happened by chance).
We, however, by using Bayesian techniques can say that given the difference in mean beliefs: there is a 7.2% chance that the null hypothesis (equal belief) or the opposite hypothesis (lower belief) is true in this sample.
We also get an effect-size for free from the difference in means. -0.132 (mode) isn't too impressive, but it's there.
However, both BEST and the t-test are normal tests. The histograms look like the data may be a bimodal distribution: a hump of skeptics at 10%, a hump of believers in the 70%s - and the weirdly low 40s in both groups is just a low point in both? I don't know how much of an issue this is.
For what it's worth, I interpreted his "there is a 26% probability this occurs by chance" exactly as "if there's no real difference, there's a 26% probability of getting this sort of result by chance alone" or equivalently "conditional on the null hypothesis Pr(something at least this good) = 26%". I'd expect that someone who was making the classic error would have said "there is a 26% probability this occurred by chance".