gjm comments on 2012 Survey Results - Less Wrong

80 Post author: Yvain 07 December 2012 09:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (640)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 02 December 2012 04:28:09AM 8 points [-]

Are people who understand quantum mechanics are more likely to believe in Many Worlds? We perform a t-test, checking whether one's probability of the MWI being true depends on whether or not one can solve the Schrodinger Equation. People who could solve the equation had on average a 54.3% probability of MWI, compared to 51.3% in those who could not. The p-value is 0.26; there is a 26% probability this occurs by chance. Therefore, we fail to establish that people's probability of MWI varies with understanding of quantum mechanics.

Some Bayesian analysis using the BEST MCMC library for normal two-group comparisons:

R> lw <- read.csv("lw-2012.csv")
R> R> lwm <- subset(lw, !(" " == as.character(SchrodingerEquation)))
R> lwm <- subset(lwm, !is.na(as.integer(as.character(PManyWorlds))))
R> mwiyes <- as.integer(as.character(subset(lwm, SchrodingerEquation == "Yes")$PManyWorlds))
R> mwino <- as.integer(as.character(subset(lwm, SchrodingerEquation == "No")$PManyWorlds))
R> R> source("BEST.R")
R> mcmcChain = BESTmcmc(mwino, mwiyes)
R> show(postInfo)
SUMMARY.INFO
PARAMETER mean median mode HDIlow HDIhigh pcgtZero
mu1 51.1693 51.1675 51.1964 48.8152 53.48281 NA
mu2 55.5708 55.5647 55.4871 50.2376 61.03888 NA
muDiff -4.4016 -4.4010 -4.1635 -10.1932 1.52931 7.154
sigma1 30.5558 30.5355 30.4243 28.9332 32.24333 NA
sigma2 32.8187 32.7136 32.5672 29.0621 36.73604 NA
sigmaDiff -2.2629 -2.1800 -2.1232 -6.4056 1.97323 14.226
nu 106.4690 98.6244 84.7466 36.2142 194.12061 NA
nuLog10 1.9929 1.9940 1.9853 1.6566 2.33430 NA
effSz -0.1389 -0.1388 -0.1323 -0.3208 0.04864 7.154

<code>postInfo = BESTplot(mwino, mwiyes, mcmcChain)</code>

(Full size image.)

The results are interesting and not quite the same as a t-test:

  1. we get estimates of standard deviations, among other things, for free - they look pretty different and there's an 85.8% chance the deviations of the Schrodinger-knowers and not-knowers are different on MWI, suggesting to me a polarizing effect where the more you know, the more extreme your view either for or against, which seems reasonable since the more information you have, the less should your uncertainty be.
  2. the difference in means estimate is sharper than the t-test: Yvain's t-test gave a p-value of 0.26 if the null hypothesis were true (he makes the classic error when he says "there is a 26% probability this occurs by chance" - no, there's a 26% chance this happened by chance if one assumes the null hypothesis is true, which says absolutely nothing about whether this happened by chance).

    We, however, by using Bayesian techniques can say that given the difference in mean beliefs: there is a 7.2% chance that the null hypothesis (equal belief) or the opposite hypothesis (lower belief) is true in this sample.

    We also get an effect-size for free from the difference in means. -0.132 (mode) isn't too impressive, but it's there.

However, both BEST and the t-test are normal tests. The histograms look like the data may be a bimodal distribution: a hump of skeptics at 10%, a hump of believers in the 70%s - and the weirdly low 40s in both groups is just a low point in both? I don't know how much of an issue this is.

Comment author: gjm 06 December 2012 11:23:37PM 3 points [-]

he makes the classic error

For what it's worth, I interpreted his "there is a 26% probability this occurs by chance" exactly as "if there's no real difference, there's a 26% probability of getting this sort of result by chance alone" or equivalently "conditional on the null hypothesis Pr(something at least this good) = 26%". I'd expect that someone who was making the classic error would have said "there is a 26% probability this occurred by chance".