lessdazed comments on "Friends do not let friends compute p values." - Less Wrong

7 Post author: lukeprog 09 September 2011 07:10AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (4)

You are viewing a single comment's thread.

Comment author: lessdazed 09 September 2011 10:09:53AM *  4 points [-]

To test whether DID-patients were really affected by interidentity amnesia or whether they were simulating their amnesia, the authors assessed the performance of four groups of subjects on a multiple-choice recognition test. The dependent measure was the number of correct responses. The first group were the DID-patients, the second group were Controls, the third group were controls instructed to simulate interidentity amnesia (Simulators), and the fourth group were controls who had never seen the study list and were therefore True amnesiacs.

...

For instance, consider again the case of the Huntjens et al. study on DID discussed in Section 9.2.6 and throughout this book. For the data from the study, hypothesis H1a states that the mean recognition scores µ for DID-patients and True amnesiacs are the same and that their scores are higher than those of the Simulators: µcon > {µamn = µpat} > µsim, whereas hypothesis H1b states that the mean recognition scores µ for DID-patients and Simulators are the same and that their scores are lower than those of the True amnesiacs: µcon > µamn > {µpat = µsim}. Within the frequentist paradigm, a comparison of these models is problematical. Within the Bayesian paradigm, however, the comparison is natural and elegant

What does this mean? How does nesting work? How does frequentism fail? How does Bayesianism succeed? I do not understand the example at all.

Comment author: janos 09 September 2011 06:51:07PM *  1 point [-]

You can comfortably do Bayesian model comparison here; have priors for µcon, µamn, and µsim, and let µpat be either µamn (under hypothesis Hamn) or µsim (under hypothesis Hsim), and let Hamn and Hsim be mutually exclusive. Then integrating out µcon, µamn, and µsim, you get a marginal odds-ratio for Hamn vs Hsim, which tells you how to update.

The standard frequentist method being discussed is nested hypothesis testing, where you want to test null hypothesis H0 with alternative hypothesis H1, and H0 is supposed to be nested inside H1. For instance you could easily test null hypothesis µcon >= µamn >= µpat = µsim against µcon >= µamn >= µpat >= µsim. However, for testing non-nested hypotheses, the methodology is weaker, or at least less standard.