janos comments on "Friends do not let friends compute p values." - Less Wrong

7 Post author: lukeprog 09 September 2011 07:10AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (4)

You are viewing a single comment's thread. Show more comments above.

Comment author: janos 09 September 2011 06:51:07PM *  1 point [-]

You can comfortably do Bayesian model comparison here; have priors for µcon, µamn, and µsim, and let µpat be either µamn (under hypothesis Hamn) or µsim (under hypothesis Hsim), and let Hamn and Hsim be mutually exclusive. Then integrating out µcon, µamn, and µsim, you get a marginal odds-ratio for Hamn vs Hsim, which tells you how to update.

The standard frequentist method being discussed is nested hypothesis testing, where you want to test null hypothesis H0 with alternative hypothesis H1, and H0 is supposed to be nested inside H1. For instance you could easily test null hypothesis µcon >= µamn >= µpat = µsim against µcon >= µamn >= µpat >= µsim. However, for testing non-nested hypotheses, the methodology is weaker, or at least less standard.