Jonathan_Graehl comments on Existential Risk and Public Relations - Less Wrong

36 Post author: multifoliaterose 15 August 2010 07:16AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (613)

You are viewing a single comment's thread. Show more comments above.

Comment author: Jonathan_Graehl 16 August 2010 11:23:28PM *  0 points [-]

Good comment.

However,

arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made

This was hard to parse. I would have named "p-value" directly. My understanding is that a stated "p-value" will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.

Comment author: FAWS 16 August 2010 11:56:56PM 1 point [-]

I should have said "a set of observations" and "sets of observations". I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be "confirmed" with statistically significant relations.

Comment author: Jonathan_Graehl 17 August 2010 12:12:03AM 1 point [-]

Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.