I liked the initial discussion, because broad heuristics are good for quickly evaluating things, but I think the second example really falls down. A poorly designed study shouldn't be able to affect your odds as much as a well designed study, which is basically what his scoring system implies. He goes from 1/10 odds to 1/160 odds based on a study design which should provide very little evidence. One could argue that a poorly designed study finding a small effect should lower your odds slightly (because of publication bias, for example), or that it should raise your odds slightly because there was at least a small effect, but I find it hard to believe that it could decrease your odds substantially. Suppose it were something you felt was extremely likely (perhaps because of previous medium-quality studies), and you found an extremely poorly designed study that supported the conclusion. His reasoning would suggest that you decrease your odds from, say 4/1 to 1/4 based on the poorly designed study!
Yeah. This is an example where using the actual formula is helpful rather than just speaking heuristically. It's actually somewhat difficult to translate from the author's hand-wavy model to the real Bayes' Theorem (and it would be totally opaque to someone who hadn't seen Bayes before).
"Study support for headline" is supposed to be the Bayes factor P(study supports headline | headline is true) / P(study supports headline | headline is false). (Well actually, everything is also conditioned on you hearing about the study.) If you actually think ab...
http://fivethirtyeight.com/features/a-formula-for-decoding-health-news/
Their actual implementation is rather hand wavy, but I think getting people pointed in the right direction is a lot more important than the particulars. The high visibility of FiveThirtyEight makes me think this is a big win for sanity. I'm curious as to what others think. Which parts were done well? What might have been done differently?