mfb comments on Frequentist Statistics are Frequently Subjective - Less Wrong

59 Post author: Eliezer_Yudkowsky 04 December 2009 08:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (81)

You are viewing a single comment's thread.

Comment author: mfb 11 February 2012 04:46:23PM 0 points [-]

Although this post is old, I really dislike the description of confidence levels. An appropriate confidence level is made out of the points with the "best" likelihoods: Every point in the point has a higher likelihood than every point outside. Therefore, an output of '["Cheesecake"-"Cheddar"]' is a clear sign of bad science. A good confidence level is just a summary of likelihood ratios. And for symmetric likelihood profiles (at least near the minimum), it gives a very good idea of the function in the interesting range.

And most problems with p-values can be reduced by choosing the study and analysis technique in advance (or with blinded data, for the analysis part). If you can flip the coin again, make a small number of hypotheses and test them. Writing down good prior probabilities after looking at coin tosses does not really work with human brains, and writing down 100000 different options (including things like "the coin gives always the repeating pattern TTTTTH") before doing coin tosses is even worse.