philh comments on This is why we can't have social science - Less Wrong

36 Post author: Costanza 13 July 2014 09:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (82)

You are viewing a single comment's thread. Show more comments above.

Comment author: Azathoth123 13 July 2014 09:55:48PM *  45 points [-]

The amusing thing is that Mitchel's argument proves much more than he wants it to prove.

Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them.

Notice that the above argument applies just as well to the original experiment being replicated.

Comment author: philh 14 July 2014 01:22:19PM 4 points [-]

My interpretation is that he's saying false positives are unlikely but false negatives are common. If that's the case, then getting a positive result should greatly affect your beliefs; but a negative replication shouldn't shift them much.

(I can't actually justify this interpretation by reading the text, but it does make the text seem fairly reasonable.)