I found this to be a good and informative post, nonetheless.
Really? Are you really surprised that people are reluctant to broadcast data that doesn't fit their theory? Have you read any political blogs?
By my model, it takes a pretty unusual person to give anywhere near equal weight to confirming and disconfirming evidence. We're holding Seth Roberts to a very high standard here--one that Gwern himself has not necessarily achieved. Criticizing is easy.
This is a great example of what Frank Adamek talked about in his recent post re: lowering other people's status. The reason folks subconsciously avoid disconfirming evidence is so they can preserve their status. In an ideal world preserving status would be a nonissue and disconfirming evidence would be fine. But then someone like Gwern comes along and snipes someone's status, validating the concern with status that leads to confirmation bias in the first place.
(Stop violating useful social norms Gwern! Punish the norm violator! Just kidding, saying that would make me a hypocrite. I'll assume Gwern posted this in good faith and didn't mean to erode useful social norms.)
So can future articles on individual irrationality please be restricted to people writing about themselves?
To clarify: I'm in support of doing psychological tests on small scales and writing up the results on Less Wrong. I'm not in support of breaking certain ethical injunctions in the process.
If gwern had legitimately gotten a different self-experiment than Seth Roberts, and then the same process had transpired, I'd be entirely in favor of this post. It's an important caveat to self-experimentation that you need to really watch out for confirmation bias, and trust people more if they're willing to publicize negative results as well as positive ones.
But falsify...
Master copy lives on gwern.net