I can see why you'd criticize someone for saying "the problem is that the setting wasn't formal enough" but that's not exactly what I was getting at. What I was getting at is that there's a limit to how much thinking that one can do in a day, everyone's limit is different, and a lot of people do things to ration their brainpower so they avoid running out of it. This comment on mental stamina explains more.

My point was, more clearly worded: It would be a very rare person who possesses enough mental stamina to be rational in literally every single situation. That's a wonderful ideal, but the reality is that most people are going to ration brainpower. If your expectation is that rationalists should never ration brainpower and should be rational constantly, this is an unrealistic expectation. A more realistic expectation is that people should identify the things they need to think extra hard about, and correctly use rational thinking skills at those times. Therefore, testing for the skills when they're trying is probably the only way to detect a difference. There are inevitably going to be times when they're not trying very hard, and if you catch them at one of those times, well, you're not going to see rational thinking skills. It may be that some of these things can be ingrained in ways that don't use up a person's mental stamina, but to expect that rationality can be learned in such a way that it is applied constantly strikes me as an unreasoned assumption.

Now I wonder if the entire difference between the control groups results and LessWrong's results was that Yvain asked the control group only one question, whereas LessWrong had answered 14 pages of questions prior to that.