"I can't explain why you are wrong. But, honestly, I can't fully explain why you are right either. Until I can, I'm going to go with the trusty old combination of tradition and gut instinct."
When I understand a valid argument, I do believe it. I don't have a choice in the matter. I'm not sure if anyone has a choice in the matter, although some people may be better at ignoring that the inner voice that requires you to act in accordance with your beliefs.
But I think there is a big difference between being understanding and being unable to counter. Accepting arguments which you are merely unable to counter opens you up to all kinds of manipulation.
I don't think I'm overselling myself to think that I could wield questions accurately enough to knock down false arguments given by someone who is significantly smarter than me, so long as they aren't allowed to fabricate evidence.
The key is that one must recognize when one has not yet understood something. I think most of the posts on lesswrong are devoted to honing this very skill, even if it is not explicitly mentioned. I'd go so far as to say that a rationalist's self evaluation is essentially equivalent to the extent to which she trusts herself to accurately assign certainty to statements.
I suppose I'm saying the same things as Yvain in different words... here is my phrasing, which I think is better: be careful about accepting beliefs. But please do take your beliefs seriously.
Yvain's blog: Epistemic learned helplessness.
A friend in business recently complained about his hiring pool, saying that he couldn't find people with the basic skill of believing arguments. That is, if you have a valid argument for something, then you should accept the conclusion. Even if the conclusion is unpopular, or inconvenient, or you don't like it. He told me a good portion of the point of CfAR was to either find or create people who would believe something after it had been proven to them.
And I nodded my head, because it sounded reasonable enough, and it wasn't until a few hours later that I thought about it again and went "Wait, no, that would be the worst idea ever."
I don't think I'm overselling myself too much to expect that I could argue circles around the average high school dropout. Like I mean that on almost any topic, given almost any position, I could totally demolish her and make her look like an idiot. Reduce her to some form of "Look, everything you say fits together and I can't explain why you're wrong, I just know you are!" Or, more plausibly, "Shut up I don't want to talk about this!"