Consider how, in 51-54, Harry decides to trust Quirrell. No one ought to trust Quirrell, at all. He has some agenda, which he does not let on to. Even if he did describe his complete agenda, you'd never be able to trust that he was telling the truth, because he's so rational and self-controlled that he would be equally able to tell you something almost the truth, except for certain modifications made to make your cooperation more likely.
And few people trust Harry; and with good reason.
The more rational someone is, the less you can trust them. The less rational someone is, the more you can trust them. You can trust a bigot to keep acting bigoted. You can trust a religious zealot to stay true to her faith. You can trust someone who votes the party line without thinking to keep voting the party line.
Whereas, a religious zealot who actually thinks about her religious principles is much less reliable. The Jesuits are a perfect example of this - religious, but prone to thinking about their religion, and thus a neverending source of heresy and controversy within the Church. A politician who actually thinks about the issues might break with his party, or vote differently than the people who elected him expected.
How much does society rely on our irrationality, on our inability to change our minds or avoid signalling our true intent, and our inability to avoid following through on our emotional commitments (revenge, punishment, reward, nepotism)? What's the social cost of rationality? Is it reasonable to think that people have evolved to be less-than-optimally rational?
The more rational someone is, the less you can trust them. The less rational someone is, the more you can trust them.
This is not always the case. In fact, in practice I trust rational people far more than those that are irrational. If I know what a rational person's goals are, know the respective payoffs for each of the various available options then I can reliably predict what the rational person will do. Irrational people can not always be counted on to do what is in their own best interest - this makes them less predictable than they would be if they were rational unless they are an adversary trying specifically to foil your intelligence.
[Update: and now there's a fifth discussion thread, which you should probably use in preference to this one. Later update: and a sixth -- in the discussion section, which is where these threads are living for now on. Also: tag for HP threads in the main section, and tag for HP threads in the discussion section.]
The third discussion thread is above 500 comments now, just like the others, so it's time for a new one. Predecessors: one, two, three. For anyone who's been on Mars and doesn't know what this is about: it's Eliezer's remarkable Harry Potter fanfic.
Spoiler warning and helpful suggestion (copied from those in the earlier threads):
Spoiler Warning: this thread contains unrot13'd spoilers for Harry Potter and the Methods of Rationality up to the current chapter and for the original Harry Potter series. Please continue to use rot13 for spoilers to other works of fiction, or if you have insider knowledge of future chapters of Harry Potter and the Methods of Rationality.
A suggestion: mention at the top of your comment which chapter you're commenting on, or what chapter you're up to, so that people can understand the context of your comment even after more chapters have been posted. This can also help people avoid reading spoilers for a new chapter before they realize that there is a new chapter.