The more rational someone is, the less you can trust them. The less rational someone is, the more you can trust them.
I think in this post when you say 'trust' you really mean 'predict'. A trivial counterexample: the more rational someone is, the more I can trust them to be free of errors in their reasoning. And it IS easier to predict a religious zealot staying religious, or predict that a bigot will remain bigoted, than it is to predict a rational agent attempting to maximize their utility (especially if you're an obstacle to their utility).
Is it reasonable to think that people have evolved to be less-than-optimally rational?
Well, yes, if there was some shortcut that gave the mostly-optimal answer, or gave the optimal answer most of the time, and gave it in a significantly faster time than optimal rationality. The common example is, I think, reacting to the presence of a lion. Abject, heart-pounding, run-for-your-life terror is not optimally rational (it generally precludes climbing a tree) but it gives a mostly-optimal answer in a much shorter time than attempting to reason out the optimal course of action.
But what if our irrationalities aren't quick-and-dirty heuristics optimized for speed? What known cognitive biases are even applicable to running away from a lion?
What if some of our cognitive biases are evolved adaptations that make human society work better? It would be pretty surprising to me if this weren't the case!
[Update: and now there's a fifth discussion thread, which you should probably use in preference to this one. Later update: and a sixth -- in the discussion section, which is where these threads are living for now on. Also: tag for HP threads in the main section, and tag for HP threads in the discussion section.]
The third discussion thread is above 500 comments now, just like the others, so it's time for a new one. Predecessors: one, two, three. For anyone who's been on Mars and doesn't know what this is about: it's Eliezer's remarkable Harry Potter fanfic.
Spoiler warning and helpful suggestion (copied from those in the earlier threads):
Spoiler Warning: this thread contains unrot13'd spoilers for Harry Potter and the Methods of Rationality up to the current chapter and for the original Harry Potter series. Please continue to use rot13 for spoilers to other works of fiction, or if you have insider knowledge of future chapters of Harry Potter and the Methods of Rationality.
A suggestion: mention at the top of your comment which chapter you're commenting on, or what chapter you're up to, so that people can understand the context of your comment even after more chapters have been posted. This can also help people avoid reading spoilers for a new chapter before they realize that there is a new chapter.