A person who does your will because it makes them happy to do so, or because they are irrationally biased to do so, is more reliable than one who does your will as long as his calculus tells him to.

Not if the latter explicitly exhibits the form of that calculus; then you can extrapolate their future decisions yourself, more easily than you can extrapolate the decisions of the former. Higher-order rationality includes finding a decision algorithm which can't be exploited if known in this manner.

Of course, actually calculating and reliably acting accordingly is a high standard for unmodified humans, and it's a meaningful question whether incremental progress toward that ideal will lead to a more reliable or less reliable agent. But that's an empirical question, not a logical one.