The trouble there is the claim of a rational mind, in my opinion. It's not logically necessary that our evolved brains, hacked by culture, are going to mirror reality in their most basic perceptions and intuitions. The space of all possible minds includes some which have a notion of number and counting and an intuitive mental arithmetic, but for which 2 and 2 really do seem to make 3 when they think of it. These minds, of course, would notice empirical contradictions everywhere: they would put two objects together with two more, count them, and count four instead of three, when it's obvious by visualizing in their heads that two and two make three instead. Eventually, a sufficiently reflective mind of this type would entertain the possibility that maybe two and two do actually make four, and that its system of visualization and mental arithmetic are in fact wrong, as obvious as they seem from the inside. Switching "three" and "four" in this paragraph just illustrates how difficult accepting that hypothesis might actually be for such a mind. The thing is, we ourselves are in this situation, not with arithmetic (fortunately, we receive constant empirical reinforcement that 2+2=4 and that our mental faculties for arithmetic work properly) but with our biases of thought. Things like our preferences and valuations seem to be rational and coherent, in that we can usually defend them all with arguments that look solid and persuasive to us. But occasionally this fiction becomes untenable, as when we are shown to have circular preferences in situations of risk and reward. As Eliezer put it in Zut Allais: [...] Or, in this case, "Don't start by assuming that our minds work rationally whenever they see something as obvious! If this is true, it is an empirical fact; and you should be able to see the alternative as possible!"