For me, in just about every case, the credence I'd assign to an AI's wacky claims would depend on its ability to answer followup questions. For instance, in Eliezer's examples:

1) Tin-foil hats actually do block the Orbital Mind Control Lasers

What Orbital Mind Control Lasers? Who uses them? What do they do with them? Why haven't they come up with a way to get around the hats?

2) All mathematical reasoning involving "infinities" involves self-evident contradictions, but human mathematicians have a blind spot with respect to them.

I'm actually strangely comfortable with this one, possibly because I'm bad at math.

3) You are not above-average; most people believe in the existence of a huge fictional underclass in order to place themselves at the top of the heap, rather than in the middle. This is why so many of your friends seem to have PhDs despite PhDs supposedly constituting only 0.5% of the population. You are actually in the bottom third of the population; the other two-thirds have already built their own AIs.

Why haven't I heard of any of these other AIs before? How do all of the people producing statistics indicating that there are a lot of dumb people coordinate their efforts to perpetuate the fiction?

4) The human bias toward overconfidence is far deeper than we are capable of recognizing; we have a form of species overconfidence which denies all evidence against itself. Humans are much slower runners than we think, muscularly weaker, struggle to keep afloat in the water let alone move, and of course, are poorer thinkers.

Why do so few of us die of drowning (or any of the other things that would kill us if we were so dramatically more pathetic than we believe)? If this bias is so pervasive, why can I see these words on the AI's screen, when it seems that I should block them out as with all over evidence that we are pathetic in this way?

5) Dogs, cats, cows, and many other mammals are capable of linguistic reasoning and have made many efforts to communicate with us, but humans are only capable of recognizing other humans as capable of thought.

If we have this incapability, what explains the abundant fiction in which nonhuman animals (both terrestrial and non) are capable of speech, and childhood anthropomorphization of animals? Can you teach me to talk to the stray cat in my neighborhood? Why only mammals, not birds and the like? What about people who are actively trying to communicate with animals like gorillas, or are those not capable of communication?

6) Humans cannot reproduce without the aid of the overlooked third sex.

Are they overlooked in the sense that people we can otherwise detect are not recognized as being part of this sex, or in the sense that we literally do not notice the existence of the members of this sex? In the former case, how do so many people manage to reproduce without apparently wanting to or involving third parties? In the latter case, how can I get in touch with these people? By what mechanism are they involved in human reproduction?

7) The Earth is flat.

Are we talking Euclidean spacetime here? What is the explanation for the observations of a spheroid Earth?

8) Human beings are incapable of writing fiction; all supposed fiction you have read is actually true.

In this universe? What about stories with plot holes? I think that I have written fiction in the past; am I in causal contact with the events I describe? When I make an edit that changes the plot, how does that work? What about people who write self-insertions?