I strongly suspect that a Less Wrong reader with anosognosia would at first reply, "Well, of course I'm not certain that my left arm isn't paralyzed - you can't be infinitely certain about anything." And they might well go on to say, "But the fact that I don't feel any absolute certainty along these lines is, in fact, evidence that I don't have anosognosia".
I have seen many would-be rationalists who say "I'm not certain" and then, secure in having proven their rationality as much as anyone could possibly ask, forge straight on without a second glance. See The Proper Use of Humility.
I'm still having trouble with this one. I don't know why this particularly morbid example popped into my head, but here it is: we have very strong survival instincts. These can be overcome by overdosing on pills or jumping off a bridge. However, they cannot be overcome by holding your head in a bucket of water and trying to drown. You may be determined to kill yourself, but every time you try, as soon as you start breathing water you're going to pull your head back out. Now, I can say that "I'm not certain" I can't kill myself this way -- but in ...
Followup to: The Strangest Thing An AI Could Tell You
Brain damage patients with anosognosia are incapable of considering, noticing, admitting, or realizing even after being argued with, that their left arm, left leg, or left side of the body, is paralyzed. Again I'll quote Yvain's summary:
A brief search didn't turn up a base-rate frequency in the population for left-arm paralysis with anosognosia, but let's say the base rate is 1 in 10,000,000 individuals (so around 670 individuals worldwide).
Supposing this to be the prior, what is your estimated probability that your left arm is currently paralyzed?
Added: This interests me because it seems to be a special case of the same general issue discussed in The Modesty Argument and Robin's reply Sleepy Fools - when pathological minds roughly similar to yours update based on fabricated evidence to conclude they are not pathological, under what circumstances can you update on different-seeming evidence to conclude that you are not pathological?