I'm still having trouble with this one. I don't know why this particularly morbid example popped into my head, but here it is: we have very strong survival instincts. These can be overcome by overdosing on pills or jumping off a bridge. However, they cannot be overcome by holding your head in a bucket of water and trying to drown. You may be determined to kill yourself, but every time you try, as soon as you start breathing water you're going to pull your head back out. Now, I can say that "I'm not certain" I can't kill myself this way -- but in reality, I know it's not possible. My brain has a very real physical process that just won't let me. Therefore, I don't feel that it is honest to say "I'm not certain" in this case. In fact, saying "I'm not certain" feels very much like saying "the sky is green" or "I like to eat glass", i.e. it feels like bullshit. Is that something a rationalist needs to overcome, if only so he can admit, "I could say I am not certain, but I am?" The argument against this seems to be saying, "I am not certain I am not a butterfly," even though it is not possible for a butterfly to have such a thought.
The Proper Use of Humility is one of my favorite articles of yours, by the way, and I do feel like I'm making progress in a worthy direction, even if it can seem like I'm random-walking on the way there.
However, they cannot be overcome by holding your head in a bucket of water and trying to drown. You may be determined to kill yourself, but every time you try, as soon as you start breathing water you're going to pull your head back out. Now, I can say that "I'm not certain" I can't kill myself this way -- but in reality, I know it's not possible. My brain has a very real physical process that just won't let me.
How do you know that? It doesn't seem intuitively obvious to me that you can't train to successfully drown this way. It'll take more than intuition, and I can't think of a way this could've been reliably studied, so I don't believe you can have a good reason to have this belief.
Followup to: The Strangest Thing An AI Could Tell You
Brain damage patients with anosognosia are incapable of considering, noticing, admitting, or realizing even after being argued with, that their left arm, left leg, or left side of the body, is paralyzed. Again I'll quote Yvain's summary:
A brief search didn't turn up a base-rate frequency in the population for left-arm paralysis with anosognosia, but let's say the base rate is 1 in 10,000,000 individuals (so around 670 individuals worldwide).
Supposing this to be the prior, what is your estimated probability that your left arm is currently paralyzed?
Added: This interests me because it seems to be a special case of the same general issue discussed in The Modesty Argument and Robin's reply Sleepy Fools - when pathological minds roughly similar to yours update based on fabricated evidence to conclude they are not pathological, under what circumstances can you update on different-seeming evidence to conclude that you are not pathological?