I wrote out a long response involving an analogy to a CPU self-test program, but at the end I realised that I had arrived at the same conclusion you stated. :-) So I'm voting you up and wish to extend you an Internet high-five.
However, on this topic, it seems like there's no good approach for handling the scenario where your brain messes with your internal tests in such a way as to point them invariably at a false positive, i.e. anosognosia.
I agree that a good self-test of the sort you describe would reduce the probability for most kinds of anything-goes insanity, but what sort of test could be used to check against the not-insignificant subset of insanity that specifically acts against self-tests and forces them to return false positive at the highest level?
It's always possible to produce insane minds that cannot fix themselves - the interesting question is how big a diff can be bridged at what price. And that's a bit more difficult to answer.
I wonder, however, whether a sufficiently educated anosognosiac could determine that the sources informing them of their paralysis were more reliable than their firsthand observations. It seems unlikely, of course.
Update: Please post new comments in the latest HPMOR discussion thread, now in the discussion section, since this thread and its first few successors have grown unwieldy (direct links: two, three, four, five, six, seven).
As many of you already know, Eliezer Yudkowsky is writing a Harry Potter fanfic, Harry Potter and the Methods of Rationality, starring a rationalist Harry Potter with ambitions to transform the world by bringing the rationalist/scientific method to magic. But of course a more powerful Potter requires a more challenging wizarding world, and ... well, you can see for yourself how that plays out.
This thread is for discussion of anything related to the story, including insights, confusions, questions, speculation, jokes, discussion of rationality issues raised in the story, attempts at fanfic spinoffs, comments about related fanfictions, and meta-discussion about the fact that Eliezer Yudkowsky is writing Harry Potter fan-fiction (presumably as a means of raising the sanity waterline).
I'm making this a top-level post to create a centralized location for that discussion, since I'm guessing people have things to say (I know I do) and there isn't a great place to put them. fanfiction.net has a different set of users (plus no threading or karma), the main discussion here has been in an old open thread which has petered out and is already near the unwieldy size that would call for a top-level post, and we've had discussions come up in a few other places. So let's have that discussion here.
Comments here will obviously be full of spoilers, and I don't think it makes sense to rot13 the whole thread, so consider this a spoiler warning: this thread contains unrot13'd spoilers for Harry Potter and the Methods of Rationality up to the current chapter and for the original Harry Potter series. Please continue to use rot13 for spoilers to other works of fiction, or if you have insider knowledge of future chapters of Harry Potter and the Methods of Rationality.
A suggestion: mention at the top of your comment which chapter you're commenting on, or what chapter you're up to, so that people can understand the context of your comment even after more chapters have been posted. This can also help people avoid reading spoilers for a new chapter before they realize that there is a new chapter.