AAT has two very important qualifiers that you have to include if you're discussing it. First, both individuals have to have the same priors. If I have some reason to believe that the other party and I have different priors, then my confidence that AAT applies will decrease as well. In the case of the creationist, I think it is at least likely that we have some different priors (occam's razor, nature of causality, etc), so I won't be terribly surprised if AAT doesn't seem to work in this case. Second, there is that picky clause about "perfect Bayesians". If a creationist is not accurately updating on the evidence, then I shouldn't expect AAT to work. If you really want to check yourself with AAT, you could (assuming you have the same priors) determine what an ideal bayesian would do with the evidence the other person has and determine if AAT works then. That math seems like it would get really complicated really fast.
With regards to the superintelligence/schizophrenics question: we can all agree that the AI would actually get the wrong answer if it inferred, "I think I am a superintelligence, therefore I probably am a schizophrenic". However, every schizophrenic with delusions of deification would draw the right conclusion. If I was trying to code a superintelligence, I wouldn't deliberately write the AI to do this, but if anyone ever reads this who thinks that they are a superintelligence, maybe you should try to test that claim, just to be sure. Write down some predictions and test how accurate you actually are.
On a side note, I've often wondered what would happen if a schizophrenic was taught the methods of rationality, or a rationalist developed schizophrenia. Have any cases of that ever happened? Is it something we should try to test?
I was inspired by the later scenes in A Beautiful Mind, where Nash was still hallucinating as he went about his day but he chose to just ignore his visions of people he knew were not real.
Today's post, The Modesty Argument, was originally published on December 10, 2006. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was The Proper Use of Humility, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.