The response Anna and I give in our forthcoming chapter "Intelligence Explosion: Evidence and Import" is the following:
Chalmers (2010) suggested that AI will lead to intelligence explosion if an AI is produced by an "extendible method," where an extendible method is "a method that can easily be improved, yielding more intelligent systems." McDermott (2012a, 2012b) replies that if P≠NP (see Goldreich 2010 for an explanation) then there is no extendible method. But McDermott's notion of an extendible method is not the one essential to the possibility of intelligence explosion. McDermott's formalization of an "extendible method" requires that the program generated by each step of improvement under the method be able to solve in polynomial time all problems in a particular class — the class of solvable problems of a given (polynomially step-dependent) size in an NP-complete class of problems. But this is not required for an intelligence explosion in Chalmers' sense (and in our sense). What intelligence explosion (in our sense) would require is merely that a program self-improve to vastly outperform humans, and we argue for the plausibility of this in section 3 of our chapter. Thus while we agree with McDermott that it is probably true that P≠NP, we do not agree that this weighs against the plausibility of intelligence explosion. (Note that due to a miscommunication between McDermott and the editors, a faulty draft of McDermott (2012a) was published in Journal of Consciousness Studies. We recommend reading the corrected version at http://cs-www.cs.yale.edu/homes/dvm/papers/chalmers-singularity-response.pdf.)
I sent this to Drew and he said he agreed with our rebuttal.
Do you feel this is a full rebuttal to McDermott's paper? I agree that his generalized argument against "extendible methods" is a straw man; however, he has other points about Chalmers' failure to argue for existing extendible methods being "extendible enough."
I'm doing an undergraduate course on the Free Will Theorem, with three lecturers: a mathematician, a physicist, and David Chalmers as the philosopher. The course is a bit pointless, but the company is brilliant. Chalmers is a pretty smart guy. He studied computer science and math as an undergraduate, before "discovering that he could get paid for doing the kind of thinking he was doing for free already". He's friendly; I've been chatting with him after the classes.
So if anyone has any questions for him, if they seem interesting enough I could approach him with them.
Emails to him also work, of course, but discussion in person lets more understanding happen faster. For example, in a short discussion with him I understood his position on consciousness way better than I would have just from reading his papers on the topic.