I appear today on Bloggingheads.tv, in "Science Saturday: Singularity Edition", speaking with John Horgan about the Singularity. I talked too much. This episode needed to be around two hours longer.
One question I fumbled at 62:30 was "What's the strongest opposition you've seen to Singularity ideas?" The basic problem is that nearly everyone who attacks the Singularity is either completely unacquainted with the existing thinking, or they attack Kurzweil, and in any case it's more a collection of disconnected broadsides (often mostly ad hominem) than a coherent criticism. There's no equivalent in Singularity studies of Richard Jones's critique of nanotechnology - which I don't agree with, but at least Jones has read Drexler. People who don't buy the Singularity don't put in the time and hard work to criticize it properly.
What I should have done, though, was interpreted the question more charitably as "What's the strongest opposition to strong AI or transhumanism?" in which case there's Sir Roger Penrose, Jaron Lanier, Leon Kass, and many others. None of these are good arguments - or I would have to accept them! - but at least they are painstakingly crafted arguments, and something like organized opposition.
Nick: "what makes you think the things humans do aren't implied by its initial state, nature, and inputs?"
What humans do is determined by their nature, just like with a computer. The difference is, human nature is to be able to choose, and computer nature is not.
"The form of choice reason demands (different outputs given different inputs) is perfectly compatible with determinism, in fact it requires determinism, since nondeterministic factors would imply less entanglement between beliefs and reality. If your conclusion is not totally determined by priors and evidence, you're doing something wrong."
You're not doing something wrong, because I don't think reason is pure discipline, pure modus-ponens. I think it's more like tempered creativity - utilizing mental actions such as choice, focus, imagination as well as pure logic. The computer just doesn't have what it takes.
But the point I was making is that the whole idea of reason wouldn't arise in the first place without prior acceptance of free will. It is only by accepting that we control our minds that the question of how best to do so arises, and ideas like reason, deduction etc. come to be.
All these ideas therefore presuppose free will in their very genesis, and can not validly be used to argue against it. It would be like trying to use the concept "stealing" in a proof against the validity of "property" - there is no such thing as stealing without property. Likewise there is no such thing as reason without free will.