I appear today on Bloggingheads.tv, in "Science Saturday: Singularity Edition", speaking with John Horgan about the Singularity. I talked too much. This episode needed to be around two hours longer.
One question I fumbled at 62:30 was "What's the strongest opposition you've seen to Singularity ideas?" The basic problem is that nearly everyone who attacks the Singularity is either completely unacquainted with the existing thinking, or they attack Kurzweil, and in any case it's more a collection of disconnected broadsides (often mostly ad hominem) than a coherent criticism. There's no equivalent in Singularity studies of Richard Jones's critique of nanotechnology - which I don't agree with, but at least Jones has read Drexler. People who don't buy the Singularity don't put in the time and hard work to criticize it properly.
What I should have done, though, was interpreted the question more charitably as "What's the strongest opposition to strong AI or transhumanism?" in which case there's Sir Roger Penrose, Jaron Lanier, Leon Kass, and many others. None of these are good arguments - or I would have to accept them! - but at least they are painstakingly crafted arguments, and something like organized opposition.
Eli, Enjoyed your conversation with John today, though I suspect he would have tried to convince the Wright brothers to quit because so many had failed.
I read your essay on friendly AI, and think this essay is off the mark. If the singularity happens, there will be so many burdens the AI can throw off (such as anthropomorphism) it will be orders of magnitude superior very quickly. I think the apt analogy isn't we would be apes among men, but more like worms among men. Men need not nor should be concerned with worms, and worms aren't all that important in a world with men.
Ed