Cynical explanations of FAI critics (including myself)

Wei Dai

Lately I've been pondering the fact that while there are many critics of SIAI and its plan to form a team to build FAI, few of us seem to agree on what SIAI or we should do instead. Here are some of the alternative suggestions offered so far:

work on computer security
work to improve laws and institutions
work on mind uploading
work on intelligence amplification
work on non-autonomous AI (e.g., Oracle AI, "Tool AI", automated formal reasoning systems, etc.)
work on academically "mainstream" AGI approaches or trust that those researchers know what they are doing
stop worrying about the Singularity and work on more mundane goals

Given that ideal reasoners are not supposed to disagree, it seems likely that most if not all of these alternative suggestions can also be explained by their proponents being less than rational. Looking at myself and my suggestion to work on IA or uploading, I've noticed that I have a tendency to be initially over-optimistic about some technology and then become gradually more pessimistic as I learn more details about it, so that I end up being more optimistic about technologies that I'm less familiar with than the ones that I've studied in detail. (Another example of this is me being initially enamoured with Cypherpunk ideas and then giving up on them after inventing some key pieces of the necessary technology and seeing in more detail how it would actually have to work.)

I'll skip giving explanations for other critics to avoid offending them, but it shouldn't be too hard for the reader to come up with their own explanations. It seems that I can't trust any of the FAI critics, including myself, nor do I think Eliezer and company are much better at reasoning or intuiting their way to a correct conclusion about how we should face the apparent threat and opportunity that is the Singularity. What useful implications can I draw from this? I don't know, but it seems like it can't hurt to pose the question to LessWrong.

work on computer security
work to improve laws and institutions
work on mind uploading
work on intelligence amplification
work on non-autonomous AI (e.g., Oracle AI, "Tool AI", automated formal reasoning systems, etc.)
work on academically "mainstream" AGI approaches or trust that those researchers know what they are doing
stop worrying about the Singularity and work on more mundane goals

I didn't downvote you, but I can see why someone reasonably might. Off the top of my head, in no particular order:

Whole brain emulation isn't the consensus best path to general AI. My intuition agrees with yours here, but you don't show any sign that you understand the subtleties involved well enough to be as certain as you are.
Lots of problematic unsupported assertions, e.g. "intelligent people generally have less children than those on the left half of the Bell curve", "[rich people] are also more likely to have above-average IQs, else they wouldn't be rich", and "[violence and docility are] in the genes and the brain that they produce".
Eugenics!?!
Ok, fine, eugenics, let's talk about it. Your discussion is naive: you assume that IQ is the right metric to optimize for (see Raising the Sanity Waterline for another perspective), you assume that we can measure it accurately enough to produce the effect you want, you assume that it will go on being an effective metric even after we start conditioning reproductive success on it, and your policy prescriptions are socially inept even by LW standards.
Also, it's really slow. That seems ok to you because you don't believe that we'll otherwise have recursive self-improvement in our lifetimes, but that's not the consensus view here either.

I'm not interested in debating any of this, I just wanted to give you an outside perspective on your own writing. I hope it helps, and I hope you decide to stick around.

31

Cynical explanations of FAI critics (including myself)

31

31

31

Cynical explanations of FAI critics (including myself)

31

31