Cynical explanations of FAI critics (including myself)

Wei Dai

31 Cynical explanations of FAI critics (including myself)

13th Aug 2012

1 min read

31

Lately I've been pondering the fact that while there are many critics of SIAI and its plan to form a team to build FAI, few of us seem to agree on what SIAI or we should do instead. Here are some of the alternative suggestions offered so far:

work on computer security
work to improve laws and institutions
work on mind uploading
work on intelligence amplification
work on non-autonomous AI (e.g., Oracle AI, "Tool AI", automated formal reasoning systems, etc.)
work on academically "mainstream" AGI approaches or trust that those researchers know what they are doing
stop worrying about the Singularity and work on more mundane goals

Given that ideal reasoners are not supposed to disagree, it seems likely that most if not all of these alternative suggestions can also be explained by their proponents being less than rational. Looking at myself and my suggestion to work on IA or uploading, I've noticed that I have a tendency to be initially over-optimistic about some technology and then become gradually more pessimistic as I learn more details about it, so that I end up being more optimistic about technologies that I'm less familiar with than the ones that I've studied in detail. (Another example of this is me being initially enamoured with Cypherpunk ideas and then giving up on them after inventing some key pieces of the necessary technology and seeing in more detail how it would actually have to work.)

I'll skip giving explanations for other critics to avoid offending them, but it shouldn't be too hard for the reader to come up with their own explanations. It seems that I can't trust any of the FAI critics, including myself, nor do I think Eliezer and company are much better at reasoning or intuiting their way to a correct conclusion about how we should face the apparent threat and opportunity that is the Singularity. What useful implications can I draw from this? I don't know, but it seems like it can't hurt to pose the question to LessWrong.

Personal Blog

31

New Comment

Rendering 0/49 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 7:55 PM

Moderation Log

31 Cynical explanations of FAI critics (including myself)

by Wei Dai

13th Aug 2012

1 min read

31

work on computer security
work to improve laws and institutions
work on mind uploading
work on intelligence amplification
work on non-autonomous AI (e.g., Oracle AI, "Tool AI", automated formal reasoning systems, etc.)
work on academically "mainstream" AGI approaches or trust that those researchers know what they are doing
stop worrying about the Singularity and work on more mundane goals

Personal Blog

31

New Comment

Rendering 0/49 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 7:55 PM

Moderation Log

More from Wei Dai

Curated and popular this week

49Comments

Comment Permalink

nykos14y-110

OK, I got two minuses already, can't say I'm surprised because what I wrote is not politically correct, and probably some of you thought that I broke the "politics is the mind-killer" informal rule (which is not really rational if you happen to believe that the default political position - the one most likely to pass under the radar as non-mindkilling - is not static, but in fact is constantly shifting, usually in a leftwards direction).

For the sake of all rationalists, I hope I was downvoted because of the latter. Otherwise, all hope for rational argument is lost, if even people in the rationalist community adopt thought processes more similar to those of politicians (i.e., demotism) than true scientists.

The unfortunate fact is that you cannot separate the speed of scientific progress from public policy or the particular structure of the society engaged in science. Science is not some abstract ideal, it is the triumph of the human mind, of the still-rare people possessing both intelligence and rationality (the latter may even be restricted only to their area of expertise, see Abdus Salam or Georges Lemaître). Humans are inherently political animals. The quality of science depends directly, first and foremost, on the number and quality of minds performing it, and some political positions happen to be ways to increase that number more than others. Simply ignoring the connection is not an option if you really believe in the promise of science to help improve the lives of every human being no matter his IQ or mental profile (like I do).

If you downvote me, I have one request: I would at least like to read why.

CarlShulman14y100

Discussion of intelligence enhancement via reproductive biotechnology can occur smoothly here, e.g. in Wei Dai's post and associated comment thread several months ago. Looking at those past comments, I am almost certain that I could rewrite your comment to convey the same core points and yet have it be upvoted.

I think your comment was relatively ill-received because:

1) It threw in a number of other questionable claims on different topics without extensive support, rather than focusing on one at a time, and suggested very high confidence in the agglomerati... (read more)

6fezziwig14y

I didn't downvote you, but I can see why someone reasonably might. Off the top of my head, in no particular order: 1. Whole brain emulation isn't the consensus best path to general AI. My intuition agrees with yours here, but you don't show any sign that you understand the subtleties involved well enough to be as certain as you are. 2. Lots of problematic unsupported assertions, e.g. "intelligent people generally have less children than those on the left half of the Bell curve", "[rich people] are also more likely to have above-average IQs, else they wouldn't be rich", and "[violence and docility are] in the genes and the brain that they produce". 3. Eugenics!?! 4. Ok, fine, eugenics, let's talk about it. Your discussion is naive: you assume that IQ is the right metric to optimize for (see Raising the Sanity Waterline for another perspective), you assume that we can measure it accurately enough to produce the effect you want, you assume that it will go on being an effective metric even after we start conditioning reproductive success on it, and your policy prescriptions are socially inept even by LW standards. 5. Also, it's really slow. That seems ok to you because you don't believe that we'll otherwise have recursive self-improvement in our lifetimes, but that's not the consensus view here either. I'm not interested in debating any of this, I just wanted to give you an outside perspective on your own writing. I hope it helps, and I hope you decide to stick around.

See in context