wedrifid comments on Reply to Holden on 'Tool AI' - Less Wrong

94 Post author: Eliezer_Yudkowsky 12 June 2012 06:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (348)

You are viewing a single comment's thread.

Comment author: wedrifid 12 June 2012 06:08:04AM 9 points [-]

This is the first time I can recall Eliezer giving an overt indication regarding how likely an AGI project is to doom us. He suggests that 90% chance of Doom given intelligent effort is unrealistically high. Previously I had only seem him declare that FAI is worth attempting once you multiply. While he still hasn't given numbers (not saying he should) he has has given a bound. Interesting. And perhaps a little more optimistic than I expected - or at least more optimistic than I would have expected prior to Luke's comment.

Comment author: Lightwave 12 June 2012 09:56:04AM *  9 points [-]

how likely an AGI project is to doom us

Isn't it more like "how likely a formally proven FAI design is to doom us", since this is what Holden seems to be arguing (see his quote below)?

Suppose that it is successful in the "AGI" part of its goal, i.e., it has successfully created an intelligence vastly superior to human intelligence and extraordinarily powerful from our perspective. Suppose that it has also done its best on the "Friendly" part of the goal: it has developed a formal argument for why its AGI's utility function will be Friendly, it believes this argument to be airtight, and it has had this argument checked over by 100 of the world's most intelligent and relevantly experienced people. .. What will be the outcome?

Comment author: JamesAndrix 16 June 2012 10:47:44PM 1 point [-]

"When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong."

http://en.wikipedia.org/wiki/Clarke%27s_three_laws