Comment on Muehlhauser-Goertzel Dialogue, Part 1

Mon, 19 Mar 2012 10:13:50 GMT

Many here disagree with the conclusion that superintelligences are likely to be super-moral?

If so, I didn't really know that. The only figure I have ever seen from Yudkowsky for the chance of failure is the rather vague one of "easily larger than 10%". The "GLOBAL CATASTROPHIC RISKS SURVEY" - presumably a poll of the ultra-paranoid - came with a broadly similar chance of failure by 2100 - far below 50%. Like many others, I figure that, if we don't fail, then we are likely to succeed.

Do the pessimists have an argument? About the only argument I have seen argues that superintelligences will by psychopaths "by default" since most goal-directed agents are psychopaths. That argument is a feeble one. Similarly, the space of all possible buildings is dominated by piles of rubble - and yet the world is filled with skyscrapers. Looking at evolutionary trends - as I proposed - is a better way of forecasting than looking at the space of possible agents.

Discuss