Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

wedrifid comments on SIAI - An Examination - Less Wrong

143 Post author: BrandonReinhart 02 May 2011 07:08AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (203)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 15 May 2011 03:40:50PM *  1 point [-]

AI will be designed to try to seize absolute power, and take every possible measure to prevent humans from creating another AI. If your name appears on this website, you're already on its list of people whose continued existence will be risky.

Not much risk. Hunting down irrelevant blog commenters is a greater risk than leaving them be. There isn't much of a window during which any human is a slightest threat and during that window going around killing people is just going to enhance the risk to it.

Comment author: timtyler 15 May 2011 04:15:06PM *  2 points [-]

The window is presumably between now and when the winner is obvious - assuming we make it that far.

IMO, there's plenty of scope for paranoia in the interim. Looking at the logic so far some teams will reason that unless their chosen values get implemented, much of value is likely to be lost. They will then mulitiply that by a billion years and a billion planets - and conclude that their competitors might really matter.

Killing people might indeed backfire - but that still leaves plenty of scope for dirty play.

Comment author: wedrifid 15 May 2011 04:25:24PM 2 points [-]

The window is presumably between now and when the winner is obvious

No. Reread the context. This is the threat from "F"AI, not from designers. The window opens when someone clicks 'run'.

Comment author: timtyler 15 May 2011 05:54:30PM *  0 points [-]

Uh huh. So: world view difference. Corps and orgs will most likely go from 90% human to 90% machine through the well-known and gradual process of automation, gaining power as they go - and the threats from bad organisations are unlikely to be something that will appear suddenly at some point.