Doug_S. comments on The Magnitude of His Own Folly - Less Wrong

26 Post author: Eliezer_Yudkowsky 30 September 2008 11:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (127)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Doug_S. 01 October 2008 06:45:00AM 0 points [-]

"The level of "intelligence" (if you can call it that) you're talking about with an AI whose able to draw up plans to destroy Earth (or the solar system), evade detection or convince humans to help it, actually enact its plans and survive the whole thing, is beyond the scope of realistic dreams for the first AI. It amounts to belief in a trickster deity, one which only FAI, the benevolent god, can save you from."

It's not necessarily the "first AI" as such. It's the first AI capable of programming an AI smarter than itself that we're worried about. Because that AI will make another, smarter one, and that one will make one smarter yet, and so on, until we end up with something that's as smart as the laws of physics and local resources will allow.

Bit of sci-fi speculation here: What would "computronium" actually look like? It might very well look almost exactly like a star. If our sun actually was a giant computer, running some vastly complex calculation, would we here on Earth be able to tell?

Comment author: Luke_A_Somers 11 September 2012 02:02:44PM *  0 points [-]

It might very well look almost exactly like a star.

No, it won't. The argument in favor of that is a strict upper bound, but there are far stricter upper bounds you can set, if you require things like the computer being capable of performing operations, or storing data.