Dolores1984 comments on Thoughts on the Singularity Institute (SI) - Less Wrong

256 Post author: HoldenKarnofsky 11 May 2012 04:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (1270)

You are viewing a single comment's thread.

Comment author: Dolores1984 10 May 2012 07:26:04PM 7 points [-]

Leaving aside the question of whether Tool AI as you describe it is possible until I've thought more about it:

The idea of a "self-improving algorithm" intuitively sounds very powerful, but does not seem to have led to many "explosions" in software so far (and it seems to be a concept that could apply to narrow AI as well as to AGI).

Looking to the past for examples is a very weak heuristic here, since we have never dealt with software that could write code at a better than human level before. It's like saying, before the invention of the internal combustion engine, "faster horses have never let you cross oceans before." Same goes for the assumption that strong AI will resemble extremely narrow AI software tools that already exist in specific regards. It's evidence, but it's very weak evidence, and I for one wouldn't bet on it.