jacobt comments on Superintelligent AGI in a box - a question. - Less Wrong

14 Post author: Dmytry 23 February 2012 06:48PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (77)

Comments (77)

You are viewing a single comment's thread. Show more comments above.

Comment author: jacobt 25 February 2012 09:29:25AM *  0 points [-]

The framework, as we already have established, would not keep an AI from maximizing what ever the AI wants to maximize.

That's only if you plop a ready-made AGI in the framework. The framework is meant to grow a stupider seed AI.

The framework also does nothing to prevent AI from creating a more effective problem solving AI that is more effective at problem solving by not evaluating your problem solving functions on various candidate solutions, and instead doing something else that's more effective.

Program (3) cannot be re-written. Program (2) is the only thing that is changed. All it does is improve itself and spit out solutions to optimization problems. I see no way for it to "create a more effective problem solving AI".

So what does the framework do, exactly, that would improve safety here?

It provides guidance for a seed AI to grow to solve optimization problems better without having it take actions that have effects beyond its ability to solve optimization problems.

Comment author: Dmytry 25 February 2012 01:22:21PM *  1 point [-]

A lot goes into solving the optimization problems without invoking the scoring function a trillion times (which would entirely prohibit self improvement).

Look at where similar kind of framework got us, the homo sapiens. We were minding our business evolving, maximizing own fitness, which was the all we could do. We were self improving (the output being next generation's us). Now there's talk of Large Hadron Collider destroying the world. It probably won't, of course, but we're pretty well going along the bothersome path. We also started as a pretty stupid seed AI, a bunch of monkeys. Scratch that, as unicellular life.

Powered by Reddit Powered by Reddit