= 27d567bc2d48d9f40e9ccc20ea370b15)
JStewart comments on Open thread, Nov. 24 - Nov. 30, 2014 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (317)
This may be a naive question, which has a simple answer, but I haven't seen it. Please enlighten me.
I'm not clear on why an AI should have a utility function at all.
The computer I'm typing this on doesn't. It simply has input-output behavior. When I hit certain keys it reacts in certain, very complex ways, but it doesn't decide. It optimizes, but only when I specifically tell it to do so, and only on the parameters that I give it.
We tend to think of world-shaping GAI as an agent with it's own goals, which it seeks to implement. Why can't it be more like a computing machine in a box. We could feed it questions, like "given this data, will it rain tomorrow?", or "solve this protein folding problem", or "which policy will best reduce gun-violence?", or even "given these specific parameters and definitions, how do we optimize for human happiness?" For the complex answers like the last of those, we could then ask the AI to model the state of the world that results from following this policy. If we see that it leads to tiling the universe with smiley faces, we know that we made a mistake somewhere (that wasn't what we were trying to optimize for), and adjust the parameters. We might even train the AI over time, so that it learns how to interpret what we mean from what we say. When the AI models a state of the world that actually reflects our desires, then we implement it's suggestions ourselves, or perhaps only then hit the implement button, by with the AI takes the steps to carry out it's plan. We might even use such a system to check the safety of future generations of the AI. This would slow recursive self improvement, but it seems it would be much safer.
This has been proposed before, and on LW is usually referred to as "Oracle AI". There's an entry for it on the LessWrong wiki, including some interesting links to various discussions of the idea. Eliezer has addressed it as well.
See also Tool AI, from the discussions between Holden Karnofsky and LW.
I was just reading though the Eliezer article. I'm not sure I understand. Is he saying that my computer actually does have goals?
Isn't there a difference between simple cause and effect and an optimization process that aims at some specific state?
Maybe it would help to "taboo" the word "goal".
A process can progress towards some end state even without having any representation of that state. Imagine a program that takes a positive number at the beginning, and at each step replaces the current number "x" with value "x/2 + 1/x". Regardless of the original number, the values will gradually move towards a constant. Can we say that this process has a "goal" or achieving the given number? It feels wrong to use this word here, because the constant is nowhere in the process, it just happens.
Typically, when we speak about having a "goal" X, we mean that somewhere (e.g. in human brain, or in the company's mission statement) there is a representation of X, and then the reality is compared with X, various paths from here to X are evaluated, and then one of those paths is followed.
I am saying this to make more obvious that there is a difference between "having a representation of X" and "progressing towards X". Humans typically create representations of their desired end states, and then try finding a way to achieve them. Your computer doesn't have this, and neither does "Tool AI" at the beginning. Whether it can create representations later, that depends on technical details, how specifically such "Tool AI" is programmed.
Maybe there is a way to allow superhuman thinking even without creating representations corresponding to things normally perceived in our world. (For example AIXI.) But even in such case, there is a risk of having a pseudo-goal of the "x/2 + 1/x" kind, where the process progresses towards an outcome even without having a representation of it. AI can "escape from the box" even without having a representation of "box" and "escape", if there exists a way to escape from it.
I don't get this explanation. Sure, a process can tend toward a certain result, without having an explicit representation of that result. But such tendencies often seem to be fragile. For example, a car engine homeostatically tends toward a certain idle speed. But take out one or all spark plugs, and the previously stable performance evaporates. Goals-as-we-know-them, by contrast, tend to be very robust. When a human being loses a leg, they will obtain a synthetic one, or use a wheelchair. That kind of robustness is part of what makes a very powerful agent scary, because it is intimately related to the agent's seeing many things as potential resources to use toward its ends.