You are filling in a pattern rather than making a useful observation. E_Y expressed incredulity and ignorance on the subject of making some parts of the code running on a computer harder to modify than other parts of the code on that same computer; I cited a source demonstrating that it is, in fact, a well-established thing. Not impossible to modify, not infallibly isolated from the outside world. Just more of a challenge to alter.

Comment author: Incorrect 14 June 2012 01:10:37AM 0 points [-]

I do not believe I am only filling in a pattern.

Putting the self-modifying parts of the AI (which we might as well call the actual AI) in the equivalent of a VM is effectively the same as forcing it to interact with the world through a limited interface which is an example of the AI box problem.

Comment author: Vaniver 14 June 2012 01:02:40AM 1 point [-]

Right- I think the issue is more that I (at least) view the AI as operating entirely in ring 3. It might be possible to code one where the utility function is ring 0, I/O is ring 1, and action-plans are ring 3, but for those distinctions to be meaningful they need to resist bad self-modifying and allow good self-modification.

For example, we might say "don't make any changes to I/O drivers that have a massively positive effect on the utility function" to make it so that the AI can't hallucinate its reward button being pressed all the time. But how do we differentiate between that and it making a change in ring 3 from a bad plan to a great plan, that results in a massive increase in reward?

Comment author: Eliezer_Yudkowsky 14 June 2012 02:23:17AM 7 points [-]

Suppose your utility function U is in ring 0 and the parts of you that extrapolate consequences are in ring 3. If I can modify only ring 3, I can write my own utility function Q, write ring-3 code that first extrapolates consequences fairly, pick the one that maximizes Q, and then provides a "prediction" to ring 0 asserting that the Q-maximizing action has consequence X that U likes, while all other actions have some U-disliked or neutral consequence. Now the agent has been transformed from a U-maximizer to a Q-maximizer by altering only ring 3 code for "predicting consequences" and no code in ring 0 for "assessing utilities".

One would also like to know what happens if the current AI, instead of "self"-modifying, writes a nearly-identical AI running on new hardware obtained from the environment.

Comment author: Vaniver 15 June 2012 12:58:35AM * 0 points [-]

Suppose your utility function U is in ring 0 and the parts of you that extrapolate consequences are in ring 3.

Sure; that looks like the hallucination example I put forward, except in the prediction instead of the sensing area. My example was meant to highlight that it's hard to get a limitation with high specificity, and not touch the issue of how hard it is to get a limitation with high sensitivity. (I find that pushing people in two directions is more effective at communicating difficulty than pushing them in one direction.)

The only defense I've thought of against those sorts of hallucinations is a "is this real?" check that feeds into the utility function- if the prediction or sensation module fails some test cases, then utility gets cratered. It seems too weak to be useful: it only limits the prediction / sensation module when it comes to those test cases, and a particularly pernicious modification would know what the test cases are, leave them untouched, and make everything else report Q-optimal predictions. (This looks like it turns into a race / tradeoff game between testing to keep the prediction / sensation software honest and the costs of increased testing, both in reduced flexibility and spent time / resources. And the test cases might be vulnerable, and so on.)

Comment author: Strange7 14 June 2012 01:45:04AM 1 point [-]

I don't think the utility function should be ring 0. Utility functions are hard, and ring zero is for stuff where any slip-up crashes the OS. Ring zero is where you put the small, stupid, reliable subroutine that stops the AI from self-modifying in ways that would make it unstable, or otherwise expanding it's access privileges in inappropriate ways.

Comment author: Eliezer_Yudkowsky 14 June 2012 02:18:55AM 0 points [-]

I'd like to know what this small subroutine looks like. You know it's small, so surely you know what's in it, right?

Comment author: wedrifid 14 June 2012 03:09:51AM 3 points [-]

I'd like to know what this small subroutine looks like. You know it's small, so surely you know what's in it, right?

Doesn't actually follow. ie. Strange7 is plainly wrong but this retort still fails.

Comment author: thomblake 19 June 2012 08:00:39PM 0 points [-]

It doesn't follow necessarily, but Eliezer has justified skepticism that someone who doesn't know what's in the subroutine would have good reason to say that it's small.

Comment author: wedrifid 19 June 2012 08:10:39PM * 9 points [-]

It doesn't follow necessarily, but Eliezer has justified skepticism that someone who doesn't know what's in the subroutine would have good reason to say that it's small.

He knows that there is no good reason (because it is a stupid idea) so obviously Strange can't know a good reason. That leaves the argument as the lovechild of hindsight bias and dark-arts rhetorical posturing.

I probably wouldn't have comment if I didn't notice Eliezer making a similar error in the opening post, significantly weakening the strength of his response to Holden.

If Holden says there's 90% doom probability left over no matter what sane intelligent people do (all of which goes away if you just build Google Maps AGI, but leave that aside for now) I would ask him what he knows now, in advance, that all those sane intelligent people will miss. I don't see how you could (well-justifiedly) access that epistemic state.

I expect much, much better than this from Eliezer. It is quite possibly the dumbest thing I have ever heard him say and the subject of rational thinking about AI is supposed to be pretty much exactly his area of expertise.

Comment author: Eliezer_Yudkowsky 20 June 2012 08:06:26PM 3 points [-]

Not all arguing aimed at people with different premises is Dark Arts, y'know. I wouldn't argue from the Bible, sure. But trying to make relatively vague arguments accessible to people in a greater state of ignorance about FAI, even though I have more specific knowledge of the issue that actually persuades me of the conclusion I decided to argue? I don't think that's Dark, any more than it's Dark to ask a religious person "How could you possibly know about this God creature?", when you're actually positively convinced of God's nonexistence by much more sophisticated reasoning like the general argument against supernaturalism as existing in the model but not the territory. The simpler argument is valid - it just uses less knowledge to arrive at a weaker version of the same conclusion.

Likewise my reply to Strange; yes, I secretly know the problem is hard for much more specific reasons, but it's also valid to observe that if you don't know how to make the subroutine you don't know that it's small, and this can be understood with much less explanation, albeit it reaches a weaker form of the conclusion.

continue this thread »

Comment author: Strange7 20 June 2012 09:04:04AM -2 points [-]

Hence my "used to be cool" comment.

continue this thread »

Comment author: Strange7 14 June 2012 02:56:52AM -1 points [-]

As I previously mentioned, the design of software is not my profession. I'm not a surgeon or an endocrinologist, either, even though I know that an adrenal gland is smaller, and in some ways simpler, than the kidney below it. If you had a failing kidney, would you ask me to perform a transplant on the basis of that qualification alone?

Comment author: Eliezer_Yudkowsky 13 June 2012 11:00:18PM 0 points [-]

And what does a multi-ring agent architecture look like? Say, the part of the AI that outputs speech to a microphone - what ring is that in?

Comment author: Strange7 13 June 2012 11:32:02PM 1 point [-]

Say, the part of the AI that outputs speech to a microphone - what ring is that in?

I am not a professional software designer, so take all this with a grain of salt. That said, hardware I/O is ring 1, so the part that outputs speech to a speaker would be ring 1, while an off-the-shelf 'text to speech' app could run in ring 3. No part of a well-designed agent would output anything to an input device, such as a microphone.

Comment author: Eliezer_Yudkowsky 14 June 2012 02:16:51AM 0 points [-]

Let me rephrase. The part of the agent that chooses what to say to the user - what ring is that in?

Comment author: Strange7 14 June 2012 03:31:20AM 1 point [-]

That's less of a rephrasing and more of a relocating the goalposts across state lines. "Choosing what to say," properly unpacked, is approximately every part of the AI that doesn't already exist.

Comment author: Eliezer_Yudkowsky 14 June 2012 03:34:30AM 3 points [-]

Yes. That's the problem with the ring architecture.

Comment author: Strange7 14 June 2012 03:59:42AM 3 points [-]

As opposed to a problem with having a massive black box labeled "decisionmaking" in your AI plans, and not knowing how to break it down into subgoals?

Strange7 comments on Reply to Holden on 'Tool AI' - Less Wrong

Article Navigation

Comments (348)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (348)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login