nshepperd comments on David Chalmers' "The Singularity: A Philosophical Analysis" - Less Wrong

33 Post author: lukeprog 29 January 2011 02:52AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (202)

Comments (202)

Sort By: Best

Show:

You are viewing a single comment's thread. Show more comments above.

Comment author: nshepperd 30 January 2011 06:05:05AM * 2 points [-]

Why would an AI which optimises for one thing create another AI that optimises for something else? Not every change is an improvement, but every improvement is necessarily a change. Building an AI with a different utility function is not going to satisfy the first AI's utility function! So whatever AI the first one builds is necessarily going to either have the same utility function (in which case the first AI is working correctly), or have a different one (which is a sign of malfunction, and given the complexity of morality, probably a fatal one).

It's not possible to create an AGI that is "somewhat better than us" in the sense that it has a better utility function. To the extent that we have a utility function at all, it would refer to the abstract computation called "morality", which "better" is defined by. The most moral AI we could create is therefore one with precisely that utility function. The problem is that we don't exactly know what our utility function is (hence CEV).

There is a sense in which a Friendly AGI could be said to be "better than us", in that a well-designed one would not suffer from akrasia and whatever other biases prevent us from actually realizing our utility function.

Comment author: Stuart_Armstrong 30 January 2011 06:17:57PM 2 points [-]

AI's without utility functions, but some other motivational structure, will tend to self-improve to a utility function AI. Utility-function AI's seem more stable under self-improvement, but there are many reasons it might want to change its utility (eg speed of access, multi-agent situations).

Comment author: Oligopsony 30 January 2011 06:53:26PM 0 points [-]

Could you clarify what you mean by an "other motivational structure?" Something with preference non-transitivity?

Comment author: Stuart_Armstrong 30 January 2011 07:47:18PM 1 point [-]

For instance. http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf

Comment author: Perplexed 30 January 2011 05:04:21PM 2 points [-]

Why would an AI which optimises for one thing create another AI that optimises for something else?

It wouldn't if it initially considered itself to be the only agent in the universe. But if it recognizes the existence of other agents and the impact of other agents' decisions on its own utility, then there are many possibilities:

The new AI could be created as a joint venture of two existing agents.
The new AI could be built because the builder was compensated for doing so.
The new AI could be built because the builder was threatened into doing so.

Building an AI with a different utility function is not going to satisfy the first AI's utility function!

This may seem intuitively obvious, but it is actually often false in a multi-agent environment.

Comment author: jacob_cannell comment score below threshold [+] (26 children)

Comment author: Vladimir_Nesov 30 January 2011 09:43:09AM 0 points [-]

Please read the sequences, and stop talking about AI until you do.

Comment author: jacob_cannell 30 January 2011 09:43:42AM 1 point [-]

I've read the sequences. Discuss or leave me alone.

Comment author: Vladimir_Nesov 30 January 2011 09:46:13AM * 0 points [-]

I've read the sequences.

Thanks, that's useful to know.

Edit: Seriously, no irony, that's useful. Disagreement should be treated differently depending on background.

Comment author: jacob_cannell 30 January 2011 09:52:53AM 0 points [-]

Also, very little of the sequences have much of anything to do with AI. If I want to learn more about that I would look to Norvig's book or more likely the relevant papers online. No need to be rude just because I don't hold all your same beliefs.

Comment author: Vladimir_Nesov 30 January 2011 10:00:07AM * 1 point [-]

Also, very little of the sequences have much of anything to do with AI.

It's more of a problem with your understanding of ethics, as applied to AI (and since this is the main context in which AI is discussed here, I referred to that as simply AI). You might be very knowledgeable in contemporary machine learning or other AI ideas while not seeing, for example, the risks of building AGIs.

No need to be rude just because I don't hold all your same beliefs.

Unfortunately there is (in some senses of "rude", such as discouraging certain conversational modes).

Comment author: jacob_cannell 30 January 2011 10:23:34AM 0 points [-]

You might be very knowledgeable in contemporary machine learning or other AI ideas while not seeing, for example, the risks of building AGIs

I see the potential risks in building AGIs.

I don't see that risk being dramatically high for creating AGIs based loosely on improving the human brain, and this approach appears to be mainstream now or becoming the mainstream (Kurzweil, Hawkins, Darpa's neuromorphic initiative, etc).

I'm interested in the serious discussion or analysis of why that risk could be high.

Comment author: wedrifid 30 January 2011 12:42:39PM 3 points [-]

I see the potential risks in building AGIs.

You have been discussing favourably the creation of AGIs that are programmed to create AGIs with different values to their own. No, you do not understand the potential risks.

Comment author: jacob_cannell 31 January 2011 04:50:57PM 0 points [-]

We create children that can have different values than our own, and over time this leads to significant value drift. But perhaps it should be called 'value evolution'.

This process is not magically guaranteed to preserve our best interests from our current perspective when carried over to AGI, but nor is guaranteed to spontaneously destroy the world.

Comment author: DSimon 31 January 2011 04:59:16PM * 0 points [-]

We create children that can have different values than our own, and over time this leads to significant value drift. But perhaps it should be called 'value evolution'.

Your analogy with evolution is spot on: if the values are going to drift at all, we want to drift towards some target point, by selecting against sub-AIs that have values further from the point.

However, if we can do that, why not just put that target point right in the first AI's utility function, and prevent any value drift at all? It seems like it ends up with the same result, but with slightly less complication.

And, if we can't set a target point for the value drift evolution... then it might drift anywhere at all! The chances that it would drift somewhere we'd like are pretty small. This applies even if it were a human-brain-based AGI; in general people are quite apt to go corrupt when given only a tiny bit of extra power. A whole load of extra power, like superintelligence would grant, would have a good chance of screwing with that human's values dramatically, possibly with disastrous effects.

Comment author: jacob_cannell 01 February 2011 12:35:03AM 0 points [-]

Your analogy with evolution is spot on: if the values are going to drift at all, we want to drift towards some target point, by selecting against sub-AIs that have values further from the point.

Yes.

However, if we can do that, why not just put that target point right in the first AI's utility function, and prevent any value drift at all?

The true final 'target point' is unknown, and unknowable in principle. We don't have the intelligence/computational power right now to know it, no AGI we can build will know it exactly, and this will forever remain true.

Our values are so complex that the 'utility function' that describes them is our entire brain circuit - and as we evolve into more complex AGI designs our values will grow in complexity as well.

Fixing them completely would be equivalent to trying to stop evolution. It's pointless, suicidal, impossible.

And, if we can't set a target point for the value drift evolution... then it might drift anywhere at all!

Yes evolution could in principle take us anywhere, but we can and already do exert control over it's direction.

This applies even if it were a human-brain-based AGI; in general people are quite apt to go corrupt when given only a tiny bit of extra power.

Humans today have a range of values, but an overriding universal value is not-dying. To this end it is crucially important that we reverse engineer the human mind.

Ultimately if what we really value is conscious human minds, and computers will soon out-compete human brains, then clearly we need to transfer human minds over to computers.

Comment author: Vladimir_Nesov 30 January 2011 10:52:55AM 1 point [-]

One simple point is that there is no reason to expect AGIs to stop at exactly human level. Even if progress and increase in intelligence is very slow, eventually they become an existential risk, or at least a value risk. Every step in that direction we make now is a step in the wrong direction, which holds even if you believe it's a small step.

Comment author: FAWS 30 January 2011 02:19:30PM * 1 point [-]

One simple point is that there is no reason to expect AGIs to stop at exactly human level.

This isn't the first time I heard this, but I don't think it's exactly right.

We know that human level is possible, but while super human level being possible seems overwhelmingly likely from considerations like imagining a human with more working memory and running faster we don't technically know that.
We have a working example of a human level intelligence.
It's human level intelligences doing the work. Martians work on AI might asymptotically slow down when approaching martian level intelligence without that level being inherently significant for anyone else, and the same for humans, or any AGI of any level working on its own successor for that matter (not that I have any strong belief that this is the case, it's just an argument for why human level wouldn't be completely arbitrary as a slow down point)

I'd completely agree with "there is no strong reason to expect AGIs to stop at exactly human level", "High confidence* in AGIs stopping at exactly human level is irrational" or "expecting AGIs not to stop at exactly human level would be prudent."

*Personally I'd assign a probability of under 0.2 to the best AGI's being on a level roughly comparable to human level (let's say being able to solve any problem except human relationship problems that every IQ 80+ human can solve, but not being better at every task than any human) for at least 50 years (physical time in Earth's frame of reference, not subjective time; probably means inferior at an equal clock rate but making up for that with speed for most of that time). That's a lot more than I would assign any other place on the intelligence scale of course.

Comment author: FAWS 30 January 2011 03:09:17PM * 0 points [-]

Could the downvoter please say what they are disagreeing with? I can see at least a dozen mutually contradictory possible angles so "someone thinks something about posting this is wrong" provides almost no useful information.

Comment author: jacob_cannell 30 January 2011 07:22:21PM 0 points [-]

Thanks for the value risk link - that discussion is what I'm interested in.

I guess I'll reply to it there. The initial quotes from Ben G. and Hanson are similar to my current view.

Comment author: Perplexed 30 January 2011 02:58:38PM 1 point [-]

very little of the sequences have much of anything to do with AI.

There is some discussion of the dangers of a uFAI Singularity, particularly in this debate between Robin Hanson and Eliezer. Much of the danger arises from the predicted short time period required to get from a mere human-level AI to a superhuman AI+. Eliezer discusses some reasons to expect it to happen quickly here and here. The concept of a 'resource overhang' is crucial in dismissing Robin's skepticism (which is based on historical human experience in economic growth - particularly in the accumulation of capital).

For an analysis of the possibility of a hard takeoff in approaches to AI based loosely on modeling or emulating the human brain, see this posting by Carl Schulman, for example.

Comment author: timtyler 30 January 2011 03:59:27PM * 0 points [-]

The concept of a 'resource overhang' is crucial in dismissing Robin's skepticism (which is based on historical human experience in economic growth - particularly in the accumulation of capital).

If civilisation(t+1) can access resources much better than civilisation(t), then that is just another way of saying things are going fast - one must beware of assuming what one is trying to demonstrate here.

The problem I see with this thinking is the idea that civilisation(t) is a bunch of humans while civilisation(t+1) is a superintelligent machine.

In practice, civilisation(t) is a man-machine symbiosis, while civilisation(t+1) is another man-machine symbiosis with a little bit less man, and a little bit more machine.

Comment author: nshepperd 30 January 2011 10:37:03AM * 1 point [-]

Yes it certainly can - if that new AI helps it's creator.

The same issue applies to children - they don't necessarily have the same 'utility function', sometimes they even literally kill us, but usually they help us.

That would be not so much a benevolence explosion as a single AI creating "slave" AIs for its own purposes. If some of the child AI's goals (for example those involved in being more good) are opposed to the parent's goals (for example those which make the parent AI less good), the parent is not going to just let the child achieve its goals. Rational agents do not let their utility functions change.

Sure it is - this part at least is easy. For example an AGI that is fully altruistic and only experiences love as it's single emotion would be clearly "somewhat better than us" from our perspective in every sense that matters.

If you mean that the AI doesn't suffer from the akrasia and selfishness and emotional discounting and uncertainty about our own utility function which prevents us from acting out our moral beliefs then I agree with you. That's the AI being more rational than us, and therefore better optimising for its utility function. But a literally better utility function is impossible, given that "better" is defined by our utility function.

Moreover, if our utility function describes what we truly want (which is the whole point of a utility function), it follows that we truly want an AI that optimizes for our utility function. If "better" were a different utility function then it would be unclear why we are trying to create an AI that does that, rather than what we want.

The real problem is the idea that morality can or should be simplified down to a 'utility function' simple enough for a human to code.

That's why the plan is for the AI to figure it out by inspecting us. Morality is very much not simple to code.

Comment author: jacob_cannell 30 January 2011 09:05:48PM 0 points [-]

The same issue applies to children - they don't necessarily have the same 'utility function', sometimes they even literally kill us, but usually they help us.

That would be not so much a benevolence explosion as a single AI creating "slave" AIs for its own purposes

So do we create children as our 'slaves' for our own purposes? You seem to be categorically ruling out the entire possibility of humans creating human-like AIs that have a parent-child relationship with their creators.

So just to make it precisely clear, I'm talking about that type of AI specifically. The importance and feasibility of that type of AGI vs other types is a separate discussion.

Sure it is - this part at least is easy. For example an AGI that is fully altruistic and only experiences love as it's single emotion would be clearly "somewhat better than us" from our perspective in every sense that matters.

If you mean that the AI doesn't [ .. ] That's the AI being more rational than us, and therefore better optimising for its utility function.

I don't see it as having anything to do with rationality.

The altruistic human-ish AGI mentioned above would be better than current humans from our current perspective - more like what we wish ourselves to be, and more able to improve our world than current humans.

Moreover, if our utility function describes what we truly want (which is the whole point of a utility function), it follows that we truly want an AI that optimizes for our utility function.

Yes.

This is obvious if it's 'utility function' is just a projection of my own - ie it simulates what I would want and uses that as it's utility function, but that isn't even necessary - it's utility function could be somewhat more complex than just a simulated projection of my own and still help fulfill my utility function.

That's why the plan is for the AI to figure it out by inspecting us. Morality is very much not simple to code.

If by inspection you just mean teach the AI morality in human language, then I agree, but that's a side point.

Comment author: TheOtherDave 30 January 2011 04:40:20PM * 0 points [-]

So: I want to finish my novel, but I spend the day noodling around the Internet instead.

Then Omega hands me an AI which it assures me is programmed error-free to analyze me and calculate my utility function and optimize my environment in terms of it.

I run the AI, and it determines exactly which parts of my mind manifest a desire to finish the novel, which parts manifest a desire to respond to the Internet, and which parts manifest a desire to have the novel be finished. Call them M1, M2 and M3. (They are of course overlapping sets.) Then it determines somehow which of these things are part of my utility function, and which aren't, and to what degree.

So...

Case 1: The AI concludes that M1 is part of my utility function and M2 and M3 are not. Since it is designed to maximize my utility, it constructs an environment in which M1 triumphs. For example, perhaps it installs a highly sophisticated filter that blocks out 90% of the Internet. Result: I get lots more high-quality work done on the novel. I miss the Internet, but the AI doesn't care, because that's the result of M2 and M2 isn't part of my utility function.
Case 2: The AI concludes that M3 and M2 are part of my utility function and M1 is not, so it finishes the novel itself and modifies the Internet to be even more compelling. I miss having the novel to work on, but again the AI doesn't care.
Case 3: The AI concludes that all three things are part of my utility function. It finishes the novel but doesn't tell me about it, thereby satisfying M3 (though I don't know it). It makes a few minor tweaks to my perceived environment, but mostly leaves them alone, since it is already pretty well balanced between M1 and M2 (which is not surprising, since I was responding to those mental structures when I constructed my current situation).

If I'm understanding you correctly, you're saying that I can't really know which of these results (or of countless other possibilities) will happen, but that whichever one it is, I should have high confidence that all other possibilities would by my own standards have been worse... after all, that's what it means to maximize my utility function.

Yes?

It seems to follow that if the AI has an added feature whereby I can ask it to describe what it's about to do before it does it and then veto doing it, I ought not invoke that feature. (After all, I can't make the result better, but I might make the result worse.)

Yes?

Comment author: nshepperd 31 January 2011 12:43:29AM 0 points [-]

Assuming you trust Omega to mean the same thing as you do when talking about your preferences and utility function, then yes. If the AI looks over your mind and optimizes the environment for your actual utility function (which could well be a combination of M1, M2 and M3), then any veto you do must make the result worse than the optimal one.

Of course, if there's doubt about the programming of the AI, use of the veto feature would probably be wise, just in case it's not a good genie.

Comment author: Zack_M_Davis 30 January 2011 07:07:20PM 0 points [-]

You seem to be imagining a relatively weak AI. For instance, given the vast space of possibilities, there are doubtlessly environmental tweaks that would result in more fun on the internet and more high-quality work on the novel. (This is to say nothing of more invasive interventions.)

The answer to your questions is yes: assuming the AI does what Omega says it does, you won't want to use your veto.

Comment author: TheOtherDave 30 January 2011 07:36:37PM 0 points [-]

Not necessarily weak overall, merely that it devotes relatively few resources to addressing this particular tiny subset of my preference-space. After all, there are many other things I care about more.

But, sure, a sufficiently powerful optimizer will come up with solutions so much better that it will never even occur to me to doubt that all other possibilities would be worse. And given a sufficiently powerful optimizer, I might as well invoke the preview feature if I feel like it, because I'll find the resulting preview so emotionally compelling that I won't want to use my veto.

That case obscures rather than illustrates the question I'm asking, so I didn't highlight it.

Comment author: katydee 30 January 2011 06:18:42PM 0 points [-]

Case 4: The AI makes tweaks to your current environment in order to construct it in accordance with your mental structures, but in a way more efficient than you could have in the first place.

Comment author: TheOtherDave 30 January 2011 07:40:52PM 0 points [-]

Sure. In which case I still noodle around on the Internet a bunch rather than work on my novel, but at least I can reassure myself that this optimally reflects my real preferences, and any belief I might have that I would actually rather get more work done on my novel than I do is simply an illusion.

Comment author: katydee 30 January 2011 10:38:46PM 0 points [-]

If those are, in fact, your real preferences, then sure.

Comment author: timtyler 30 January 2011 08:27:33PM * 0 points [-]

If you actually research this it quickly becomes clear that real world intelligences make decisions using much more complexity than a simple utility-maximizing algorithm.

I occasionally point out that you can model any computable behaviour using a utility-maximizing algorithm, provided you are allowed to use a partially-recursive utility function.

nshepperd comments on David Chalmers' "The Singularity: A Philosophical Analysis" - Less Wrong

Article Navigation

Comments (202)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (202)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login