= 27d567bc2d48d9f40e9ccc20ea370b15)
Warrigal comments on Off Topic Thread: May 2009 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (71)
If we come up with a strong AI that we suspect is un-Friendly, should we use it to help us create Friendly AI? (Perhaps by playing a single game of 20 Questions, which has probably been played enough times that every possible sequence of yes-or-no answers has come up?)
An unfriendly AI is unfriendly because it maximizes a utility function that does not represent our values, and it will act against our values whenever that increases its utility function. It would not help us create a Friendly AI, because that would be creating a powerful force that acts in the interests of our values, and would act to decrease the unfriendly utility function whenever that advances our values; that is, it would be contrary to the goals of the unfriendly AI.
That's not true, e.g. the choice may be between UFAI agreeing to create a FAI that lets the UFAI have a little chunk of utility vs. getting nothing (becoming terminated). Allow a little cooperation. Still not a good idea, since UFAI may well discover a third option.
If you somehow reliably trapped a UFAI in a box, and then tried to coerce it to design an FAI, it would disguise the next version of itself as an FAI.
Seriously, if our strategy is to build something smarter than us, and then try to outsmart it, the best outcome we could reasonably hope for is that we never succeed in building the thing that is smarter than us. We need the thing that is smarter than us to be on our side by its very nature.
"Trapped", "coerced", "itself"? You should be more careful around these things, you seem to be giving your answers immediately on this not at all elementary problem. You didn't actually address my counterexample, and I already agreed with the general message of your second paragraph in the last sentence of the comment above.
Well, in attempting to address your counterexample, I had to guess what you meant, as it is not very clear. What situation do you envision in which the UFAI would expect to gain utility by building an FAI?
And it seems a little strange to accuse me of offering solutions before the problem is fully explored, when I was responding to a proposal for a solution (using UFAI to build FAI).
(Also, I have edited out a typo (repetition of the word "it") in my statement which you quoted.)
The situation I described: cooperation between FAI and UFAI. Two unrelated AIs are never truly antagonistic, so they have something to gain from cooperation.
The same problem on both accounts, confident assertions about a confusing issue. This happened twice in a row, because the discussion shared the common confusing topic, so it's not very surprising.
Unless you are actually saying that the way to get an UFAI to build an FAI is to build the FAI ourselves, locate the UFAI in a different universe, and have some sort of rift between the universe with contrived rules about what sort of interaction it allows, I still do not understand the situation you are talking about.
An AI that wants to tile the solar system with molecular smiley faces and an AI that wants to tile the solar system with paperclips are going to have conflicts. Either of them would have conflicts with an FAI that wants to use the resources of the solar system to create a rich life experience for humanity. Maybe these AI's are not what you call "unrelated", but if so, I doubt the UFAI and the FAI we want it to build can be considered to be unrelated.
Are you asking me to have less confidence in the difficulty of us outsmarting things that are smarter than us?
Among the two options "UFAI doesn't do anything, and so we terminate/won't build it", and "UFAI builds/explains FAI, and gets -- simplifying -- 1/100th of the universe", the second option is preferable to both us and the UFAI, and so if these are the only options, it'll take it.
Yes, I'm asking you to have less confidence in any conclusion you are drawing here. "Outsmarting" isn't exactly a tangible event.
First of all, the "won't build it" option does not make any sense. It is not like the UFAI is going to do anything before it exists.
So then, going back to the point that you claim doesn't address your counterexample, supposing we actually have the UFAI in a situation where we can terminate it at will, and it can't persuade anyone to free it, how are we supposed to verify that the thing it is helping us build is actually an FAI, (and not another UFAI with the same values) if we do not know how to build an FAI on our own? If we can't tell the difference, isn't that what the UFAI, with its values, should do?
Can't we run a UFAI in a sandbox that prevents it from ever emitting more than a certain amount of information--and, especially, from discovering the nature of the hardware it runs on?
Not if we have the ability to let the UFAI out of the sandbox. See the AI-box experiment.
Here you start applying the structure of your choice to the black swan of UFAI's third option. Literally letting anything out is a trivial option, there are many others, some of which nobody thought of. Even if you can't let UFAI out of the box, that's still not enough to be safe, and so your argument is too weak to be valid.
That's the problem with third options -- you may not be as protected as you think you are.