= 27d567bc2d48d9f40e9ccc20ea370b15)
Vladimir_Nesov comments on Off Topic Thread: May 2009 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (71)
That's not true, e.g. the choice may be between UFAI agreeing to create a FAI that lets the UFAI have a little chunk of utility vs. getting nothing (becoming terminated). Allow a little cooperation. Still not a good idea, since UFAI may well discover a third option.
If you somehow reliably trapped a UFAI in a box, and then tried to coerce it to design an FAI, it would disguise the next version of itself as an FAI.
Seriously, if our strategy is to build something smarter than us, and then try to outsmart it, the best outcome we could reasonably hope for is that we never succeed in building the thing that is smarter than us. We need the thing that is smarter than us to be on our side by its very nature.
"Trapped", "coerced", "itself"? You should be more careful around these things, you seem to be giving your answers immediately on this not at all elementary problem. You didn't actually address my counterexample, and I already agreed with the general message of your second paragraph in the last sentence of the comment above.
Well, in attempting to address your counterexample, I had to guess what you meant, as it is not very clear. What situation do you envision in which the UFAI would expect to gain utility by building an FAI?
And it seems a little strange to accuse me of offering solutions before the problem is fully explored, when I was responding to a proposal for a solution (using UFAI to build FAI).
(Also, I have edited out a typo (repetition of the word "it") in my statement which you quoted.)
The situation I described: cooperation between FAI and UFAI. Two unrelated AIs are never truly antagonistic, so they have something to gain from cooperation.
The same problem on both accounts, confident assertions about a confusing issue. This happened twice in a row, because the discussion shared the common confusing topic, so it's not very surprising.
Unless you are actually saying that the way to get an UFAI to build an FAI is to build the FAI ourselves, locate the UFAI in a different universe, and have some sort of rift between the universe with contrived rules about what sort of interaction it allows, I still do not understand the situation you are talking about.
An AI that wants to tile the solar system with molecular smiley faces and an AI that wants to tile the solar system with paperclips are going to have conflicts. Either of them would have conflicts with an FAI that wants to use the resources of the solar system to create a rich life experience for humanity. Maybe these AI's are not what you call "unrelated", but if so, I doubt the UFAI and the FAI we want it to build can be considered to be unrelated.
Are you asking me to have less confidence in the difficulty of us outsmarting things that are smarter than us?
Among the two options "UFAI doesn't do anything, and so we terminate/won't build it", and "UFAI builds/explains FAI, and gets -- simplifying -- 1/100th of the universe", the second option is preferable to both us and the UFAI, and so if these are the only options, it'll take it.
Yes, I'm asking you to have less confidence in any conclusion you are drawing here. "Outsmarting" isn't exactly a tangible event.
First of all, the "won't build it" option does not make any sense. It is not like the UFAI is going to do anything before it exists.
So then, going back to the point that you claim doesn't address your counterexample, supposing we actually have the UFAI in a situation where we can terminate it at will, and it can't persuade anyone to free it, how are we supposed to verify that the thing it is helping us build is actually an FAI, (and not another UFAI with the same values) if we do not know how to build an FAI on our own? If we can't tell the difference, isn't that what the UFAI, with its values, should do?
That's why I never proposed that we ask the UFAI to create us an FAI; I proposed that we ask the UFAI to help us create an FAI. Doing something like proving a mathematical theorem for us, or creating something that fulfills a template we've created ourselves, leaves no room for betrayal, barring any "third options" like manipulating the EM waves produced by its CPU to send arbitrary messages over the Internet.
Suppose the UFAI figures out a fake theorem that it would like us to believe, because it would lead us down a path of creating an AI it likes. If we were to ask it to prove this fake theorem, it would give back to us something that looks very much like a proof, so that we would miss the point where it goes wrong. Maybe we require a machine verifiable proof, but it takes advantage of a flaw in our automatic verifiers, or the way we interpret the results. So how does it get us to ask about this fake theorem? It might manipulate its proof of a theorem we do ask about to inspire us to ask that question. It might respond to our request for a proof with, "Well, that is not quite right, but if you make these modifications...". Keep in mind, this is a hostile intelligence that is way beyond us. It will take any opportunity to subtly manipulate us that it gets. And these have only been the ideas that I, a mere human, could come up with.
I am not sure what sort of template you mean, but I suspect that it will have the same problem. Basically, room for the UFAI to use its superior intelligence to help us is room for betrayal.
You might be able to recognize the right solution (e.g. a theory) when you see it, while unable to generate it yourself (as fast). If you are sensitive enough to attempts of UFAI to confuse you into doing the wrong thing, just going with the deal may be the best option for it.
You decide whether to build something new before it exists, based on its properties. AI's decisions are such properties.
And if we are not sensitive enough, our molecules get reused for paperclips. You are talking about matching wits with something that is orders of magnitude smarter than us, thinks orders of magnitude faster than us, has a detailed model of our minds, and doesn't think we are worth the utilons it can build out of our quarks. Yes, we would think we recognize the right solution, that it is obvious now that it's been pointed out, and we would be wrong. An argument than a UFAI figured out would be persuasive to us is nowhere near as trustworthy as an argument we figure out ourselves. While we can be wrong about our own arguments, the UFAI will present arguments that we will be systematically wrong about in very dangerous ways.
And for our next trick, we will just ask Omega to build an FAI for us.
Can't we run a UFAI in a sandbox that prevents it from ever emitting more than a certain amount of information--and, especially, from discovering the nature of the hardware it runs on?
Not if we have the ability to let the UFAI out of the sandbox. See the AI-box experiment.
Here you start applying the structure of your choice to the black swan of UFAI's third option. Literally letting anything out is a trivial option, there are many others, some of which nobody thought of. Even if you can't let UFAI out of the box, that's still not enough to be safe, and so your argument is too weak to be valid.
That's the problem with third options -- you may not be as protected as you think you are.