= 27d567bc2d48d9f40e9ccc20ea370b15)
Stuart_Armstrong comments on Siren worlds and the perils of over-optimised search - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (411)
Under what circumstances? There are situations - torture, seduction, a particular way of asking the question - that can make any brain give any answer. Defining "non-coercive yet informative questioning" about a piece of software (a simulated brain) is... hard. AI hard, as some people phrase it.
Why would that .be more of a problem for an AI than a human?
? The point is that having a simulated brain and saying "do what this brain approves of" does not make the AI safe, as defining the circumstance in which the approval is acceptable is a hard problem.
This is a problem for us controlling an AI, not a problem for the AI.
I still don't get it. We assume acceptability by default. We don't constantly stop and ask "Was that extracted under torture".
I do not understand your question. It was suggested that an AI run a simulated brain, and ask the brain for approval for doing its action. My point was that "ask the brain for approval" is a complicated thing to define, and puts no real limits on what the AI can do unless we define it properly.
Ok. You are assuming the superintelligent .AI will pose the question in a dumb way?
No, I am assuming the superintelligent AI will pose the question in the way it will get the answer it prefers to get.
Oh, you're assuming it's malicious. In order to prove...?
No, not assuming it's malicious.
I'm assuming that it has some sort of programming along the lines of "optimise X, subject to the constraint that uploaded brain B must approve your decisions."
Then it will use the most twisted definition of "approve" that it can find, in order to best optimise X.
The programme it with:
Prime directive - interpret all directives according to your makers intentions.
Secondary directive - do nothing that goes against the uploaded brain
Tertiary objective - optimise X.