mamert comments on Ethical Injunctions - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (67)
There is something I don't understand about the "fooling programmers -> shutdown" idea - how "It looks like it should be possible"
Am I correct assuming this requires an AI that:
Implements changes in discrete iterations
Understands how it itself works, both on low and very high levels (getting fractal here)
Can do the same for its more advanced iteration
Monitors the more advanced iteration (a simulation or a live deployment) for a certain behavior, preferably in real time or faster
Has the concept of deception (shutdown condition, AND a salient factor in other calculations)
At all times allows itself to be killed by its previous iteration (accidental Robin Hood/Friar Tuck hacks come to mind)
Has a full-proof definition of "deceiving programmers" which:
a) remains consistent with programmers' intentions across iterations, immune to drift
b) remains applicable across iterations, cannot be bypassed or cheated
c) allows n to give different output than in n-1 (else why bother changing?)
d) does not have too significant an impact on improvement process
e) has a satisfactory solution for the concept of manipulating people, which somehow doesn't paralyze the system
situation: output data that is true but would be misinterpreted by / cause a glitch in some other system the results will be fed to, leading people to act upon the wrong results of in a way desirable by the AI
"It is not deceit if the format change is communicated along with the output data. Or somewhere in the status update PDFs sent 40 times a second."
"this output is not false, it is merely unrelated"
"but that would make the programmers' business competitors draw wrong conclusions... wait, that is allowed? Moving non-programmers to Resource.Actuator category, calculating new self-improvement strategy."