mamert comments on Ghosts in the Machine - Less Wrong

18 Post author: Eliezer_Yudkowsky 17 June 2008 11:29PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (27)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: mamert 05 October 2015 03:59:33AM *  1 point [-]

And I'm disturbed by your dismissal.

Neural nets, etc, get surprisingly creative. The conflict between an AI's directives will be given high priority. Solutions not forbidden are fair game.

What judges the AI's choices? It would try to model the judgement function and seek maximums. Even to manipulate the development of the function by fabricating reports. Poison parts of its own 'understanding' to justify assigning low weights to them. And that's IF it is limited in its self-modification. If not, the best move would be to ignore inputs and 'return true'. All without a shred of malice.

It is not the idea of the threat, but of 'friendliness' in AI that feels ridiculous. At least until you define morality im mathematical terms. Till then, we have literal-minded genies.