= 27d567bc2d48d9f40e9ccc20ea370b15)
RobinZ comments on Fusing AI with Superstition - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (75)
There have already in this thread been a lot of problems listed with this. I'm going to add just two more: consider an otherwise pretty friendly AI that is curious about the universe and wants to understand the laws of physics. No matter how much the AI learns, it will conclude that it and humans misunderstand the basic laws of physics. The AI will likely spend tremendous resources trying to understand just what is wrong with its understanding. And given the prior of 1, it will never resolve this issue.
Consider also the same scenario but if there's an otherwise potentially friendly second AI that finds out about the way this other AI has been programmed. If this AI is at all close to what humans are like (again it is a mildly friendly AI) it will become paranoid about the possibility that there's some similar programming issue in it. It might also use this as strong evidence that humans are jerks. The second AI isn't going remain friendly for very long.
For your second AI, it is worth distinguishing between "friendly" and "Friendly" - it is Friendly, in the sense that it understands and appreciates the relatively narrow target that is human morality, it just is unimpressed with humans as allies.
That's a valid distinction. But from the perspective of serious existential risks, an AI that has a similar morality but really doesn't like humans has almost as much potential existential risk as an Unfriendly AI.
I agree.