whowhowho comments on Decision Theory FAQ - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (467)
Let's say the paperclipper reaches the point where it considers making people suffer for the sake of paperclipping. DP's point seems to be that either it fully understands suffering--in which case, it realies that inflicing suffering is wrong --or it it doesn't fully understand. He sees a conflict between superintelligence and ruthlessness -- as a moral realist/cognitivist would
is that full understanding.?.
ETA: Unless there is -- eg. what qualiaphiles are always banging on about; what it feels like. That the clipper can conjectures that are true by correspondence , that it can narrow down possible universes, that it can predict, are all necessary criteria for full understanding. It is not clear that they are sufficient. Clippy may be able to figure out an organisms response to pain on a basis of "stimulus A produces response B", but is that enough to tell it that pain hurts ? (We can make guesses about that sort of thing in non-human organisms, but that may be more to do with our own familiarity with pain, and less to do with acts of superintelligence). And if Clippy can't know that pain hurts, would Clippy be able to work out that Hurting People is Wrong?
further edit; To put it another way, what is there to be moral about in a qualia-free universe?
As Kawoomba colorfully pointed out, clippy's subroutines simulating humans suffering may be fully sentient. However, unless those subroutines have privileged access to clippy's motor outputs or planning algorithms, clippy will go on acting as if he didn't care about suffering. He may even understand that inflicting suffering is morally wrong--but this will not make him avoid suffering, any more than a thrown rock with "suffering is wrong" painted on it will change direction to avoid someone's head. Moral wrongness is simply not a consideration that has the power to move a paperclip maximizer.