= 27d567bc2d48d9f40e9ccc20ea370b15)
Unknowns comments on AI cooperation in practice - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (157)
In the first version, are you sure? I gave an argument that A will output 1, have you looked at it?
In the second version, you confuse truth and provability. If both programs output 0, that doesn't imply they can prove that their output will be the same.
I've looked at the argument, and it seems to me that Tim Tyler is right, the first case depends on the implementation of "proves (code, maxsteps)."
Regarding the second case, suppose A runs through checking for proofs, and doesn't find any. It then is about to output a 0. But in this case, it is impossible that B should output a 1, because then B would have a proof that A and B output the same thing, and this would be false. Therefore B also must output a 0. Therefore A and B output the same thing. So we should add a line of code to A to express this: "If A is about to output 0, then output 1, because it is proved (by the above reasoning) that A and B output the same thing." The same code should be included in B, and so both will output 1.
Your proposed change will make A always output 1, and thus become a rather poor player in the Prisoner's Dilemma. You don't want to cooperate with everybody.
However, B will as well, and you do want to cooperate with everybody who cooperates with you, so my proposed change works very well.
Not if they will cooperate regardless of what you do. Then you can win more by defecting.
A knows B's source code, and so it knows that B will only output a 1 if A and B output the same thing, i.e. only if A and B both output 1.
No, A doesn't know that because it doesn't know B's proof checker is correct.
You really have to be triple extra careful when talking about this stuff. I'm trying to.
Yes, I see that.
I replied here.
I changed my mind anyway.