Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Eugine_Nier comments on Robust Cooperation in the Prisoner's Dilemma - Less Wrong

69 Post author: orthonormal 07 June 2013 08:30AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (145)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eugine_Nier 12 June 2013 08:12:33AM 2 points [-]

This is true for meat-based evolution. In a software-based world, it can be trivial to prove what your source code is, and sometimes it's advantageous too.

So hide the malicious behavior in hardware or in the compiler.

Comment author: DanArmak 12 June 2013 09:42:23AM 3 points [-]

I was thinking of a scenario where the verifier controls the hardware and the compiler.

An em that's good at some kind of work might want to convince you to run copies of it

E.g. it sends you its source code to run on your own hardware because it wants to work for you.

Comment author: Eugine_Nier 15 June 2013 07:31:43AM -1 points [-]

So hide it in some other computer you interact with.

Comment author: DanArmak 15 June 2013 03:41:52PM 1 point [-]

Concerns about what I do on my own computer are separate from proofs about the software I send to you - they might still be valid, and if so, trusted and influential.

Yes, I could achieve malicious ends by outsmarting you: I could give you software that does exactly what it says (and a formal proof of that), but through its seemingly innocent interactions with other servers (which are perfectly valid under that proof), actually create a malicious effect you haven't foreseen. But that concern doesn't invalidate the usefulness of the proof.