You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Vaniver comments on xkcd on the AI box experiment - Less Wrong Discussion

15 Post author: FiftyTwo 21 November 2014 08:26AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (229)

You are viewing a single comment's thread. Show more comments above.

Comment author: Vaniver 21 November 2014 10:33:04PM 0 points [-]

Wouldn't a trivial "way to repair this obstacle" be for the agent to appear stupid enough to be credible?

What do you mean by 'appear' here? I know how to observe a real agent and think "hmm, this person will punish me without reflectively considering whether or not punishing me advances their interests," but I don't know how to get that impression about a hypothetical agent.

Comment author: shminux 21 November 2014 10:39:30PM *  0 points [-]

I don't understand your distinction between real and hypothetical here. Your first sentence was about a hypothetical "real" agent, right? What is the hypothetical "hypothetical" agent you describe in the second part?

Comment author: Vaniver 22 November 2014 01:02:00AM 0 points [-]

I don't understand your distinction between real and hypothetical here.

Basically, my understanding of acausal trades is "ancestor does X because of expectation that it will make descendant do Y, descendant realizes the situation and decides to do Y because otherwise they wouldn't have been made, even though there's no direct causal effect."

If you exist simultaneously with another agent (the 'real agent' from the grandparent), you can sense how they behave and they can trick you by manipulating what you sense. (The person might reflectively consider whether or not to punish you, and decide the causal link to their reputation is enough justification, even though there's no causal link to the actions you took, but try to seem unthinking so you will expect they'll always do that.)

If you're considering hypothetical descendants (the 'hypothetical agent' from the grandparent), though, it's not clear to me how to reason about their appearance to you now, and particular any attempts they make to 'appear' to be stupid. But now that I think about it more, I think I was putting too much intentionality into 'appear'- hypothetical agent A can't decide how I reason about it, but I can reason about it incorrectly or incompletely and thus it appears to be something it isn't.

Comment author: shminux 22 November 2014 02:05:47AM 0 points [-]

As far as I understand Eliezer's point, the "acausal" part is irrelevant, the same issue of trusting that another agent really means what it says and will not change its mind later comes up, anyway. I could easily be wrong, though.