JGWeissman comments on Sneaky Strategies for TDT - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (22)
I think that merely noting that if the TDT agent had the goal of not being outperformed by an agent with another decision theory, it could achieve it is enough to undermine Problem 1 as a criticism of TDT. If it predicts that undermining a competitor is of sufficient instrumental value to offset the loss of immediate direct rewards of terminal value, then it will undermine the competitor. If it doesn't make the prediction (correctly), then it is rational to seek the greater reward for itself, even if this helps another agent even more.