= 27d567bc2d48d9f40e9ccc20ea370b15)
MichaelGR comments on Rationality Quotes: November 2010 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (354)
If you can't tell whose side someone is on, they are not on yours. -Warren E. Buffett
If after 1/2 hr of poker you can't tell who's the patsy, it's you. - Charles T. Munger
Sounds like one for the quotes page for "Default to Good" at TV Tropes. (Link omitted due to time hazard.)
Wouldn't this be a problem for tit for tat players going up against other tit for tat players (but not knowing the strategy of their opponent)?
Only if it's common knowledge that both players are human.
ETA: Since I got downvoted, maybe I wasn't being clear. I think that the Warren Buffett quote applies to human psychology more than to game theory in general. If outright deception were easy, it would probably become a good strategy to keep your allies in some doubt about your intentions, as a bargaining chip. But we humans don't seem to be good at pulling that off, and so ambivalence is a strong signal of opposition.
Now that you have clarified, I wish I could downvote a second time.
Tit-for-tat is a good strategy in the iterated prisoner's dilemma regardless of whether the players are human and regardless of whether the other player is "on your side". In fact, it is pretty much taken for granted that there are no sides in the PD. Dre was downvoted by me for a complete misunderstanding of how Tit-for-tat relates to "sides". You were downvoted for continuing the confusion.
Oh, you're right- my response would have made sense talking about players in a one-shot PD with communication beforehand, but it's a non sequitur to Dre's mistaken comment. Don't know how I missed that.
Upvoted, but even with communication beforehand, the rational move in a one-shot PD is to defect. Unless there is some way to make binding commitments, or unless there is some kind of weird acausal influence connecting the players. Regardless of whether the other player is human and rational, or silicon and dumb as a rock.
Taboo "rational".
Acausal control is not something additional, it's structure that already exists in a system if you know where to look for it. And typically, it's everywhere, to some extent.
Highest-scoring move, adjective applied to the course that maximises fulfillment of desires.
The best move in a one-shot PD is to defect against a cooperator.
With no communication or precommitment, and with the knowledge that it is a one-shot PD, the overwhelming outcome is both defect. Adding communication to the mix creates a non-zero chance you can convince your opponent to cooperate - which increases the utility of defecting.
There is a question of what will actually happen, but also more relevant questions of what will happen if you do X, for various values of X. If you convince the opponent to cooperate, it's one thing, not related to the case of convincing your opponent to cooperate if you cooperate.
Determine what kinds of control influence your opponent, appear to also be influenced by the same, and then defect when they think you are forced into cooperating because they are forced into cooperating?
Is that a legitimate strategy, or am I misunderstanding what you mean by convincing your opponent to cooperate if you cooperate?
Perplexed, have you come across the decision theory posts here yet? You'll find them pretty interesting, I think.
LW Wiki for the Prisoner's Dilemma
LW Wiki for timeless decision theory (start with the posts- Eliezer's PDF is very long and spends more time justifying than explaining).
Essentially, this may be beyond the level of humans to implement, but there are decision theories for an AI which do strictly better than the usual causal decision theory, without being exploitable. Two of these would cooperate with each other on the PD, given a chance to communicate beforehand.
Yes, I have read them, and commented on them. Negatively, for the most part. If any of these ideas are ever published in the peer reviewed literature, I will be both surprised and eager to read more.
I think that you may have been misled by marketing hype. Even the proponents of those theories admit that they do not do strictly better (or at least as good) on all problems. They do better on some problems, and worse on others. Furthermore, sharing source code only provides a guarantee that the observed source is current if that source code cannot be changed. In other words, an AI that uses this technique to achieve commitment has also forsaken (at least temporarily) the option of learning from experience.
I am intrigued by the analogy between these acausal decision theories and the analysis of Hamilton's rule in evolutionary biology. Nevertheless, I am completely mystified as to the motivation that the SIAI has for pursuing these topics. If the objective is to get two AIs to cooperate with each other there are a plethora of ways to do that already well known in the game theory canon. An exchange of hostages, for example, is one obvious way to achieve mutual enforceable commitment. Why is there this fascination with the bizarre here? Why so little reference to the existing literature?
So far as I understand the situation, the SIAI is working on decision theory because they want to be able to create an AI that can be guaranteed not to modify its own decision function.
There are circumstances where CDT agents will self-modify to use a different decision theory (e.g. Parfit's Hitchhiker). If this happens (they believe), it will present a risk of goal-distortion, which is unFriendly.
Put another way: the objective isn't to get two AIs to cooperate, the objective is to make it so that an AI won't need to alter its decision function in order to cooperate with another AI. (Or any other theoretical bargaining partner.)
Does that make any sense? As a disclaimer, I definitely do not understand the issues here as well as the SIAI folks working on them.
I don't think that's quite right- a sufficiently smart Friendly CDT agent could self-modify into a TDT (or higher decision theory) agent without compromising Friendliness (albeit with the ugly hack of remaining CDT with respect to consequences that happened causally before the change).
As far as I understand SIAI, the idea is that decision theory is the basis of their proposed AI architecture, and they think it's more promising than other AGI approaches and better suited to Friendliness content.
Not to me. But a reference might repair that deficiency on my part.
Do you have an example of a problem on which CDT or EDT does better than TDT?
I have yet to see a description of TDT which allows me to calculate what TDT does on an arbitrary problem. But I do know that I have seen long lists from Eliezer of problems that TDT does not solve that he thinks it ought to be improved so as to solve.
Not necessarily. Various decision theories can come into play here. It depends precisely on what you mean by the prisoner's paradox. If you are playing a true one shot where you have no information about the entity in question then that might be true. But if you are playing a true one shot where you each before making the decision have each player have access to the other player's source code then defecting may not be the best solution. Some of the decision theory posts have discussed this. (Note that knowing each others' source code is not nearly as strong an assumption as it might seem since one common idea in game theory is to look at what game theory occurs when people know when the other players know your strategy. (I'm oversimplifying some technical details here. I don't fully understand all the issues. I'm not a game theorist. Add any other relevant disclaimers.))
No one on this thread has mentioned a "prisoner's paradox". We have been discussing the Prisoner's Dilemma, a well known and standard problem in game theory which involves two players who must decide without prior knowledge of the other player's decision.
A different problem in which neither player is actually making a decision, but instead is controlled by a deterministic algorithm, and in which both players, by looking at source, are able to know the other's decision in advance, is certainly an interesting puzzle to consider, but it has next to nothing in common with the Prisoner's Dilemma besides a payoff matrix.
Prisoner's paradox is another term for the prisoner's dilemma. See for example this Wikipedia redirect. You may want to reread what I wrote in that light. (Although there's some weird bit of illusion of transparency going on here in that part of me has a lot of trouble understanding how someone wouldn't be able to tell from context that they were the same thing.)
No. The problem of what to do is actually closely related when one has systems which are able to understand each others source code. It is in fact related to the problem of iterating the problem.
In general, given no information, the problem still has relevant decision theoretic considerations.
I'm curious why you assert this. Game theorists have a half dozen or so standard simple one-shot two person games which they use to illustrate principles. PD is one, matching pennies is another, Battle of the Sexes, Chicken, ... the list is not that long.
They also have a handful of standard ways of taking a simple one-shot game and turning it into something else - iteration is one possibility, but you can also add signaling, bargaining with commitment, bargaining without commitment but with a correlated shared signal, evolution of strategies to an ESS, etc. I suppose that sharing source code can be considered yet another of these basic game transformations.
Now we have the assertion that for one (PD is the only one?) of these games, one (iteration is the only one?) of these transformations is closely related to this new code-sharing transformation. Why is this assertion made? Is there some kind of mathematical structure to this claimed relationship? Some kind of proof? Surely there is more evidence for this claimed relationship than just pointing out that both transformations yield the same prescription - "cooperate" - when there are only two possible prescriptions to choose among.
Is the code-sharing version of Chicken also closely related to the iterated version? How about Battle of the Sexes?