Perplexed, have you come across the decision theory posts here yet? You'll find them pretty interesting, I think.
Yes, I have read them, and commented on them. Negatively, for the most part. If any of these ideas are ever published in the peer reviewed literature, I will be both surprised and eager to read more.
there are decision theories for an AI which do strictly better than the usual causal decision theory, without being exploitable. Two of these would cooperate with each other on the PD, given a chance to communicate beforehand.
I think that you may have been misled by marketing hype. Even the proponents of those theories admit that they do not do strictly better (or at least as good) on all problems. They do better on some problems, and worse on others. Furthermore, sharing source code only provides a guarantee that the observed source is current if that source code cannot be changed. In other words, an AI that uses this technique to achieve commitment has also forsaken (at least temporarily) the option of learning from experience.
I am intrigued by the analogy between these acausal decision theories and the analysis of Hamilton's rule in evolutionary biology. Nevertheless, I am completely mystified as to the motivation that the SIAI has for pursuing these topics. If the objective is to get two AIs to cooperate with each other there are a plethora of ways to do that already well known in the game theory canon. An exchange of hostages, for example, is one obvious way to achieve mutual enforceable commitment. Why is there this fascination with the bizarre here? Why so little reference to the existing literature?
So far as I understand the situation, the SIAI is working on decision theory because they want to be able to create an AI that can be guaranteed not to modify its own decision function.
There are circumstances where CDT agents will self-modify to use a different decision theory (e.g. Parfit's Hitchhiker). If this happens (they believe), it will present a risk of goal-distortion, which is unFriendly.
Put another way: the objective isn't to get two AIs to cooperate, the objective is to make it so that an AI won't need to alter its decision function in order to...
A monthly thread for posting rationality-related quotes you've seen recently (or had stored in your quotesfile for ages).