So far as I understand the situation, the SIAI is working on decision theory because they want to be able to create an AI that can be guaranteed not to modify its own decision function.
There are circumstances where CDT agents will self-modify to use a different decision theory (e.g. Parfit's Hitchhiker). If this happens (they believe), it will present a risk of goal-distortion, which is unFriendly.
Put another way: the objective isn't to get two AIs to cooperate, the objective is to make it so that an AI won't need to alter its decision function in order to cooperate with another AI. (Or any other theoretical bargaining partner.)
Does that make any sense? As a disclaimer, I definitely do not understand the issues here as well as the SIAI folks working on them.
There are circumstances where CDT agents will self-modify to use a different decision theory (e.g. Parfit's Hitchhiker).
Does that make any sense?
Not to me. But a reference might repair that deficiency on my part.
A monthly thread for posting rationality-related quotes you've seen recently (or had stored in your quotesfile for ages).