Wei_Dai comments on indexical uncertainty and the Axiom of Independence - Less Wrong

9 Post author: Wei_Dai 07 June 2009 09:18AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (70)

You are viewing a single comment's thread. Show more comments above.

Comment author: Vladimir_Nesov 08 June 2009 09:15:10PM *  3 points [-]

Together with Eliezer's idea that agents who know each other's source code ought to play cooperate in one-shot PD, doesn't it imply that all sufficiently intelligent and reflective agents across all possible worlds should do a global trade and adopt a single set of preferences that represents a compromise between all of their individual preferences?

It does, and I discussed that here. An interesting implication that I noticed a few weeks back is that an UFAI would want to cooperate with a counterfactual FAI, so we get a slice of the future even if we fail to build FAI, depending on how probable it was that we would be able to do that. A Paperclip maximizer might wipe out humanity, then catch up on its reflective consistency, look back, notice that there was a counterfactual future where a FAI is built, allot some of the collective preference to humanity, and restore it from the info remaining after the initial destruction (effectively constructing a FAI in the process). (I really should make a post on this. Some of the credit due to Rolf Nelson for UFAI deterrence idea.)

Comment author: Wei_Dai 08 June 2009 10:00:41PM *  1 point [-]

Ok, so I see that probability plays a role in determining one's "bargaining power", which makes sense. We still need a rule that outputs a compromise set of preferences when given a set of agents, their probabilities, individual preferences, and resources as input, right? Does the rule need to be uniquely fair or obvious, so that everyone can agree to it without discussion? Do you have a suggestion for what this rule should be?

Edit: I see you've answered some of my questions already in the other reply. This is really interesting stuff!