Comment on Open Thread June 2010, Part 3

Wed, 16 Jun 2010 17:28:50 GMT

We should assign some substantial probability to getting some weighting of our preference (from bargaining with transparency, acausal trade, altruistic brain emulations, etc).

Game-theoretic considerations are only relevant when you have non-trivial control, not when your atoms are used for something else. If singleton's preference gives some weight to your preference, this is a case of having control directly through the singleton's preference, but the origin of this control is not game-theoretic. If the singleton's preference has sympathy for your preference, your explicit instantiation in the world doesn't need to have any control, in order to win through the implicit control via singleton's preference.

Game-theoretic aggregation, on the other hand, doesn't work by influence on other agent's preference. You only get your slice of the world because you already control it. Another agent may perform trade, but this is trade of control, rearranging what specifically each of you controls, without changing your preferences.

I assume that control will be winner-takes-all, so preferences of other agents existing at the time only matter if the winner's preference directly pays to their preferences any attention, but not if they had some limited control from the start.

If a moderate weighting of our preferences gets most of the potential utility, then the expected utility of inhuman AIs getting powerful won't be astronomically less than the expected utility of, e.g. a 'Friendly AI' acting on idealized human preferences.

My point is that inhuman AI may give no weight to our preference, while FAI may give at least some weight to everyone's preference. Game-theoretic trade won't matter here because agents other than the singleton have no control to bargain with. FAI gives weight to other preferences not because of trade, but by construction from the start, even if people it gives weight to don't exist at all (FAI giving them weight in optimization might cause them to appear, or a better event at least as good from their perspective).

Discuss