= 27d567bc2d48d9f40e9ccc20ea370b15)
Lumifer comments on Open thread, September 8-14, 2014 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (295)
The contact between the FAIs is not a one-decision-to-fight-or-share deal. It's a process that will take some time and each party will have to take many decisions during that process. Besides, the payoff matrix is quite uncertain -- if one initially cooperates and one initially defects does the defecting one get more? No one knows. For example, the start of the hostilities between Hitler and Stalin was the case where Stalin (initially) cooperated and Hitler (initially) defected. The end result -- not so good for Hitler.
There are many options here -- fully cooperate (and potentially merge), fight till death, divide spheres of influence, set up a DMZ with shared control, modify self, etc.
The first interesting question is, I guess, how friendly to aliens will a FAI be? Will it perceive another alien FAI as an intolerable obstacle in its way to implement friendliness as it understands it?
More questions go along the lines of how likely it is that one FAI will be stronger (or smarter) than the other one. If they fight, what might it look like (assume interstellar distances and speed of light limits). How might an AI modify itself on meeting another AI, etc. etc.
As much as is reasonable in given situation. If it is stronger, and if conquering the other AI is a net gain, it will fight. If it is not stronger, or the peace could be more efficient than the war, it will try to negotiate.
The costs of peace will depend on the differences between those two AIs. "Let's both self-modify to become compatible" is one way to make peace, forever. It has some cost, but it also saves some cost. Agreeing to split the universe into two parts, each governed by one AI, also has some cost. Depending on specific numbers, the utility maximizing choice could be "winner takes all" or "let's split the universe" or "let's merge into one" or maybe something else I didn't think about.
The critical question is, whose utility?
The Aumann theorem will not help here since the FAIs will start with different values and different priors.
Each AI tries to maximize their own utility, of course. When they consider merging, they make an estimate: how much of the original utility can I expect to get after we both self-modify to maximize the new utility function.
Then each AI makes its own choice and the two choices might well turn out to be incompatible.
There is also the issue of information exchange -- basically, it will be hard for the two AIs to trust each other.