= 27d567bc2d48d9f40e9ccc20ea370b15)
Larks comments on Extraterrestrial paperclip maximizers - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (157)
"Fighting" is a narrow class of strategies, while in "trading" I include a strictly greater class of strategies, hence expectation of there being a better strategy within "trading".
But they'll be even better off without a fight, with staple maximizer surrendering most of its control outright, or, depending on disposition (preference) towards risk, deciding the outcome with a random number and then orderly following what the random number decided.
Isn't this a Hawk-Dove situation, where pre-committing to fight even if you'll probably lose could be in some AGI's interests, by deterring others from fighting them?
Threats are not made to be carried out. Possibility of actual fighting sets the rules of the game, worst-case scenario which the actual play will improve on, to an extent for each player depending on the outcome of the bargaining aspect of the game.
For a threat to be significant, it has to be believed. In the case of AGI, this probably means the AGI itself being unable to renege on the threat. If two such met, wouldn't fighting be inevitable? If so, how do we know it wouldn't be worthwhile for at least some AGIs to make such a threat, sometimes?
Then again, 'Maintain control of my current level of resources' could be a schelling point that prevents descent into conflict.
But it's not obvious why an AGI would choose to draw their line in the sand their though, when 'current resources plus epsilon% of the commons' is available. The main use of schelling points in human games is to create a more plausible threat, whereas an AGI could just show its source code.
An AGI won't turn itself into a defecting rock, when there is a possibility of pareto improvement over that.