= 27d567bc2d48d9f40e9ccc20ea370b15)
lessdazed comments on Wanted: backup plans for "seed AI turns out to be easy" - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (62)
Humans have dealt with the tragedy of the commons and we can't even merge resources and utility functions to become a new, larger entity!
We have dealt with TotC by imposing costs larger than the benefits that could be derived from abusing the commons.
The benefits an AI could derive from abusing the commons are possibly unlimited.
OK, so a reason a group of AIs wouldn't be able to do that is because the advantage of exploiting the commons might be nearly infinite. How likely is this?
If there were millions of AIs, what's a scenario in which one gets so much more powerful than all the others combined by striking first, despite all of them guarding against it?
As AIs can merge, I would think refusal to do so and combine utility functions might be a first sign of intent to defect, that's a warning humans never have.
Regardless, a million is a constant factor. Sufficient self-reinforcing development (as is kind of the point of seed AI) can outstrip any such factor. And the more self-reinforced the development of our AI pool becomes, the less relevant are "mere" constant factors.
I'm not saying it won't work, but I wouldn't like to bet on it.
Don't worry, I'm not saying it would work! We might put similar odds on it, or maybe not - less than .5 and more than .001, I'm not sure about what the full range of fleshing out the possible scenarios would look like, but there's probably a way in which I could fill in variables to end up with a .05 confidence in a specific scenario working out.