= 27d567bc2d48d9f40e9ccc20ea370b15)
niceguyanon comments on Open Thread, Aug. 22 - 28, 2016 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (67)
(1) Given: AI risk comes primarily from AI optimizing for things besides human values.
(2) Given: humans already are optimizing for things besides human values. (or, at least besides our Coherent Extrapolated Volition)
(3) Given: Our world is okay.^[CITATION NEEDED!]
(4) Therefore, imperfect value loading can still result in an okay outcome.
This is, of course, not necessarily always the case for any given imperfect value loading. However, our world serves as a single counterexample to the rule that all imperfect optimization will be disastrous.
(5) Given: A maxipok strategy is optimal. ("Maximize the probability of an okay outcome.")
(6) Given: Partial optimization for human values is easier than total optimization. (Where "partial optimization" is at least close enough to achieve an okay outcome.)
(7) ∴ MIRI should focus on imperfect value loading.
Note that I'm not convinced of several of the givens, so I'm not certain of the conclusion. However, the argument itself looks convincing to me. I’ve also chosen to leave assumptions like “imperfect value loading results in partial optimization” unstated as part of the definitions of those 2 terms. However, I’ll try and add details to any specific areas, if questioned.
3) World is OK with humans optimizing for the wrong things because humans eventually die and take their ideas with them good or bad. Power and wealth is redistributed. Humans get old, they get weak, they get dull, they lose interest. AI gets it wrong then well that's it.
Not necessarily, depends on your AI and how god-like it is.
In the XIX century you could probably make the same argument about corporations: once one corporation rises above the rest, it will use its power to squash competition and install itself as the undisputed economic ruler forever and ever. The reality turned out to be rather different and not for the lack of trying.
I hope you're right. I just automatically think that AI is going to be god-like by default.
Not just corporations; you could make the some argument for sovereign states, foundations, trusts, militaries, and religious orgs.
Weak argument is that corporations with their visions, charters, and mission statements are ultimately run by a meatbag or run jointly by meatbags that die/retire, at least that's how it currently is. You can't retain humans forever. Corporations lose valuable and capable employee brains over time and replace them with new brains, which maybe better or worse, but you certainly cant keep your best humans forever. Power is checked; Bill Gates plans his legacy, while Sumner Redstone is infirm with kids jockeying for power and Steve Jobs is dead.
Well, the default on LW is EY's FOOM scenario where an AI exponentially bootstraps itself into Transcendent Realms and, as you say, that's it. The default in the rest of the world... isn't like that.