= 27d567bc2d48d9f40e9ccc20ea370b15)
Vladimir_Nesov comments on How Not to be Stupid: Adorable Maybes - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (54)
Over the last couple years I went from believing that statement to deeply doubting it. If you want a chess player that will win games by holding the opponents' kids hostage, sure, build a Bayesian optimizer. My personal feeling is that even an ordinary human modified to be deeply and genuinely driven by an explicit utility function would pose a substantial danger to this world. No need for AIs.
That is a right sentiment about strength: there are no simple rules, only goals, which makes a creative mind extremely dangerous. And we shouldn't build things like this without understanding what the outcome will be. This is one of the reasons it's important to understand human values in this light, to guard them from this destructive potential.
Whatever you want accomplished, whatever you want averted, instrumental rationality defines an optimal way of doing that (without necessarily giving the real-world means, that's a next step). If you really want life to continue as before, the correctly implemented explicit utility function for doing that won't lead a Bayesian optimizer to do something horrible. (Although inaction may be considered horrible in itself, where so much more could've been done.)