= 27d567bc2d48d9f40e9ccc20ea370b15)
Vladimir_Nesov comments on The Domain of Your Utility Function - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (94)
You inevitably draw specific actions, so there is no escaping forming a preference over actions (a decision procedure, not necessarily preference over things that won't play), and "discarding too much" can't be an argument against the inevitable. (Not that I particularly espouse the form of preference being utility+prior.)
Sorry, I meant something like "whether there is a relatively simple decision algorithm with consistent preferences that we can extrapolate from that mess without discarding too much". If not, then a superintelligence might be able to extrapolate us, but until then we'll be stymied in our attempts to think rationally about large unfamiliar decisions.
Fair enough. Note that the superintelligence itself must be a simple decision algorithm for it to be knowably good, if that's at all possible (at the outset, before starting to process the particular data from observations), which kinda defeats the purpose of your statement. :-)
Well, the code for the seed should be pretty simple, at least. But I don't see how that defeats the purpose of my statement; it may be that short of enlisting a superintelligence to help, all current attempts to approximate and extrapolate human preferences in a consistent fashion (e.g. explicit ethical or political theories) might be too crude to have any chance of success (by the standard of actual human preferences) in novel scenarios. I don't believe this will be the case, but it's a possibility worth keeping an eye on.