= 27d567bc2d48d9f40e9ccc20ea370b15)
red75 comments on Value Deathism - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (118)
Direct question. I cannot infer answer from you posts. If human values do not exist in closed form (i.e. do include updates on future observations including observations which in fact aren't possible in our universe), then is it better to have FAI operating on some closed form of values instead?
I don't understand the question. Unpack closed form/no closed form, and where updating comes in. (I probably won't be able to answer, since this deals with observations, which I don't understand still.)
Then it seems better to demonstrate it on toy model as I've done for no closed form already.
One way I can think of to describe closed/no closed distinction is that latter does require unknown amount of input to be able to compute final/unchanging ordering over (internal representations of) world-states, former doesn't require input at all or requires predictable amount of input to do the same.
Another way to think about value with no closed form is that it gradually incorporates terms/algorithms acquired/constructed from environment.
I understand the grandparent comment now. Open/closed distinction can in principle be extracted from values, so that values of the original agent only specify what kind of program the agent should self-improve into, while that program is left to deal with any potential observations. (It's not better to forget some component of values.)
I'm not sure I understand you. Values of the original agent specify a class of programs it can become. Which program of this class should deal with observations?
Forget? Is it about "too smart to optimize"? This meaning I didn't intend.
When computer encounters borders of universe, it will have incentive to explore every possibility that it is not true border of universe such as: active deception by adversary, different rules of game's "physics" for the rest of universe, possibility that its universe is simulated and so on. I don't see why it is rational for it to ever stop checking those hypotheses and begin to optimize universe.