= 27d567bc2d48d9f40e9ccc20ea370b15)
Vladimir_Nesov comments on Value Deathism - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (118)
I understand the grandparent comment now. Open/closed distinction can in principle be extracted from values, so that values of the original agent only specify what kind of program the agent should self-improve into, while that program is left to deal with any potential observations. (It's not better to forget some component of values.)
I'm not sure I understand you. Values of the original agent specify a class of programs it can become. Which program of this class should deal with observations?
Forget? Is it about "too smart to optimize"? This meaning I didn't intend.
When computer encounters borders of universe, it will have incentive to explore every possibility that it is not true border of universe such as: active deception by adversary, different rules of game's "physics" for the rest of universe, possibility that its universe is simulated and so on. I don't see why it is rational for it to ever stop checking those hypotheses and begin to optimize universe.