= 27d567bc2d48d9f40e9ccc20ea370b15)
abramdemski comments on Toy model for wire-heading [EDIT: removed for improvement] - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (1)
I agree with your point as stated, but I think a sharper distinction between utility-maximizing and reward-maximizing reveals more alternatives.
A reward-maximizing agent attempts to predict A; D maximizes this predicted future A.
A utility-maximizing agent has direct access to A; D applies A to evaluate possible futures, and maximizes A.
In the first case, a superintelligent D would want to wrestle control of A and modify it.
In the second case, when D thinks about the planned modification of A, it evaluates this possible future using the current A. It sees that the current A does not value this future particularly highly. Therefore, it does not wirehead.