= 27d567bc2d48d9f40e9ccc20ea370b15)
V_V comments on AI caught by a module that counterfactually doesn't exist - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (22)
Isn't that essentially a false beliefs about one's own preferences?
I mean, the AI "true" VNM utility function, to the extent that it has one, is going to be different than the utility function the AI think reflectively it has. In principle the AI could find out the difference and this could cause it to alter its behavior.
Or maybe not, I don't have a strong intuition about this at the moment. But if I recall correctly, in the previous work on corrigibility (I didn't read the last version you linked yet), Soares was thinking of using causal decision nodes to implement utility indifference for the shutdown problem. This effectively introduces false beliefs into the agent, as the agent is mistaken about what causes the button to be pressed.
My preferred interpretation of that particular method is not "the agent has false beliefs," but instead "the agent cares both about the factual and the counterfactual worlds, and is trying to maximize utility in both at once." That is, if you were to cry
It will look at you funny, and say "Because I care about that counterfactual world. See? It says so right here in my utility function." It knows the world is counterfactual, it just cares about "what would have happened" anyway. (Causal decision nodes are used to formalize "what would have happened" in the agent's preferences, this says nothing about whether the agent uses causal reasoning when making decisions.)
This greatly clarified the distinction for me. Well done.
Makes sense.
No. It's an adjusted preference that functions in practice just like a false belief.