= 27d567bc2d48d9f40e9ccc20ea370b15)
AlexMennen comments on Proper value learning through indifference - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (50)
Problem: not only will such an AI not resist its utility function being altered by you, it will also not resist its utility function being altered by a saboteur or by accident. I don't think I'd want to call this proposal a form of value learning, since it does not involve the AI trying to learn values, and instead just makes the AI hold still while values are force-fed to it.
The AI will not resist its values being changed in the particular way that is specified in to trigger a U transition. It will resist other changes of value.
That's true; it will resist changes to its "outer" utility function U. But it won't resist changes to its "inner" utility function v, which still leaves a lot of flexibility, even though that isn't its true utility function in the VNM sense. That restriction isn't strong enough to avoid the problem I pointed out above.
I will only allow v to change if that change will trigger the "U adaptation" (the adding and subtracting of constants). You have to specify what processes count as U adaptations (certain types of conversations with certain people, eg) and what doesn't.
Oh, I see. So the AI simply losing the memory that v was stored in and replacing it with random noise shoudn't count as something it will be indifferent about? How would you formalize this such that arbitrary changes to v don't trigger the indifference?
By specifying what counts as an allowed change in U, and making the agent in to a U maximiser. Then, just as standard maximises defend their utilities, it should defend U(un clubbing the update, and only that update)
You can't always solve human problems with AI design.
I'm not sure what you mean. The problem I was complaining about is an AI design problem, not a human problem.
No, I would say that if you start entering false utility data into the AI and it believes you, because after all it was programmed to be indifferent to new utility data, that's your problem.
If the AI's utility function changes randomly for no apparent reason because the AI has litterally zero incentive to make sure that doesn't happen, then you have an AI design problem.
It didn't change for no reason. It changed because someone fed new data into the AI's utility-learning algorithm which made it change. Don't give people root access if you don't want them using it!
Being changed by an attacker is only one of the scenarios I was suggesting. And even then, presumably you would want the AI to help prevent them from hacking its utility function if they aren't supposed to have root access, but it won't.
Anyway, that problem is just a little bit stupid. But you can also get really stupid problems, like the AI wants more memory, so it replaces its utility function with something more compressible so that it can scavange from the memory where its utility function was stored.