Comment Permalink

Jon Garcia3y50

This might have a better chance of working if you give the AI strong inductive biases to perceive rewarding stimuli not as intrinsically rewarding in themselves, but rather as evidence of something of value happening that merely generates the stimulus. We want the AI to see smiles as good things, not because upwardly curving mouths are good, but because they are generated by good mental states that we actually value.

For that, it would need a generative model (likelihood function) whose hidden states (hypotheses) are mapped to its value function, rather than mapping the sensory data (evidence) directly. The latter type of feed-forward mapping, which most current deep learning is all about, can only offer cheap heuristics, as far as I can tell.

The AI should also priviledge hypotheses about the true nature of the value function where the mental/physiological states of eudaimonic agents are the key predictors. With your happy-video example, the existence of the video player and video files is sufficient to explain its training data, but that hypothesis (R_2) does not involve other agents. R_1, on the other hand, does assume that the happiness of other agents is what predicts both its input data and its value labels, so it should be given priority. (For this to work, it would also need a model of human emotional expression to compare its sensory data against in the likelihood function.)

Reply

Stuart_Armstrong3y30

Thanks - this feels somewhat similar to my vague idea of "define wireheading, tell AI to avoid situations like that".

Reply

See in context

24 Value extrapolation partially resolves symbol grounding

by Stuart_Armstrong

12th Jan 2022

AI Alignment Forum

1 min read

10

24 Ω 12

Take the following AI, trained on videos of happy humans:

Since we know about AI wireheading, we know that there are at least two ways the AI could interpret its reward function^[1]: either we want it to make more happy humans (or more humans happy); call this . Or we want it to make more videos of happy humans; call this $R_{2}$ .

We would want the AI to learn to maximise $R_{1}$ , of course. But even without that, if it generates $R_{1}$ as a candidate and applies a suitable diminishing return to all its reward functions, then we will have a positive outcome - the AI may fill the universe with videos of happy humans, but it will also act to make us happy.

Thus solving value extrapolation will solve symbol grounding, at least in part.

This is a massive over-simplification of what would be needed to define "happy" or anything similar. ↩︎

Value LearningAI

Frontpage

24 Ω 12

Mentioned in

48Different perspectives on concept extrapolation

16Value extrapolation, concept extrapolation, model splintering

New Comment

10 comments, sorted by

top scoring

Click to highlight new comments since: Today at 5:55 AM

[-]johnswentworth3yΩ11150

That might work in a tiny world model with only two possible hypotheses. In a high-dimensional world model with exponentially many hypotheses, the weight on happy humans would be exponentially small.

Reply

[-]Quintin Pope3y30

Wouldn't there also be exponentially many variants of the "happy humans" hypothesis? We're really interested in the probability assigned to all hypotheses whose fulfillment leads to human happiness. Once you've trained on happy humans videos, I think there's plausibly enough probability mass assigned to happy humans hypotheses that the AI will actually cause a fair amount of happiness.

Reply

[-]JBlack3y20

There would, so long as the extra dimensions are irrelevant. If there are more relevant dimensions then the total space becomes larger much faster than the happy space. Even having lots of irrelevant dimensions can be risky because it makes the training data sparser in the space being modelled, thus making superexponentially many more alternative hypotheses viable.

Reply

[-]Gordon Seidoh Worley3yΩ690

This doesn't really seem like solving symbol grounding, partially or not, so much as an argument that it's a non-problem for the purposes of value alignment.

Reply

[-]Jon Garcia3y50

This might have a better chance of working if you give the AI strong inductive biases to perceive rewarding stimuli not as intrinsically rewarding in themselves, but rather as evidence of something of value happening that merely generates the stimulus. We want the AI to see smiles as good things, not because upwardly curving mouths are good, but because they are generated by good mental states that we actually value.

For that, it would need a generative model (likelihood function) whose hidden states (hypotheses) are mapped to its value function, rather than mapping the sensory data (evidence) directly. The latter type of feed-forward mapping, which most current deep learning is all about, can only offer cheap heuristics, as far as I can tell.

The AI should also priviledge hypotheses about the true nature of the value function where the mental/physiological states of eudaimonic agents are the key predictors. With your happy-video example, the existence of the video player and video files is sufficient to explain its training data, but that hypothesis (R_2) does not involve other agents. R_1, on the other hand, does assume that the happiness of other agents is what predicts both its input data and its value labels, so it should be given priority. (For this to work, it would also need a model of human emotional expression to compare its sensory data against in the likelihood function.)

Reply