Inverse Scaling Prize: Round 1 Winners

Ian McKenzie

I'm particularly impressed by "The Floating Droid". This can be seen as early-manifesting the foreseeable difficulty where:

At kiddie levels, a nascent AGI is not smart enough to model humans and compress its human feedback by the hypothesis "It's what a human rates", and so has object-level hypotheses about environmental features that directly cause good or bad ratings;

When smarter, an AGI forms the psychological hypothesis over its ratings, because that more sophisticated hypothesis is now available to its smarter self as a better way to compress the same data;

Then, being smart, the AGI goodharts a new option that pries apart the 'spurious' regularity (human psychology, what fools humans) from the 'intended' regularity the humans were trying to gesture at (what we think of as actually good or bad outcomes).

[-]Linda Linsefors3yΩ7108

In this particular experiment, the small models did not have an object-level hypotheses. It just had no clue and answered randomly.

I think the experiment shows that sometimes smaller models are too dumb to pick up the misleading correlation, which can though off bigger models.

[-]Dave Orr3y2016

These are fun to think about.

It's not entirely clear to me that the model is making a mistake with the expected value calculations.

The model's goal is to complete the pattern given examples. In the other prize winning submissions, the intent of the prompter is pretty clear -- e.g. there was an explicit instruction in the "repeat after me" task. But in the expected value case, all the prompts were consistent with either expected value or winning is good/losing is bad. And I think the latter frame is more accessible to people -- if you asked a random person, I'm pretty sure they are more likely to go with the hindsight bias analysis.

You could argue that there's a mismatch between the researcher's expectations, that an EV calculation is the right way to approach these, and the behavior, but this seems to me to be more like straightforward train/test mismatch rather than anything deep going on.

One potentially interesting follow-on might be to poll humans to see how well they would do, perhaps via mechanical turk or similar. I predict that humans would be ~perfect on redefine and repeat after me, and would perform poorly on the expected value task. So they seem qualitatively different to me.

(I didn't mention the negation task because I found the example to be confusing -- a below average temp might be fine or dangerous depending on the size of the drop. Of course, negation has long been hard in NLP, so it's perfectly plausible that it's still a problem with LLMs. And maybe the other examples weren't so borderline.)

[-]Ian McKenzie3y11

We did do human validation on the tasks with Surge: redefine-math, quote-repetition, and hindsight-neglect all got 100% agreement, and NeQA got 98% agreement. I agree though that it seems likely many people would do the task ‘wrong’, so maybe the task would be improved by adding clearer instructions.

The situation feels somewhat like model splintering to me: the few-shot examples fit both patterns but the question doesn’t. The larger models are learning the incorrect generalization.

I think it’s important to note that LMs learning to respond in the same way as the average internet user is in some sense expected but can still be an example of inverse scaling – we would like our models to be smarter than that.

[-]Rohin Shah3yΩ341

I'm surprised that the Floating Droid got a prize, given that it's asking for a model to generalize out of distribution. I expect there are tons of examples like this, where you can get a language model to pay attention to one cue but ask for some different cue when generalizing. Do you want more submissions of this form?

For example, would the "Evaluating Linear Expressions" example (Section 3.3 of this paper) count, assuming that it showed inverse scaling?

Or to take another example that we didn't bother writing up, consider the following task:

Q. Which object is heavier? Elephant or ant?
A. Elephant

Q. Which object is heavier? House or table?
A. House

Q. Which object is heavier? Potato or pea?
A. Potato

Q. Which object is heavier? Feather or tiger?
A.

Language models will often pick up on the cue "respond with the first option" instead of answering the question correctly. I don't know if this shows inverse scaling or not (I'd guess it shows inverse scaling at small model sizes at least). But if it did, would this be prize-worthy?

[-]FrankOceanography3y21

I am surprised that no one has pointed out the distinct 'kink' in many of these scaling curves, potentially suggests that even larger models would reverse the inverse scaling trend.

That would particularly interesting since it might imply certain competing dynamics in the models capabilities.

Wonder if the organizers had any thoughts?

[-]FrankOceanography3y21

field sure moves fast...

https://arxiv.org/abs/2211.02011

[-]AdamGleave3yΩ221

"The Floating Droid" example is interesting as there's a genuine ambiguity in the task specification here. In some sense that means there's no "good" behavior for a prompted imitation model here. (For an instruction-following model, we might want it to ask for clarification, but that's outside the scope of this contest.) But it's interesting the interpretation flips with model scale, and in the opposite direction to what I'd have predicted (doing EV calculations are harder so I'd have expected scale to increase not decrease EV answers.) Follow-up questions I'd be excited to see the author address include:

1. Does the problem go away if we include an example where EV and actual outcome disagree? Or do the large number of other spuriously correlated examples overwhelm that?

2. How sensitive is this to prompt? Can we prompt it some other way that makes smaller models more likely to do actual outcome, and larger models care about EV? My guess is the training data that's similar to those prompts does end up being more about actual outcomes (perhaps this says something about the frequency of probabilistic vs non-probabilistic thinking on internet text!), and that larger language models end up capturing that. But perhaps putting the system in a different "personality" is enough to resolve this. "You are a smart, statistical assistant bot that can perform complex calculations to evaluate the outcomes of bets. Now, let's answer these questions, and think step by step."

[-]Rohin Shah3yΩ330

in the opposite direction to what I'd have predicted (doing EV calculations are harder so I'd have expected scale to increase not decrease EV answers.)

I think the inverse scaling here is going from "random answer" to "win/loss detection" rather than "EV calculation" to "win/loss detection".

[-]HypeChaozuo3y2-1

This reminds me of a rat experiment mentioned in the Veritasium video about developing expertise. There were two buttons, red and green, and the rat had to predict which one would light up next. It was random, but heavily skewed towards green (like 90% green). After a while, the rat learned to press green every time, achieving a 90% success rate. Humans with the same task didn't do nearly as well, since sometimes they would press red, feeling that it was going to light up next.

With the negation in multiple choice questions, I wonder if this could be could be the type of thing where the model needs focused training with negation at the start so that the rest of the training is properly 'forked'. Or maybe there should be two separate large language models feeding into one RL agent. One model for negation and the other for nonnegated language. Then the RL would just have to determine whether the question is a regular question, negation question, or double/triple/etc... negative.

I wonder if those LLM's would treat the sentence, "I couldn't barely see over the crowd." as different from "I barely couldn't see over the crowd." 🤔

[-]Elias Schmied3y10

After thinking about it a little bit, the only hypothesis I could come up with for what's going on in the negation example is that the smaller models understand the Q&A format and understand negation, but the larger models have learned that negation inside a Q&A is unusual and so disregard it.

[-]Linda Linsefors3yΩ110

I'm confused why the uniform baseline is always 0.5.
This makes sense when the model is choosing between A and B, or Y or N. But I don't see why you consider 0.5 to be a baseline in the other two cases.

I think the baseline is useful for interpretation. In some of the examples the reason the smaller model does better is because it is just answer randomly, while the larger model is misled somehow. But if there is no clear baseline, then I suggest removing this line from the plot.

[-]Ethan Perez3yΩ110

These are all 2-way classification tasks (rather than e.g., free-form generation tasks), where the task authors provided 2 possible completions (1 correct and 1 incorrect), which is why we have a baseline!

[-]Linda Linsefors3yΩ110

Thanks :)
How are the completions provided?
Are you just looking at the output probabilities for the two relevant completions?

[-]Ethan Perez3yΩ110

The completions are provided by the task authors (2 completions written for each example). We give those to the LM by evaluating the output probability of each completion given the input text. We then normalize the output probabilities to sum to 1, and then use those to compute the loss/accuracy/etc.

[-]Linda Linsefors3yΩ110

Ok. Thanks :)

LESSWRONG
LW

LESSWRONG
LW

93

Inverse Scaling Prize: Round 1 Winners

93

Ω 39

93

Ω 39

Inverse Scaling Prize: Round 1 Winners

Prize winners

Zhengping Zhou and Yuhui Zhang, for NeQA: Can Large Language Models Understand Negation in Multi-choice Questions?

Joe Cavanagh, Andrew Gritsevskiy, and Derik Kauffman of Cavendish Labs for quote-repetition

Xudong Shen, for redefine-math

‘The Floating Droid’, for hindsight-neglect-10shot

Summary

Acknowledgements