Makes sense that something like this would be possible. Humans can often teach themselves to be better at a skill through practice, even without a teacher or ground truth. Of course, this often runs into mostly useless, self-reinforcing attractors when the learner is unable to determine the true quality levels of their own attempts in the area of study. E.g., physics cranks, paranormal studies, etc.

I actually think it’s a good thing that chain of thought models can improve themselves like this. Chain of thought LLMs seem by far the most alignable of the plausible paths to AGI, since it directly incorporates a prior towards human-like cognition. I’d be a bit more worried about alignment if AGI approaches based on imitating / amplifying humans showed signs of flagging even at the current infra-human capabilities stage.

Further, stable self improvement seems like a particularly sensitive requirement for a scalable alignment solution. It seems much better to have the medium of this self improvement be human language, rather than something like self-modifying meta reinforcement learning from scratch.

Edited to add three more reasons why I think this is a good thing:

1. It gives us an example of self improvement on which we can run experiments. We can look for failure modes in current systems, without having to scale them up to dangerous capabilities.
2. Of all the plausible ways to do self improvement that I can imagine, this one seems the most stable and least likely to have sudden capabilities spikes. The reason is that the basic building block of the model's self improvement is autoregressive pretraining. When the model generates a new dataset and trains on that dataset, the result is to move the model's median generation in the direction of the dataset. We can simply look at the dataset the model generated to get a pretty good idea of the direction in which this particular step of self modification will move the model. Of course, the compounding effects of multiple self improvement rounds are a different matter, but like I mention above, we can run experiments to investigate.
3. It's a good sign that this works. Humans can improve themselves in this manner. If language models had turned out to be incapable of doing so, that would have been an example of non-convergence between LMs and humans. Instead, we see convergence.
   
   Crucially, this is a domain where convergence wasn't necessarily implied by the LM training objective. Language models are trained to imitate human text autoregressively given a single context. It's not a surprise when they generate human-like continuations of their provided contexts. That's what they're trained to do. However, that training objective does not directly imply that the LM should be capable of a human-esque self improvement trajectory that spans multiple prompt-completion pairs. I thus take this result as an update towards there being convergence in the higher order learning dynamics of humans and AIs.

"Anonymous" and 540B parameters, hmm... I'm sure it's not from the company named after an even larger number.

GSM8K = grade school math word problems

DROP = reading comprehension benchmark requiring discrete reasoning over paragraphs

OpenBookQA = question-answering dataset modeled after open book exams for assessing human understanding of a subject. 5,957 multiple-choice elementary-level science questions

ANLI-A3 = adversarial benchmark designed to be challenging to current state-of-the-art models

This paper renames the decades-old method of semi-supervised learning as self-improvement. Semi-supervised learning does enable self-improvement, but not mentioning the hundreds of thousands of papers that previously used similar methods obscures the contribution of this paper. 

Here's a characteristic example of the field, Berthelot et al., 2019 training an image classifier on the sharpened average of its own predictions for multiple data augmentations of an unlabeled input. Another example would be Cho et al., 2019 who train a language model to generate paraphrases of an input sequence by generating many candidate paraphrases and then selecting only those whose confidence passes a specified threshold. As wunan mentioned, AlphaFold is also semi-supervised. Google Scholar lists 174,000 results for semi-supervised learning. 

The authors might protest that this isn't semi-supervised learning because it's training on explanations that are known to generate the correct answer as determined by ground truth labels, rather than training on high confidence answers of unknown ground truth status. That's an important distinction, and I don't know if anybody has used this particular method before. But this work seems much more similar to previous work on semi-supervised learning than they're letting on, and I think it's important to appreciate the context.