All of axlrosen's Comments + Replies

For the sake of intuition, it's useful to separate the capabilities visibly present in generated sequences from the capabilities of the model itself. Suppose you've got an untuned language model trained on a bunch of human conversations, and you generate a billion rollouts of conversations from scratch (that is, no initial prompting or conditions on the input). This process won't tend to output conversations between humans that have IQs of 400, because the training distribution does not contain those. The average simulated conversation will be, in many ways, close to the average conversation in the training set. But it would be incorrect to say that the language model has an "IQ" of 100 (even assuming the humans in the training distribution averaged 100). The capability elicited from the language model depends on the conditions of its predictions. When prompted to produce a conversation between two mathematicians trying to puzzle something out, the result is going be very different from the random sampling case. You can come up with a decent guess about how smart a character the model plays is, because strong language models tend to be pretty consistent. In contrast, it's very hard to know how smart a language model is, because its externally visible behavior is only ever a lower bound on its capability. The language model is not its characters; it is the thing that can play any of its characters. Next, keep in mind that even simple autoregressive token prediction can be arbitrarily hard. A common example is reversing a hash. Consider prompting a language model with: "0xDF810AF8 is the truncated SHA256 hash of the string" It does not take superhuman intelligence to write that prompt, but if a language model were able to complete that prompt correctly, it would imply really weird things. That's an extreme case, but it's not unique. For a closer example, try an experiment: Try writing a program, at least 25 lines of nontrivial code, starting with a blank file,

The assumption goes that after ingesting human data, it can remix it (like humans do for art, for example) and create its own synthetic data it can then train on. The go-to example is AlphaGo, which after playing a ton of simulated games against itself, became great at Go. I am not qualified enough to give my informed opinion or predictions, but that's what I know.