1. Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances.
2. However, most existing approaches suffer from several limitations:
   1. They are modality-specific (e.g., working only in text).
   2. They are problem-specific (e.g., verifiable domains like math and coding).
   3. They require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards).
3. In this paper, we ask the question “Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?”
4. Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility (unnormalized probability) between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier.
5. Specifically, we train Energy-Based Transformers (EBTs)—a new class of Energy-Based Models (EBMs)—to assign an energy (unnormalized probability) value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence.
6. This formulation enables System 2 Thinking to emerge from unsupervised learning, making it modality and problem agnostic.
7. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth.
8. During inference, EBTs improve performance with System 2 Thinking (i.e., extra computation) by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using 99% fewer forward passes.
9. Further, we find that System 2 Thinking with EBTs yields larger performance improvements on data that is farther out-of-distribution, and that EBTs achieve better results than existing models on most downstream tasks given the same or worse pretraining performance, enabling EBTs to out-generalize existing paradigms.
10. Consequently, EBTs are a flexible and exciting new paradigm for scaling both the learning and thinking capabilities of models.

FYI some images don't load.

Fantastic writeup.

Did you?

You might want to advertise on reddit or somewhere with more artists. You could ask for three specific arts for three specific posts to make your job easier. (I could do it for you.)

A car can't even take me from the fridge to the couch! Trust me, I tried all of them. You just spend all your time getting in and back out of the car. I don't see much use for cars, unless all you want to is drive down the road.

Any idea why opus 3 is exceptional? Any guess as to what was special about how it was created?

I have found it productive to ask people about what they want to happen, about what kind of future sounds good to them, instead of debating which side effect of the AI pill is the most annoying/deadly. I feel like we are presently more in need of targets than antitargets.

I asked about that here, some interesting answers https://www.lesswrong.com/posts/wxbRqep7SuCFwxqkv/what-exactly-did-that-great-ai-future-involve-again

Almost all the people who get rich and stay rich and are fairly greedy. Or "have a drive to acquire wealth" if you prefer