lemonhope

Wikitag Contributions

Comments

Sorted by

It's a nonprofit, but I would like to complement METR for raising many millions of dollars while sticking to their guns and simultaneously not screwing anybody over. Rare combination of integrity, research competence, and fundraising competence. I expect it to remain a nonprofit forever.

Misc ideas I thought of or heard of:

  • adult language learning service, maybe based on partially-translating books or TV. Eg book starts 10% Russian and gets more Russian as you click the define button less often.
  • foreign dubs for TV (ML-based audio splitting, translation, and audio generation). Sell as debrid service
  • discount claude code competitor which lets you report past sneakiness when you discover it. Sell the reports as training data.

I'm no writer or editor but you could email me. I check my email every few days lemonhope@fastmail.com

Long have I searched for an intuitive name for motte & bailey that I wouldn't have to explain too much in conversation. I might have finally found it. The "I was merely saying fallacy". Verb: merelysay. Noun: merelysayism. Example: "You said you could cure cancer and now you're merelysaying you help the body fight colon cancer only."

Application link says "no access to this page"

"Death with dignity" was clearly intended to trigger the audience to HMCF right? He was doing exactly what you are asking for

  1. Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances.
  2. However, most existing approaches suffer from several limitations:
    1. They are modality-specific (e.g., working only in text).
    2. They are problem-specific (e.g., verifiable domains like math and coding).
    3. They require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards).
  3. In this paper, we ask the question “Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?”
  4. Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility (unnormalized probability) between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier.
  5. Specifically, we train Energy-Based Transformers (EBTs)—a new class of Energy-Based Models (EBMs)—to assign an energy (unnormalized probability) value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence.
  6. This formulation enables System 2 Thinking to emerge from unsupervised learning, making it modality and problem agnostic.
  7. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth.
  8. During inference, EBTs improve performance with System 2 Thinking (i.e., extra computation) by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using 99% fewer forward passes.
  9. Further, we find that System 2 Thinking with EBTs yields larger performance improvements on data that is farther out-of-distribution, and that EBTs achieve better results than existing models on most downstream tasks given the same or worse pretraining performance, enabling EBTs to out-generalize existing paradigms.
  10. Consequently, EBTs are a flexible and exciting new paradigm for scaling both the learning and thinking capabilities of models.
Load More