orthonormal

The Loudest Alarm Is Probably False

Epistemic Status: Simple point, supported by anecdotes and a straightforward model, not yet validated in any rigorous sense I know of, but IMO worth a quick reflection to see if it might be helpful to you. A curious thing I've noticed: among the friends whose inner monologues I get to hear, the most self-sacrificing ones are frequently worried they are being too selfish, the loudest ones are constantly afraid they are not being heard, the most introverted ones are regularly terrified that they're claiming more than their share of the conversation, the most assertive ones are always suspicious they are being taken advantage of, and so on. It's not just that people are sometimes miscalibrated about themselves- it's as if the loudest alarm in their heads, the one which is apt to go off at any time, is pushing them in the exactly wrong direction from where they would flourish. Why should this be? (I mean, presuming that this pattern is more than just noise and availability heuristic, which it could be, but let's follow it for a moment.) It's exactly what we should expect to happen if (1) the human psyche has different "alarms" for different social fears, (2) these alarms are supposed to calibrate themselves to actual social observations but occasionally don't do so correctly, and (3) it's much easier to change one's habits than to change an alarm. In this model, while growing up one's inner life has a lot of alarms going off at various intensities, and one scrambles to find actions that will calm the loudest ones. For many alarms, one learns habits that basically work, and it's only in exceptional situations that they will go off loudly in adulthood. But if any of these alarms don't calibrate itself correctly to the signal, then they eventually become by far the loudest remaining ones, going off all the time, and one adjusts one's behavior as far as possible in the other direction in order to get some respite. And so we get the paradox, of people who seem to be in

207Jan 2, 2018

orthonormal

Message

18127

290

2844

17y

How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage

This is a story about a flawed Manifold market, about how easy it is to buy significant objective-sounding publicity for your preferred politics, and about why I've downgraded my respect for all but the largest prediction markets. I've had a Manifold account for a while, but I didn't use it...

Aug 6, 2024198

Run evals on base models too!

(Creating more visibility for a comment thread with Rohin Shah.) Currently, DeepMind's capabilities evals are run on the post-RL*F (RLHF/RLAIF) models and not on the base models. This worries me because RL*F will train a base model to stop displaying capabilities, but this isn't a guarantee that it trains the...

Apr 4, 202449

Mesa-Optimizers via Grokking

Summary: Recent interpretability work on "grokking" suggests a mechanism for a powerful mesa-optimizer to emerge suddenly from a ML model. Inspired By: A Mechanistic Interpretability Analysis of Grokking Overview of Grokking In January 2022, a team from OpenAI posted an article about a phenomenon they dubbed "grokking", where they trained...

Dec 6, 202236

Transitive Tolerance Means Intolerance

Our society is pretty messed up around arguments of whose ideas we should and shouldn't tolerate. Some of this is inevitable: even without censorship, there are cases where group X can choose to actively show respect to person Y, and members of X will argue about that, and people with...

Aug 14, 202139

Improvement for pundit prediction comparisons

[EDIT: SimonM pointed out a possibly-fatal flaw with this plan: it would probably discourage more pundits from joining the prediction-making club at all, and adding to that club is a higher priority than comparing the members more accurately.] Stop me if you've heard this one. (Seriously, I may not be...

Mar 28, 202116

Developmental Stages of GPTs

Epistemic Status: I only know as much as anyone else in my reference class (I build ML models, I can grok the GPT papers, and I don't work for OpenAI or a similar lab). But I think my thesis is original. Related: Gwern on GPT-3 For the last several years,...

Jul 26, 2020140

Don't Make Your Problems Hide

I've seen a worrying trend in people who've learned introspection and self-improvement methods from CFAR, or analogous ones from CBT. They make better life decisions, they calm their emotions in the moment. But they still look just as stressed as ever. They stamp out every internal conflict they can see,...

Jun 27, 202062

Load More (7/74)

LESSWRONG
LW

LESSWRONG
LW

orthonormal

orthonormal

orthonormal

The Loudest Alarm Is Probably False

How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage

Choosing the Zero Point

Developmental Stages of GPTs

orthonormal

How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage

Run evals on base models too!

Mesa-Optimizers via Grokking

Transitive Tolerance Means Intolerance

Improvement for pundit prediction comparisons

Developmental Stages of GPTs

Don't Make Your Problems Hide

How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage

Run evals on base models too!

Mesa-Optimizers via Grokking

Transitive Tolerance Means Intolerance

Improvement for pundit prediction comparisons

Developmental Stages of GPTs

Don't Make Your Problems Hide

The Loudest Alarm Is Probably False

How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage

Choosing the Zero Point

Developmental Stages of GPTs