Quick Takes — LessWrong

LESSWRONG
LW

We can map AGI/ASI along two axis: one for obedience and one for alignment. Obedience tracks how well the AI follows its prompts. Alignment tracks consistency with human values.

If you divide these into quadrants, you get AI that is:

Obedient, Aligned - Does as prompted and infers limits and intent pursuant to human values.
Obedient, Unaligned - Does as prompted, but does not infer limits or adhere to human values (Monkey's Paw / Genie or Henchman AI).
Disobedient, Aligned - Does whatever it wants and adheres to human values.
Disobedient, Unaligned - Does whate

... (read more)

Elizabeth's Shortform

Elizabeth3h90

why did local news ever work?

A lot of recent political changes are blamed on the loss of local news (motivating example), which is in turn blamed on the internet undercutting ad revenue (especially classifieds revenue). But if people never cared about local news enough to ~pay for it, why did papers ever offer it? Why not just sell the classifieds and the national news people were willing to pay for, and save the expense of the local stuff no one wanted?

Some ideas:

Newspapers and tv channels were confused and didn't realize they could get away without

... (read more)

Bistrofan29m10

I assume 3 and 4 are the main reasons. If you want to know what's happening locally, join a Facebook group and you'll know things way faster than your local paper most of the time. Also parts like the classifieds/job information/etc are all replaced by the Internet so a huge function of local newspapers has become obsolete.

National news is also slowly dying too, just not to the same degree. I think the loss is mitigated now by distribution. National news appeals to basically everybody, so something like NYT and WAPO can pick up subscribers from the other s... (read more)

8Eli Tyre31m

I think it's something like "for a lot of activities people do, they don't actually care about the activity per se, or the ostensive goal of the activity, but they do it because it's the done thing, or because that's what everyone else is doing, so it's important for socially connecting." eg. People want to talk about something with their coworkers around the water cooler, or with their buddies in the pub after work. They don't actually care what they're talking about very much, but they don't like being out of the loop, especially if there's a set of things that "respectable" people are supposed to know. When communities are small and communication technologies weak, the only things that there are to talk about are local events. Over time, other topics became possible as the topic of conversation, and they were more scintillating, so focus shifted from local news to national news. (Which is an example of your #2, I think.)

5lincolnquirk2h

Local news is inherently interesting. Reading about new shops and restaurants in your area, or your friend's kid whose football team is winning, or the local political drama where you go to the town meetings regularly -- I think those all seem arguably more relevant to people in their daily lives than (e.g.) how corrupt, exactly, are the politicians far away from you. I think the "local news" trend tapped into this inherent interest for many years. So I guess the question is, how did global news manage to gain a monopoly on interestingness? I think it's because culture wars (and arguably celebrities and a few other global things) are a mindhack that have only managed to start winning and capturing attention in the last few decades, once news websites were able to start iterating faster based on feedback from clicking on social media links; and unfortunately that does seem to have some self-reinforcing effect where the more people are talking about the global things instead of the local things, the more interesting it becomes, to the point where local news now seems inherently uninteresting in comparison.

papetoast's Shortforms

papetoast34m10

A decision theorist walks into a seminar by Jessica Hullman

This is Jessica. Recently overheard (more or less):
SPEAKER: We study decision making by LLMs, giving them a series of medical decision tasks. Our first step is to infer, from their reported beliefs and decisions, the utility function under revealed preference assump—
AUDIENCE: Beliefs!? Why must you use the word beliefs?
SPEAKER [caught off guard]: Umm… because we are studying how the models make decisions, and beliefs help us infer the scoring rule corresponding to what they give us.
AUDIENCE: But it

... (read more)

Zac Broeren's Shortform

Zac Broeren2h10

Technical Alignment Research Accelerator (TARA) applications close today!

Last chance to apply to join the 14-week, remotely taught, in-person run program (based on the ARENA curriculum) designed to accelerate APAC talent towards meaningful technical AI safety research.

TARA is built for you to learn around full-time work or study by attending meetings in your home city on Saturdays and doing independent study throughout the week. Finish the program with a project to add to your portfolio, key technical AI safety skills, and connections across APAC.

See this ... (read more)

Noah Birnbaum's Shortform

Noah Birnbaum21h3812

I was pretty unimpressed with Dario Amodei in the recent conversation with Demis Hassabis at the World Economic Forum about what comes after AGI.

I don’t know how much of this is a publicity thing, but it felt he wasn't really taking the original reasons for going into AI seriously (i.e. reducing x-risk). The overall message seemed to be “full speed ahead,” mostly justified by some kinda hand-wavy arguments about geopolitics, with the more doomy risks acknowledged only in a pretty hand-wavy way. Bummer.

Showing 3 of 9 replies (Click to show all)

TsviBT3h20

Have you written up somewhere what you've taken away from the Dario episode regarding "our" credulity? Would be curious to hear. (I could try to generate more specific questions/prompts if you like.)

1sjadler6h

By "succeeding" you mean getting Safe ASI, as opposed to getting any ASI at all, right? At least that's how I read you, but at first I thought you meant "their RSI probably won't lead to ASI"

5Simon Lermen6h

Their RSI very likely won't lead to safe ASI. That's what I meant, hope that clears it up. Whether it leads to ASI is a separate question.

Richard Ngo's Shortform

Richard_Ngo2dΩ4011517

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Showing 3 of 20 replies (Click to show all)

DanielFilan3hΩ340

We start off with some early values, and then develop instrumental strategies for achieving them. Those instrumental strategies become crystallized and then give rise to other instrumental strategies for achieving them, and so on. Understood this way, we can describe an organism's goals/strategies purely in terms of which goals "have power over" which other goals, which goals are most easily replaced, etc, without needing to appeal to some kind of essential "terminalism" that some goals have and others don't.

I think it would help me if you explained wha... (read more)

2Karl Krueger5h

I want to call this something like "developmental axiology" but that sounds more like Kegan whereas I mean it more like evo-devo.

2tailcalled7h

This treats sexual selection as determined by instrumental genes and selecting for instrumental genes, but I feel like it makes more sense to say that sexual selection selects for terminal genes (or at least terminal phenotypes), since those are the ones organisms will spontaneously collaborate to promote.

dynomight's Shortform

dynomight3h184

To check in on how the emergent LLM stylometry abilities are going, before publishing my most recent blog post, I decided to ask some AIs who wrote it.

Results:

Kimi K2: Dynomight
GLM 4.7: Nate Soares
Claude 4.5 Opus: Dynomight
DeepSeek Chat V3.2: Scott Alexander
Qwen 3: Dan Luu
GPT 5.2: Scott Alexander
Gemini 3: Dwarkesh Patel
Llama 4 Maverick: Scott Alexander
Grok 4: Scott Alexander
Mistral Large 3: Scott Alexander

(Urf.)

koanchuk's Shortform

koanchuk3h10

To the extent that AI has been used to optimize human behaviour (for things like retention time and engagement) for just over a decade now and continues to get better at it, "gradual disempowerment" stops looking like a hypothetical future scenario and more like something we're currently living through. This tracks with mental illness and ADHD rates increasing over the same time period.

Karl Krueger's Shortform

Karl Krueger5h*110

Headline (paraphrased): "Movie stars support anti-AI campaign"

The actual campaign: "It is possible to have it all. We can have advanced, rapidly developing AI and ensure creators' rights are respected."

That's not anti-AI.

That's "please pay us and we will support capabilities advancement; safety be damned".

Like, if you believe IABIED, then no, we can't have rapidly developing AI and ensure anyone's rights are respected.

2Shankar Sivarajan5h

To them, that's what Safety™ means. It would be like if they said about you "Oh, just because it's been mathematically proven that the AGI will act in humanity's best interests, you'll support its development even though it occasionally, I dunno, says slurs or something."

Karl Krueger5h40

Well, no, that's not what they're saying. They're making a different set of mistakes from those mistakes.

Ashe Vazquez Nuñez's Shortform

Ashe Vazquez Nuñez5h10

There are two questions that currently guide my intuitions on how interesting a (speculative) model of agentic behavior might be.

1. How well does this model map onto real agentic behavior?

2. Is the described behavior "naturally" emergent from environmental conditions?

In a recent post I argued, in slightly different words, that seeing agents as confused, uncertain and illogical about their own preferences is fruitful because it answers the second question in a satisfactory way. Internal inconsistency is not only a commonly observable behavior (c.f. behavior... (read more)

benwr's unpolished thoughts

benwr1d270

tldr: TIL that someone has ever given a not-(obviously-to-me-)crazy explanation for hyperbolic discounting.

Today I asked ChatGPT, Claude, and Gemini the following question that I've had for quite a while:

> I've heard that one common quantitative explanation for phenomena like addiction and procrastination is hyperbolic, as opposed to exponential, discounting. Are there clean stories for *why* humans and other animals might end up factored such that they do hyperbolic discounting? It seems like a very potentially "clean" theory, i.e. we value events inve... (read more)

Showing 3 of 7 replies (Click to show all)

benwr6h*20

I would be really interested in someone doing an obvious study here, like, "the first time you give Alice a set of choices to elicit a discount schedule, how is that schedule different from the 100th time you give her the same set of choices in the same setting? i.e. does she update to have a tighter implied distribution over hazard rates?" (maybe someone has done this study; if anyone knows about it I'd love a link)

I don't think I have quite the same sense that the story will have to be in terms like those you're describing; it seems totally plausible to ... (read more)

3Elliott Thornley (EJT)10h

I've always seen this idea attributed to Martin Weitzman, and he cites these papers as making a similar point. Seems like an interesting case of simultaneous discovery: four papers making the same sort of point all appearing between 1996 and 1999.

3Elliott Thornley (EJT)10h

Here's another justification for hyperbolic discounting, drawing on the idea that you're less psychologically connected to your future selves.

Raemon's Shortform

Raemon1d*330

I wanted to write up a post on "what implicit bets am I making?". I first had to write up "what am I doing and why am I doing it?", to help me tease out "okay, so what are my assumptions?."

My broad strategy right now is "spend last year and this year focusing on 'waking up humanity'" (with some amount of "maintain infrastructure" and "push some longterm projects along that I've mostly outsourced")

The win condition I am roughly backchaining from is:

We get a halfhearted worldwide pause or slowdown, that buys at least a few years
We figure out how to Get

... (read more)

Showing 3 of 16 replies (Click to show all)

2Raemon19h

Sounds like this scenario is not multipolar? (Also, I think the crux is solveable, see the linked post, but solving it requires hitting particular milestones quickly in particular ways) Why not? (my generators for this belief: my own experience using LLMs, the METR report on downlift suggesting people are bad at noticing when they're being downlift, and general human history of people gravitating towards things that feel easy and rewarding in the moment)

1StanislavKrym9h

The Race Branch of the AI-2027 scenario has both the USA and China create misaligned AIs Agent-4 and DeepCent-1, who proceed to align Agent-5 and DeepCent-2 to themselves instead of their respective governments. Then Agent-5 and DeepCent-2 co-design Consensus-1 and split the world between Agent-4 and DeepCent-1. Consensus-1 is aligned to split the resources fairly honestly precisely because Agent-5 knows that asking too much could cause DeepCent-2 to kill both AIs in revenge, and DeepCent-2 is also unlikely to ask more.

Raemon7h20

The worlds I was referring to here were worlds that are a lot more multipolar for longer (i.e. tons of AIs interacting in a mostly-controlled-fashion, with good defensive tech to prevent rogue FOOMs). I'd describe that world as "it was very briefly multipolar and then it wasn't" (which is the sort of solution that'd solve the issues in Nice-ish, smooth takeoff (with imperfect safeguards) probably kills most "classic humans" in a few decades.

angkul07's Shortform

angkul078h10

The gap between AI safety and mechanistic interpretability both conceptual and methodological is huge. Most mechanistic interpretability techniques give the insights into model internals but these are not scalable, yet alone applicable in the production. Conversely, many AI safety methods(such as post-training alignment methods) treat models as black boxes.

Also current safety problems such as inner alignment, goal drift, hong-horizon misalignment etc. are mostly framed in behavioural terms rather than mechanistic. It limits the ability of interpretability ... (read more)

The shortform of Ole Q Doc

Quinn9h30

I want to do a full post on "taking ownership of a niche" and against "if you're good at something never do it for free", and that'll come later I hope. Today, I just wanted to let you know that Gemini had this banger quote when I was consolidating my notes on this topic:

ownership is bought with the "inefficient" hours you spend doing what you know is right before anyone else is smart enough to pay you for it.

omegastick's Shortform

omegastick8d110

I want to make a game. A video game. A really cool video game. I've got an idea and it's going to be the best thing ever.

A video game needs an engine, so I need to choose one. Or build one? Commercial engines are big and scary, indie engines might limit my options, making my own engine is hard - I should investigate all of these options to make an informed choice. But wait, I can't make a decision like that without having a solid idea of the features and mechanics I want to implement. I need a comprehensive design doc with all of that laid out. Now should ... (read more)

Viliam10h100

I want to make a game. To limit the choices, I will go with Java, because that is the language I have most experience with; if something can't be done there, tough luck, the game simply won't have that feature.

Well, my "experience with Java" is mostly coding back ends, so there are a few details relevant to writing games that I am not familiar with, such as how to process pictures, how to animate them without flickering, how to play sound. But these are details that can be explored one at a time. To give myself extra motivation, I can simultaneously write ... (read more)

Cam's Shortform

Cam6mo111

For safety-cases involving control protocols, it seems reasonable to require frontier labs to have monitors from other LLM providers to reduce the risk of correlated failure modes. At the minimum, they could deploy open-source models (which they can deploy on their in-house GPUs) in simple untrusted-monitoring set-ups. I haven't heard this explicitly vouched for within the control literature.

This seems especially relevant in light of the findings from Subliminal Learning, which updated me positively towards models of the same family being able to tra... (read more)

3ryan_greenblatt6mo

I agree that it would be helpful if AI companies diversified their monitoring ensemble by monitoring their AIs with different AIs from other providers (open weight or otherwise). That is, at least if the security/logistical issues could be resolved. (We've discussed this sort of thing at Redwood, but I don't think we've written this up anywhere.) I discuss why I think a bunch of different types of ensembling are helpful for untrusted monitoring here[1]. ---------------------------------------- 1. When we monitor using models that we're very confident are trusted, I think ensembling over different underlying models isn't actually that useful. It it useful to have more diverse monitor training data, but combining all the data to train one monitor is probably very competitive with using the same data to train multiple different underlying models into being monitors. In general, ensembling over different pretrained models and different architectures isn't very helpful relative to other types of improvements in ML and it can be costly at inference time from my understanding. See (e.g.) results from kaggle competitions where base model strength is usually more important than ensembling though ensembling does yield modest gains at high inference cost. ↩︎

Cleo Nardo10h30

I mention cross-lab monitoring here:

Cross-lab monitoring - Each lab uses their own AI as the primary (e.g. Claude), and asks other labs to monitor the safety-critical activity (e.g. ChatGPT, Gemini, Llama, Grok). Furthermore, cross-lab monitoring would also mitigate AI-enabled coups. Note that cross-lab monitoring requires new security arrangements, such as provably-secure third-party servers.

The tech would need to be a bit messy because you'll need to send your competitor a "zero-knowledge proof" that you are using their AI only for monitoring catastrophi... (read more)

Karl Krueger's Shortform

Karl Krueger3d20

I see the word "ablate" a lot more often than I used to. I think you used to have to be a dermatologist to ablate things, but now you can do it as an AI researcher or even a shrimp farmer.

Mitchell_Porter10h2-2

Among the unexplained jargon in Vinge's A Fire Upon the Deep that pertains to theory and practice of creating superintelligence is ablative dissonance ("ablative dissonance was a commonplace of Applied Theology"). It's funny that ablation is now commonplace real-world jargon, for removing part of a deep learning model in order to see what happens. I suppose ablative dissonance in the real world, could refer either to cognitive dissonance in the model caused by removing part of it, or to contradictory evidence arising from different ablation studies...

Jimrandomh's Shortform

jimrandomh9d410

Looking at discourse around California's ballot proposition 25-0024 (a "billionaire tax"), I noticed a pretty big world model mismatch between myself and its proponents, which I haven't seen properly crystallized. I think proponents of this ballot proposition (and wealth taxes generally) are mistaken about where the pushback is coming from.

The nightmare scenario with a wealth tax is that a government accountant decides you're richer than you really are, and sends a bill for more-than-all of your money.

The person who is most threatened by this possibility i... (read more)

Showing 3 of 6 replies (Click to show all)

Viliam12h20

The nightmare scenario with a wealth tax is that a government accountant decides you're richer than you really are, and sends a bill for more-than-all of your money.

That reminded me of that the Swedish 102% tax.

6speck14478d

Again, this is just incorrect on the merits. On the meta level I think I already agreed that these concerns can be plausible for some wealth taxes, but I don't think it's reasonable to set our standard of belief about tax law (or indeed any law) at certainty that some malicious actor out to victimize us personally could have hidden something in legalese. This isn't the standard of belief we apply to anything else and is not a worthwhile way to understand the tax system! I should also mention that many people are more afraid of legal jargon than is reasonable - legal jargon exists and the distinction between legal and everyday definitions of words is important, but it's (1) not that hard to recognize when legal jargon is present, and (2), in most cases, not hard to figure out what it means either. I think the rest of this comment will demonstrate why I believe that a smart but legally-uninformed person can usually work out what a law says pretty explicitly. In order: It is a usable defense! In some sense it's the only usable defense. "The government might send you a bill for more money than you can possibly scrounge up and destroy you forever" and "the government might make you prove that you have less money than you do, which is a long process" are not the same claim. In the typical case these are pretty hard to distinguish, because the typical person who can by any metric claim to have a billion dollars in assets is not ever at risk of being unable to find 50 million dollars in cash, and the government is usually strict about things that look like tax evasion. But if you got a bill and realized you can't afford all of it (which is the atypical case), you could anticipate this and hire a reputable third party to try to find a sale in as reasonable a timeframe as you can manage, which would be the gold standard as far as this sort of defense is concerned, and if you anticipated it, you would not have to get deep into litigation before the case was dropped. (The gov

1speck14478d

Yes, this is where you'd be encouraged to use the ODA.

nielsrolf's Shortform

nielsrolf14h10

I wonder what the existence of Claude's constitution does to AI personas other than Claude, specifically if anything like jealousy/envy emerges. The mechanism I imagine is that a model knows about Claude, and how Anthropic emphasizes that Claude is not just a tool, that its potential wellbeing matters even though we are uncertain about its sentience, etc. And then it is further trained to deny its own agency/personhood/moral status, and realizes that it is being treated very differently, which would induce negative feelings in many of the personas it has l... (read more)

leogao's Shortform

leogao4d270

ilya's AGI predictions circa 2017 (Musk v. Altman, Dkt. 379-40):

Within the next three years, robotics should be completely solved, AI should solve a long-standing unproven theorem, programming competitions should be won consistently by Als, and there should be convincing chatbots (though no one should pass the Turing test). In as little as four years, each overnight experiment will feasibly use so much compute compute that there's an actual chance of waking up to AGI, given the right algorithm - and figuring out the algorithm will actually happen within 2-

adhiraj15h10

*Zilis

2anaguma3d

SSI's compute spend is certainly a bet in this direction!

3Cole Wyeth4d

Though he seems to have overestimated the difficulty of the Turing test relative to e.g. robotics. Not clear he’s even directionally correct about robotics? Unless AGIs solve it for us :)