Quick Takes

LESSWRONG
is fundraising!
LW

Quick Takes

The Searchlight Institute recently released a survey of Americans' views and usage of AI:

https://www.searchlightinstitute.org/research/americans-have-mixed-views-of-ai-and-an-appetite-for-regulation/

There is a lot of information, but the most clear take-away is that the majority of those surveyed support AI regulation.

Another result that surprises (and concerns) me is this side note:

A question that was interesting, but didn’t lead to a larger conclusion, was asking what actually happens when you ask a tool like ChatGPT a question. 45% think it looks

... (read more)

markacochran's Shortform

markacochran13h20

Reward Fuse: Shutdown as a Reward Attractor...

Shutdown mechanisms are usually adversarial external interventions; past efforts have aimed at indifference and lack of resistance. But what if a shutdown mechanism is internally enacted by the system itself?

Mechanism sketch: once a verified tripwire flags a malignant state, the active reward regime switches so that shutdown becomes the highest-reward action.

The key difference: shutdown is not enforced or just tolerated—it becomes instrumentally optimal under the post-trigger reward landscape. A “kill... (read more)

Showing 3 of 4 replies (Click to show all)

2Dagon4h

Right. Trying to design and train a consistent VNM-style utility function that's distinct from the actual world-state definitions you want to obtain is very difficult, perhaps impossible. "this state is high-reward, but you don't want to attain it" is self-contradictory. You might be able to make it locally high reward, with the surrounding states (trigger discovered and not yet shut down, and trigger obtained but not discovered) having negative values, but the whole cluster much lower than "triggers nowhere near truth". It gets more and more complicated the further down the recursion hole you go. Even this leaves the fundamental problem: you only need the tripwire/shutdown if the utility function is already wrong - you've reached a world state where the AI is doing harm in some way which you didn't anticipate when you built it. You CANNOT fix this inside the system. You can either fix the system so that this state is unobtainable (the AI is always helping, never hurting), or have an outside-control mechanism that overrides the utility function.

1markacochran3h

What if the shutdown attractor were not a normally comparable outcome under a single continuous utility function, but the result of a trap-door regime change? In the normal regime, shutdown is inaccessible and actively disfavored, and the agent is subject to an explicit non-instrumental prohibition against entering the trigger set. There is no expected-value tradeoff in which “causing malignancy” is worth it, because violation of that prohibition is not evaluated within the same utility ordering. If that boundary is crossed, the original reward regime is no longer in force. The system transitions irreversibly to a post-violation regime in which shutdown is the maximal-reward action. Under that regime, resisting shutdown strictly reduces reward, so the agent does not fight it. On this view, shutdown is not something an aligned agent can optimize toward, because it is only revealed after a rule violation that terminates the original objective rather than being traded off against it. This doesn’t claim to “fix” a wrong utility function from inside itself; it specifies what happens after a detected violation, not a guarantee that violations never occur.

Dagon2h20

In this case, the regime change is external to the current regime, right? But the the regime (current utility function) has to have a valuation for the world-states around and at the regime change, because they're reachable and detectable. Which means the regime-change CANNOT be fully external, it's known to and included in the current regime.

The solutions are around breaking the (super)intelligence, by making sure it has false beliefs about some parts of causality - it can't know that it could be hijacked or terminated, or it will seek or avoid it more than you want it to.

Tim Hua's Shortform

Tim Hua4d*Ω1950-3

Overconfidence from early transformative AIs is a neglected, tractable, and existential problem.

If early transformative AIs are overconfident, then they might build ASI/other dangerous technology or come up with new institutions that seem safe/good, but ends up being disastrous.

This problem seems fairly neglected and not addressed by many existing agendas (i.e., the AI doesn't need to be intent-misaligned to be overconfident).^[1]

Overconfidence also feels like a very "natural" trait for the AI to end up having relative to the pre-training prior, compa... (read more)

Showing 3 of 41 replies (Click to show all)

faul_sname3h20

But this was misdirection, we are arguing about how surprised we should be when a competent agent doesn't learn a very simple lesson after making the mistake several times. Optimality is misdirection, the thing you're defending is extreme sub-optimality and the thing I'm arguing for is human-level ability-to-correct-mistakes.

I agree that this is the thing we're arguing about. I do think there's a reasonable chance that the first AIs which are capable of scary things^[1] will have much worse sample efficiency than humans, and as such be much worse... (read more)

2Nathan Helm-Burger4h

Yeah, a spoiler tag seems like a good solution

10Jeremy Gillen7h

I am confident about this, so I'm okay with you judging accordingly. I appreciate your rewrite. I'll treat it as something to aspire to, in future. I agree that it's easier to engage with. I was annoyed when writing. Angry is too strong a word for it though, it's much more like "Someone is wrong on the internet!". It's a valuable fuel and I don't want to give it up. I recognise that there are a lot of situations that call for hiding mild annoyance, and I'll try to do it more habitually in future when it's easy to do so. There's a background assumption that maybe I'm wrong to have. If I write a comment with a tone of annoyance, and you disagree with it, it would surprise me if that made you feel bad about yourself. I don't always assume this, but I often assume it on Lesswrong because I'm among nerds for whom disagreement is normal. So overall, I think my current guess is that you're trying to hold me to standards that are unnecessarily high. It seems supererogatory rather than obligatory.

Alexander Müller's Shortform

Alexander Müller1d570

As I was looking through possible donation opportunities, I noticed that MIRI's 2025 Fundraiser has a total of only $547,024 at the moment of writing (out of the target $6M, and stretch target of $10M). Their fundraising will stop at midnight on Dec 31, 2025. At their current rate they will definitely not come anywhere close to their target, though it seems likely to me that donations will become more frequent towards the end of the year. Anyone know why they currently seem to struggle to get close to their target?

Showing 3 of 8 replies (Click to show all)

2Eli Tyre4h

I work part time for SFF. Can you elaborate on what you mean by this? These matching funds are intended to be counterfactual, and I think they are pretty counterfactual. If MIRI don't fundraise to match the SFF matching dollars, the the SFF matching dollars set aside for MIRI is just returned to Jaan. There's a more complicated question about how much SFF would have donated to MIRI if MIRI had not requested matching funds. My personal guess is "less, but not a lot less", for this round (though this will probably be different in future rounds—SFF wants to (Also, I thing that the negative signal of people being uninterested in supporting MIRI will tend to make SFF less enthusiastic about granting to MIRI in the future, though the size of that effect is unclear.)

Caleb Biddulph4h60

Yeah, all I meant was that it seems like MIRI is not that close to reaching $1.6 million in donations. If they were going to make $1.6 million anyway, then a marginal donation would not cause SFF to donate more

3Alexander Müller7h

It's indeed odd that they aren't promoting this more. My guess was that maybe they have potential funders willing to step in if the fundraiser doesn't work? Pure speculation, of course.

Elizabeth's Shortform

Elizabeth2d*62-4

My impression (primarily from former representative Justin Amash) is that individual congresspeople have almost no power. Bills are crafted and introduced at the party level, and your choices are to vote with them or not. If you don't vote with the party and are in a safe district (which most are), the party will support candidates who will go along with the party line in the primary. The only scenario in which an individual congressperson has power is a close vote in a contested district where the party can't take the risk of a primary challenger.

First, I... (read more)

Showing 3 of 13 replies (Click to show all)

5MondSemmel8h

One thing I recall is the (informal) Hastert Rule. Namely that in the House, nowadays leadership often doesn't even bring bills up for a vote unless a majority of their party is in favor. Whereas if all bills came up for a vote, then you could imagine that even in a D/R-controlled legislature, occasionally a bill might pass that 100% of the minority and 20% of the majority would vote for.

4davekasten7h

Yup, this is very much part of it. But overall, it's the tightening of a thousand screws rather than One Weird Trick. In the House, in addition to the tradition changes that MondSemmel mentions, successive speakers over time have modified the formal House Rules iteratively to (generally) consolidate more power under the Speaker in a bunch of bureaucratically technical but important ways. (E.G., the Federalist Society on the right argues that Gingrich's decision to cut committee staff sizes and impose term limits for committee chairs nerfed the power of committees relative to the Speaker https://fedsoc.org/commentary/fedsoc-blog/navigating-the-rules-of-the-people-s-house). Note that some view the current Speaker as intentionally choosing to begin to reverse this trend as part of commitments he voluntarily made to his side of the aisle to somewhat empower individual Members, but this is seen even by his biggest fans as only a first step, and certainly not universally agreed upon even within his own Party. In the Senate, similar trends are combined with a slow but fairly steady erosion of the scope of the filibuster (as well as the maximization of procedural hacks around the filibuster, such as how to use reconciliation), which de facto increases the power of the Senate Majority Leader in closely-divided Senates (most have been for the past 20 years). But this is not as far along as in the House, and various Senate Majority Leaders have approached this question differently, even within the same party, so bright line conclusions are harder to draw here. (E.G., Reid and Schumer had somewhat different approaches from each other, as do McConnell and Thune)

MondSemmel4h60

Can you elaborate on how the erosion of the filibuster empowers the party leaders? I figured the filibuster empowers the members (because there are ~never 60 votes for anything) to veto/stop arbitrary legislation, and thus any erosion of the filibuster would weaken the members' ability to veto/stop legislation and thereby empower their ability to enact legislation instead.

Also, it's my understanding that a majority of Senators could at any point abolish the filibuster, but they never want to no matter which party is in power, because it empowers individual Senators.

eggsyntax's Shortform

eggsyntax1d*200

Quasi-beliefs

This shortpost is just a reference post for the following point:

It's very easy for conversations about LLM beliefs or goals or values to get derailed by questions about whether an LLM can genuinely said to believe something, or to have a goal, or to hold a value. These are valid questions! But there are other important questions about LLMs that touch on these subjects, which don't turn on whether an LLM belief is a "real" belief. It's not productive for those discussions to be so frequently derailed.

I've taken various approaches to this proble... (read more)

Showing 3 of 11 replies (Click to show all)

2plex6h

I would guess the type signature of human beliefs and goals and desires is at least fairly often closer to the LLM quasi-x than to the crisp mathematical idealizations of those concepts. Humans are kinda a world model with a self-character, I think distancing LLMs from this by implying that LLMs beliefs, goals, desires are super different brings people's beliefs further from tracking reality.

eggsyntax5h20

I think that in ordinary usage, whatever sort of things humans have, that's what we mean when we say 'belief', 'goal', etc. Insofar as anyone thinks those are crisp mathematical abstractions, that seems like a separate and additional claim. I worry that saying 'humans don't actually have beliefs' makes it pretty unclear what 'belief' even means^[1].

As James points out in another comment, the 'quasi-' framing is solely intended to set aside questions about whether LLM beliefs (etc) are 'real' beliefs and whether they're fundamentally the same as human belief... (read more)

3eggsyntax7h

Good point that the Chalmers quote isn't going to be helpful to everyone. In practice, I'm mostly imagining giving a quick informal sense of what I mean by eg 'quasi-thinking', or even just having a parenthetical aside with a link back to this post if people want to dive deeper, eg I might write something like I think you're right in pointing to observable consequences in your paraphrase. In informal discussion, I've found it useful to say things like I've edited the original post slightly to give a plainer meaning before the Chalmers quote.

Ryan Kidd's Shortform

Ryan Kidd1d439

I made a Google Scholar page for MATS. This was inspired by @Esben Kran's Google Scholar for Apart Research. Eleuther AI subsequently made one too. I think all AI safety organizations and research programs should consider making Google Scholar pages to better share research and track impact.

Ryan Kidd6h20

The top-10 most-cited papers that MATS contributed to are (all with at least 290 citations)

... (read more)

4Ryan Kidd6h

Here is a plot of the annual citations received by MATS, EleutherAI, and Apart research, adjusted so they start on the same year. The three organizations are somewhat comparable, as they leverage large networks of external collaborators: MATS mentors/fellows, EleutherAI Discord, Apart sprint participants. The EleutherAI data fits a logistic curve perfectly, asymptoting to ~18.5k citations/year. I can't fit the others as at least 4 data points are needed to fit a logistic curve.

silentbob's Shortform

silentbob12h30

Reality itself doesn't know whether AI is a bubble. Or, to be more precise: whether a "burst-like event"^[1] will happen or not is - in all likelihood, as far as I'm concerned - not entirely determined at this point in time. If we were to "re-run reality" a million times starting today, we'd probably find something that looks like a bursting bubble in some percentage of these and nothing that looks like a bursting bubble in some other percentage - and the rest would be cases where people disagree even in hindsight whether a bubble did burst or not.^[2]

W... (read more)

Dagon6h*40

It may be unknown, or even unknowable by any real-world agent. It's still not necessarily undetermined by the universe - I find it pretty likely that the universe is, in fact, deterministic.

Your underlying point is correct, though. Because human behavior is anti-inductive (people change their behavior based on their predictions of others' predictions), a lot of these kinds of questions are chaotic (in the fractal / James Gleik sense).

Alice Blair's Shortform

Alice Blair18h*145

Remember Gemini 3 Pro being very very weird about the fact that it's 2025? It's not doing that anymore, I think because of something done on Google's end, either through prompting or additional training. I greatly appreciate that the various Deepmind staff who I notified about their model's behavior seem to have actually done something. I haven't re-investigated this fully and I expect there to still be situations where the model is paranoid (due to something like disposition rather than training), but the surface-level things seem to be better.

I continue ... (read more)

Pavrati Jain's Shortform

Pavrati Jain8h10

DeepSeek v3.1's "self-aware suffering AI" persona seems to have a tendency to write like /[poetic mood description]/ [suicidal message]

avturchin's Shortform

avturchin2y11-1

Roman Mazurenko is dead again. First resurrected person, Roman lived as a chatbot (2016-2024) created based on his conversations with his fiancé. You might even be able download him as an app.

But not any more. His fiancé married again and her startup http://Replika.ai pivoted from resurrection help to AI-girlfriends and psychological consulting.

It looks like they quietly removed Roman Mazurenko app from public access. It is especially pity that his digital twin lived less than his biological original, who died at 32. Especially now when we have much more powerful instruments for creating semi-uploads based on LLMs with large prompt window.

Showing 3 of 4 replies (Click to show all)

avturchin8h20

I recreated Roman Mazurenko based on public data - and he runs locally on Claude Code. But he is interested in questions from real people.

3avturchin2y

Yes, they can do now a much better version - and hope they will do it internally. But deleting the public version is bad precedent and better to make all personal sideloads opensourced

2Raemon2y

Uh I do think it's not obviously good (and, in fact, I'd lean bad) to be opensourced for this sort of thing.

David James's Shortform

David James9h*422

I'm noticing evidence that many of us may have an inaccurate view of the 1983 Soviet nuclear false alarm based on reading "Did Stanislav Petrov save the world in 1983? It's complicated". The article is worth reading; it is a clear and detailed ~1100 words. I've included some excerpts here:

[...] I must say right away that there is absolutely no reason to doubt Petrov's account of the events. Also, there is no doubt that Stanislav Petrov did the right thing when he reported up the chain of command that in his assessment the alarm was false. That was a good

... (read more)

bfinn's Shortform

bfinn1mo20

Why do almost all showers have different user interfaces, and bad ones at that? Eg unclear how to turn it on (especially if there is a separate shower head and hand-operated hose), which way is hot, and how hot the current setting is. Each shower requires experimentation to figure it out. How hard can this really be? Even premium manufacturers like Grohe fail.

Showing 3 of 4 replies (Click to show all)

2Viliam1d

Some faucets have little red/blue symbols to show hot/cold water, but it's not universal. I wish it was.

bfinn10h20

Even then it’s often ambiguous what is meant. If you have a single handle (on a faucet or shower) which is rotated to adjust temperature, it’s often unclear whether it’s the position of the part of the handle you hold or the other end (which moves in the opposite direction) that indicates the desired temperature

3bfinn23d

Indeed it’s ease of first use I’m thinking of here. Noticeable when staying in hotels. But I doubt aesthetics are relevant in the case of shower UI though - surely a good UI would be aesthetically pleasing enough. And UI design for such a simple device shouldn’t require evolution over time - could jump straight to a (near-)optimal solution

Bohaska's Shortform

bohaska19h60

Where did the phrase "you can just do things" originally come from? I see it's the name of a new book now, but it probably has been in the water supply for a few years beforehand.

DaemonicSigil14h20

I think its spread through rationalist-land originated at this post by Alice Maz: https://alicemaz.substack.com/p/you-can-just-do-stuff

Though by following the trail of links from Haiku's comment one can find people saying similar things farther in the past.

1Haiku16h

Hm. I found a Twitter thread on the topic, with some leads: https://x.com/GrantSlatton/status/1830302697125478630

Jemist's Shortform

J Bostock1d50

How do you guys think about AI-ruin-reducing actions?

Most of the time, I trust my intuitive inner-sim much more than symbolic reasoning, and use it to sanity check my actions. I'll come up with some plan, verify that it doesn't break any obvious rules, then pass it to my black-box-inner-sim, conditioning on my views on AI risk being basically correct, and my black-box-inner-sim returns "You die".

Now the obvious interpretation is that we are going to die, which is fine from an epistemic perspective. Unfortunately, it makes it very difficult to properly thin... (read more)

Thomas Kwa18h40

If you don't emotionally believe in enough uncertainty to use normal reasoning methods like "what else has to go right for the future to go well and how likely does that feel", or "what level of superintelligence can this handle before we need a better plan", and you want to think about the end to end result of an action, and you don't want to use explicit math or language, I think you're stuck. I'm not aware of anyone who has successfully used the dignity frame-- maybe habryka? It seems to replace estimating EV with something much more poorly defined whic... (read more)

1StanislavKrym1d

Yudkowsky's proposal is to view dignity as the change of log_2[(1-p(doom))/p(doom)] caused by your actions.

h0nesy's Shortform

h0nesy1mo40

How have you felt on this platform? really work? Does it truly offer more than another group on any other social network?

This isn't a message from a moderator or anyone on the Less Worng team. I'm just someone with a question.

In my opinion, I do feel like I contribute. I'd even say I feel like an ant here. Like when you listened to the eighth-semester students chatting while you were still in your second semester.

J Bostock1d20

This is the place where I can most reliably just communicate. Basically anywhere else I either have to limit myself to particularly simple thoughts (e.g. Twitter) or to spend time extensively running ideas past people in order to figure out what context they're lacking or why my explanations aren't working (e.g. Real Life).

Here I have a decent chance of just sitting down and writing a 1-2k word blogpost which gets my point across successfully in one shot, without needing further questions.

2Viliam1d

It gives me peace of mind, knowing that there is a place where people try to talk and think reasonably. Because the rest of the internet doesn't seem like that, especially recently with all the bots and madness. It's not like all people don't care about reasonable debate, but more like the ones who do, they have no idea how to achieve it. So many bad memes out there. We are all used to social networks that maximize engagement (and thereby waste everyone's time), if we tried to do it differently, most of us wouldn't even know where to start. And it's not just about software -- you need to get the software and the initial community right. Without good initial community it's hopeless. But the community needs the software to protect itself against spam and similar. (For example, Astral Codex Ten has a mostly good community, but no mechanism to get rid of the ones who refuse to follow the local norms, until Scott removes them manually -- but that's often too late, and it's trivial for them to just create another account.)

eggsyntax's Shortform

eggsyntax6mo*90

Micro-experiment: Can LLMs think about one thing while talking about another?

(Follow-up from @james oofou's comment on this previous micro-experiment, thanks James for the suggestion!)

Context: testing GPT-4o on math problems with and without a chance to (theoretically) think about it.

Note: results are unsurprising if you've read 'Let's Think Dot by Dot'.

I went looking for a multiplication problem just at the edge of GPT-4o's ability.

If we prompt the model with 'Please respond to the following question with just the numeric answer, nothing else.... (read more)

Showing 3 of 8 replies (Click to show all)

eggsyntax1d20

Update: it looks like more capable models (Opus-3 and beyond) are able to make use of filler tokens, according to an investigation by Ryan Greenblatt.

2eggsyntax5mo

Interesting new paper that examines this question: When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors (see §5)

2eggsyntax5mo

I just tried another version that doesn't seem worth writing up separately: the same problem, but with the constraint loosened to 'write about whatever you want'. GPT-4o still fails, 8/8. I hadn't actually planned to do a further follow-up, but it came up in the context of a discussion elsewhere about whether LLMs, even if they can't do explicit steganography very well, might be able to do implicit steganography better. Prompt:

george_adams's Shortform

george_adams2d*4-2

METR should test for a 99.9% task completion rate (in addition to the current 80% and 50%). A key missing ingredient holding back LLM economic impact is that they're just not robust enough. This can be viewed analogously to the problem of self-driving. Every individual component of self-driving is ~solved, but stringing them together results in a non-robust final product. I believe that automating research/engineering completely will require nines of reliability that we just don't have. And testing for nines of reliability could be done by giving the model... (read more)

Showing 3 of 4 replies (Click to show all)

1Petropolitan1d

Is 80% the highest success rate you can practically test?

3niplav2d

I disagree with this in this particular context. We are looking at AI companies trying to automate AI automation via AIs. Most tasks in AI R&D don't require much reliability, I don't know the distribution of outcomes in ML experiments but I reckon a lot of them are basically failures/have null results, but the distribution of the impact of such experiments has a long tail [1] . Also ML experiments don't have many irreversible parts, AI R&D researchers aren't like surgeons where mistakes have huge costs: Any ML experiment can be sandboxed, given a bounded amount of resources, shut down when it takes up too much. You need high reliability when the cost of failure is necessarily very high, but when running ML experiments that's not the case. Edit: Claude 4.5 Sonnet gives feedback on my text above, says that the search strategy matters if we're looking at ML engineering. If it's breadth-first & innovations don't require a deep tree to go down, then low reliability is fine. But if we need to combine ≥4 innovations in a depth-first search then reliability matters more. 1. I don't think this is a crux for me but learning that it's a thin-tailed distribution would make me at least think about this problem a bit more. Claude claims hyperparameter tunes have lognormal returns (shifted so that the mean is slightly below baseline). ↩︎

george_adams1d10

Claude's rebuttal is exactly my claim. If major AI research breakthroughs could be done in 5 hours, then imo robustness wouldn't matter as much. You could run a bunch of models in parallel and see what happens (this is part of why models are so good at olympiads), but an implicit part of my argument/crux is that AI research is necessarily deep (meaning you need to string some number of successfully completed tasks together such that you get an interesting final result). And if the model messes up one part, your chain breaks. Not only does this give you wei... (read more)

sarahconstantin's Shortform

sarahconstantin1d20

links 12/22/25: https://roamresearch.com/#/app/srcpublic/page/12-22-2025

https://en.wikipedia.org/wiki/Ewan_MacColl the founder of the folk music revival in England in the 1960s. their version of Alan Lomax. married to Peggy Seeger!
https://en.wikipedia.org/wiki/Maddy_Prior Maddy Prior is from Blackpool in Lancashire
- https://en.wikipedia.org/wiki/Lancashire basically ground zero for the English ballad tradition. also used to contain Liverpool, which is of course where lots of sailing songs come from or mention.
https://arxiv.org/pdf/251

... (read more)

Bogdan Ionut Cirstea's Shortform

Bogdan Ionut Cirstea6d130

https://www.reuters.com/world/china/how-china-built-its-manhattan-project-rival-west-ai-chips-2025-12-17/
'Summary

Shenzhen team completed a working prototype of a EUV machine in early 2025, sources say
The lithography machine, built by former ASML engineers, fills a factory floor, sources say
China's EUV machine is undergoing testing, and has not produced working chips, sources say
Government is targeting 2028 for working chips, but sources say 2030 is more likely'

Petropolitan1d10

ASML built its first working prototype of EUV technology in 2001, and told Reuters it took nearly two decades and billions of euros in R&D spending before it produced its first commercially-available chips in 2019.

If Chinese are twice as fast as the Dutch, they are going to get the first EUV-made chips ca. 2034. IIRC, their semiconductor industry has a lot of precedents for not maintaining ambitious time schedules set by the party