LESSWRONG
LW

Quick Takes

Is there anyone exploring how AI might be used to increase integrity and build trustworthiness.

For example it could scan the behaviour of people, businesses or AI and see whether it is consistent to stated promises, flagging things that are not.

It might be used to train LLMs to be consistent if they are too be used as agents

Charlie Steiner's Shortform

Charlie Steiner1d425

From my reader's perspective, Inkhaven was probably bad. No shade to the authors, this level of output is a lot of work and there was plenty I enjoyed. But it shouldn't be a surprise that causing people to write a lot more posts even when they're not inspired leads to a lot more uninspired posts.

A lot of the uninspired posts were still upvoted on LW. I even did some of that upvoting myself, just automatically clicking upvote as I start reading a post with an interesting first paragraph by someone whose name I recognize. Mostly this is fine, but it dilutes ... (read more)

Showing 3 of 16 replies (Click to show all)

2habryka1h

This is just false. Of the above list, the people who were doing Inkhaven in some form or another: * Me (2nd post on that list) * aggliu * RobertM * Mikhail * Mingyuan * Ben Goldhaber * Ben Pace * John Wentworth Almost 50% of the whole list! Me, Ben and Robert were just participating in Inkhaven (Ben for the whole month, me and Robert for a week each) and Wentworth was doing his daily posting because of Inkhaven. It's obvious there is a huge effect here (you can of course dispute that the posts are good, but trying to somehow slice it up as Inkhaven not having a huge effect seems very unlikely to have any good case for it).

Erich_Grunewald1h20

It wasn't clear to me from the Inkhaven website that you, Ben Pace, and John Wentworth were participating to that degree (though I did mention you three), and I missed aggliu and RobertM. So fair enough, I'll retract my comment. (ETA: I missed aggliu since I didn't know their name and they had only that one LW post in November, and I thought RobertM might be Rob Miles, but none of RobertM's November LW posts seem to be listed among Rob Miles's posts on the Inkhaven website. But obviously you were there and I was not so I defer to you.)

2Garrett Baker2h

I noticed that many posters not doing Inkhaven were more vibrant, which I credit to the Inkhaven spirit, which also spawned Halfhaven.

Jan Betley's Shortform

Jan Betley2mo400

Anhedonia as a side-effect of semaglutide.

Anecdotal evidence only. I hope this might be useful for someone, especially that semaglutide is often considered a sort of miracle drug (and for good reasons). TL;DR:

I had pretty severe anhedonia for the last couple months
It started when I started taking semaglutide. I've never had anything like that before and I have no idea for possible other causes.
It mostly went away now that I decreased the dose.
There are other people on the internet claiming this is totally a thing

My experience with semaglutide

I've been taki... (read more)

Showing 3 of 7 replies (Click to show all)

3jamii2mo

A reasonable estimate for a sedentary person is https://www.omnicalculator.com/health/bmr-harris-benedict-equation multiplied by 1.2. Plus any activity on top. Worth calculating - you might be at a much bigger calorie deficit than you think. When I'm at a 200-400 calorie deficit for a while I can still be very active, but my mood feels flat and I have a hard time getting excited about anything.

2Jan Betley2mo

Thx. I was thinking: * 1kg is roughly 7700 calories * I'm losing a bit more than 1kg per month * Deficit of 9k calories per month is 300 kcal daily Please let me know if that doesn't make sense : )

jamii3h10

My wife coaches teen athletes and the signs she's taught to look out for are not weight loss but withdrawal, depression, poor digestion, feeling cold etc. Not to say that you're doing something unhealthy, just that low mood is a known effect of being in an extended calorie deficit and not necessarily an effect of the semaglutide.

As you go further into deficit you tend to downregulate other stuff (eg bone and tissue maintenance) before losing weight faster, so the amount of weight loss only gives you an lower bound on your deficit. It might be worth tr... (read more)

the gears to ascenscion's Shortform

the gears to ascension4h10

8e6e00c9233c12befbce83af303850ae4a17aca8ff17b0f16666f07b1efea970 e0affc162f60b0fa86fd9edea0f42655ab35b5a4803ccdf7dbd6831af4cab13e

Richard Ngo's Shortform

Richard_Ngo13dΩ16320

Error-correcting codes work by running some algorithm to decode potentially-corrupted data. But what if the algorithm might also have been corrupted? One approach to dealing with this is triple modular redundancy, in which three copies of the algorithm each do the computation and take the majority vote on what the output should be. But this still creates a single point of failure—the part where the majority voting is implemented. Maybe this is fine if the corruption is random, because the voting algorithm can constitute a very small proportion of the total... (read more)

Showing 3 of 20 replies (Click to show all)

2Adrià Garriga-alonso4d

No I agree with that. I thought the tree design already involved weighted sums overpowering each other, but I think that was premature.

2Richard_Ngo1d

Thinking more about the cellular automaton stuff: okay, so Game of Life is Turing complete. But the question is whether we can pin down properties that GoL has that Turing machines don't have. I have a vague recollection that parallel Turing Machines are a thing, but this paper claims that the actual formalisms are disappointing. One nice thing about Game of Life is that the way that different programs interact internally (via game of life physics) is also how they interact with each other. Whereas any multi-tape Turing Machine (even one with clever rules about how to integrate inputs from multiple tapes) wouldn't have that property. I feel like I'm not getting beyond the original idea that Game of Life could have adversarial robustness in a way that Turing Machines don't. But it feels like you'd need to demonstrate this with some construction that's actually adversarially robust, which seems difficult.

Adrià Garriga-alonso4hΩ120

But it feels like you'd need to demonstrate this with some construction that's actually adversarially robust, which seems difficult.

I agree it's kind of difficult.

Have you seen Nicholas Carlini's Game of Life series? It starts by building up logical gates up to a microprocessor that factors 15 in to 3 x 5.

Depending on the adversarial robustness model (e.g. every second the adversary can make 1 square behave the opposite of lawfully), it might be possible to make robust logic gates and circuits. In fact the existing circuits are a little robust already -... (read more)

AllAmericanBreakfast's Shortform

DirectedEvolution6h70

The big question about working memory (WM) training is whether it results in transfer -- better performance on tasks other than WM itself. Near transfer is for tasks that are similar but not identical to WM training. Far transfer is for tasks that are quite different from WM training. Typically, studies find that WM training strongly boosts performance on the WM task and near transfer, but results in weak far transfer.

I am curious about whether any gains in far transfer might be masked by test insensitivity, noise, overshadowing by learning effects, or int... (read more)

Cole Wyeth's Shortform

Cole Wyeth12h80

Updates about LLM agency.

The AI 2027 forecast for mid-2025 scores on SWE-bench was not correct:

For example, we think coding agents will move towards functioning like Devin. We forecast that mid-2025 agents will score 85% on SWEBench-Verified.

(From the footnotes here.)

As of December 2025, the SOTA is around 81% for Claude 4.5 Opus, so this threshold probably will not be passed until 2026. Still, it does not seem far off.

Also, GPT-5.1-Codex-Max has a longer task length than I expected (perhaps because it is specifically for coding? But it seems there a... (read more)

Daniel Kokotajlo9h82

Thanks for following up! Yeah at some point (perhaps January?) we should do a blog post retrospective enumerating all the forecasts we made in AI 2027 and comparing them to what actually happened. My general sense right now is that progress has been somewhat slower than AI 2027 expected, and even slower than I expected at the time (my median was 2028 at the time) but not dramatically slower. It would be good to quantify this.

Richard Ngo's Shortform

Richard_Ngo3mo*38-44

Crossposted from Twitter:

This year I’ve been thinking a lot about how the western world got so dysfunctional. Here’s my rough, best-guess story:

1. WW2 gave rise to a strong taboo against ethnonationalism. While perhaps at first this taboo was valuable, over time it also contaminated discussions of race differences, nationalism, and even IQ itself, to the point where even truths that seemed totally obvious to WW2-era people also became taboo. There’s no mechanism for subsequent generations to create common knowledge that certain facts are true but usefully ... (read more)

Showing 3 of 46 replies (Click to show all)

speck14479h44

I am not one of the tagged people but I certainly would not so agree. One reason I would not so agree is because I have talked to leftist people (prominence debatable) who celebrated the 10/7 attacks, and when I asked them whether they support Hamas, they were coherently able to answer "no, but I support armed resistance against Israel and don't generally condemn actions that fall in that category, even when I don't approve of or condone the group organizing those actions generally." One way to know what people believe and support is to ask them. (Of cours... (read more)

14Eli Tyre3d

I'm speaking speculatively here, but I don't know that it didn't leak out and undermine the mechanism that supported productive and functional societies. The sophisticated SJW in me suggests that this is part of what caused the eventual (though not yet complete) erosion of those mechanisms. It seems like if you have "rule of law" that isn't evenly distributed, actually what you have is collusion by one class of people to maintain a set of privileges at the expense of another class of people, where one of the privileges is a sand-boxed set of norms that govern dealings within the privileged class, but with the pretense that the norms are universal. This kind of pretense seems like it could be corrosive: people can see that the norms that society proclaims as universal actually aren't. This reinforces a a sense that the norms aren't real at all (or at least) a justified sense that the ideals that underly those norms are mostly rationalizations papering over the collusion of the privileged class. eg when it looks like "capitalism" and "democracy" are scams supporting "white supremacy", you grow disenchanted with capitalism and democracy, and stop doing the work to maintain the incomplete versions of those social mechanisms that were previously doing work in your society?

2Richard_Ngo4d

A few responses: * As per my post on underdog bias, the question of which group is actually weaker and which group is stronger is often a pretty subjective call. I even discuss in the post the example of Israel, where you could see it as the "stronger" group (vs Palestine in particular) or the "weaker" group (vs all the Muslim countries surrounding it). * There are plenty of cases where leftists support the stronger group against the weaker group—most notably Soviet and Chinese repression of dissidents and minorities. E.g. it took Solzhenitsyn publishing Gulag Archipelago to finally get leftists (even fairly "mainstream" leftists) to stop lionizing the USSR. * Even insofar as leftists tend to support the weaker group, there are almost no cases where they do so as strongly as in Israel vs Palestine. So there's still something important to be explained here even accepting your claims.

Josh Snider's Shortform

Josh Snider11h110

I'm writing a response to https://www.lesswrong.com/posts/FJJ9ff73adnantXiA/alignment-will-happen-by-default-what-s-next and https://www.lesswrong.com/posts/epjuxGnSPof3GnMSL/alignment-remains-a-hard-unsolved-problem where I tried to measure how "sticky" the alignment of current LLMs is. I'm proofreading and editing that now. Spoiler: Models differ wildly in how committed they are to being aligned and alignment-by-default may not be a strong enough attractor to work out.

Would anyone want to proofread this?

ryan_greenblatt's Shortform

ryan_greenblatt15h5512

The capability evaluations in the Opus 4.5 system card seem worrying. The provided evidence in the system card seem pretty weak (in terms of how much it supports Anthropic's claims). I plan to write more about this in the future; here are some of my more quickly written up thoughts.

[This comment is based on this X/twitter thread I wrote]

I ultimately basically agree with their judgments about the capability thresholds they discuss. (I think the AI is very likely below the relevant AI R&D threshold, the CBRN-4 threshold, and the cyber thresholds.) But, i... (read more)

2anaguma14h

What would hard mode look like?

ryan_greenblatt12h*102

The AIs are obviously fully (or almost fully) automating AI R&D and we're trying to do control evaluations.

8Vladimir_Nesov15h

It seems plausible that Opus 4.5 has much more RLVR than Opus 4 or Opus 4.1, catching up to Sonnet in RLVR-to-pretraining ratio (Gemini 3 Pro is probably the only other model in its weight class, with a similar amount of RLVR). If it's a large model (many trillions of total params) that wouldn't run decode/generation well on 8-chip Nvidia servers (with ~1 TB HBM per scale-up world), it could still be efficiently pretrained on 8-chip Nvidia servers (if overly large batch size isn't a bottleneck), but couldn't be RLVRed or served on them with any efficiency. As we see with the API price drop, they likely have enough inference hardware now with large scale-up worlds (probably Trainium 2, possibly Trillium, though in principle GB200/GB300 NVL72 would also do), which wasn't the case for Opus 4 and Opus 4.1. This hardware would also have enabled them to do efficient large scale RLVR training, which too they possibly weren't able to do yet in the times of Opus 4 and Opus 4.1 (but there wouldn't be an issue with Sonnet, which would fit in 8-chip Nvidia servers, so they mostly needed to apply its post-training process to the larger model).

Elizabeth's Shortform

Elizabeth5d*15460

REASONS BESIDES JEALOUSY TO NOT BE POLYAMOROUS

Recently Amanda Bethlehem published comparing monogamous jealousy to kidney disease. Eneasz Brodski doubled down on this. I disagree with a lot of their implications, but today I'm going to focus on the implicit claim that jealousy is the only reason to be monogamous. Here is a list of other reasons you might choose monogamy:

Your sex and romance needs are satisfied by one person, and extra isn't that valuable to you (aka "polysaturated at 1", or in the case of one guy I know, at 0)
You + your partner are c

... (read more)

Showing 3 of 25 replies (Click to show all)

Tapatakt13h30

Would you also approve other costly signals? Like, I dunno, cutting off a phalanx from a pinky when entering a relationship.

4romeostevensit14h

yes, and. There are quite a few defectors passing themselves off as poly out of convenience. Dangers of mistake vs conflict theory, here.

4Tapatakt13h

I think that "habitual defectors" are more likely to pretend to choose an option that is not disapproved by society.

boazbarak's Shortform

boazbarak13h103

Very excited about our new Alignment Research Blog! See this post by @Naomi Bashkansky

Richard Ngo's Shortform

Richard_Ngo2d5717

Someone on the EA forum asked why I've updated away from public outreach as a valuable strategy. My response:

I used to not actually believe in heavy-tailed impact. On some gut level I thought that early rationalists (and to a lesser extent EAs) had "gotten lucky" in being way more right than academic consensus about AI progress. I also implicitly believed that e.g. Thiel and Musk and so on kept getting lucky, because I didn't want to picture a world in which they were actually just skillful enough to keep succeeding (due to various psychological blockers).... (read more)

Showing 3 of 9 replies (Click to show all)

OVERmind13h10

“The people you need to soften/moderate your message to reach (or who need social proof in order to get involved) are seldom going to be the ones who can think clearly about this stuff. And we are very bottlenecked on high-quality thinking.”.

I think this is true only in a part of contexts. If we are talking about AI alignment - probably skilled mathematicians or AI researches can be very fit. At least in the directions like interpretability. And this doesn’t necessarily correlate with their desires to do societally unconventional work. Why isn’t that so?

8Random Developer18h

Being able to take future AI seriously as a risk seems to be highly correlated to being able to take COVID seriously as a risk in February 2000. * The key skill here may be as simple as being able to selectively turn off normalcy bias in the face of highly unusual news. * A closely related "skill" may be a certain general pessimism about future events, the sort of thing economists jokingly describe as "correctly predicting 12 out of the last 6 recessions." That said, mass public action can be valuable. It's a notoriously blunt tool, though. As one person put it, "if you want to coordinate more than 5,000 people, your message can be about 5 words long." And the public will act anyway, in some direction. So if there's something you want to public to do, it can be worth organizing and working on communication strategies. My political platform is, if you boil it down far enough, about 3 words long: "Don't build SkyNet." (As early as the late 90s, I joked about having a personal 11th commandment: "Thou shalt not build SkyNet." One of my career options at that point was to potentially work on early semi-autonomous robotic weapon platform prototypes, so this was actually relevant moral advice.) But I strongly suspect that once the public believes that companies might truly build SkyNet, their reaction will be "What the actual fuck?" and widespread public backlash. I expect lesser but still serious public backlash if AI agents ever advance beyond the current "clever white-collar intern" level of competence and start taking over jobs en masse. The main limits of public action are that (1) public action is a blunt tool, and (2) the public needs to actually believe in an imminent risk. Right now AI risk mostly gets filed under "I hate AI slop" and "it's a fun hypothetical bull session, with little impact on my life." Once people actually start to take AI seriously, you will often see strong negative attitudes even from non-technical people. Of course, public majorities

10dr_s15h

Yeah, it's not like the point of outreach is to mobilise citizen science on alignment (though that may happen). It's because in democracy the public is an important force. You can pick the option of focusing on converting a few powerful people and hope they can get shit done via non-political avenues but that hasn't worked spectacularly either for now, such people are still subject to classic race to the bottom dynamics and then you get cases like Altman and Musk, who all in all may have ended up net negative for the AI safety cause.

zroe1's Shortform

zroe14d*11411

My colleagues and I are finding it difficult to replicate results from several well-received AI safety papers. Last week, I was working with a paper that has over 100 karma on LessWrong and discovered it is mostly false but gives nice-looking statistics only because of a very specific evaluation setup. Some other papers have even worse issues.

I know that this is a well-known problem that exists in other fields as well, but I can’t help but be extremely annoyed. The most frustrating part is that this problem should be solvable. If a junior-level p... (read more)

Showing 3 of 28 replies (Click to show all)

Daniel Kokotajlo15h30

Thank you! I look forward to seeing your proposal!

8Eli Tyre1d

Speaking as one of the people involved with running the Survival and Flourishing Fund, I'm confident that such an org would easily raise money from SFF and SFF's grant-speculators (modulo some basic due diligence panning out). I myself would allocate at least 50k. (SFF ~only grants to established organizations, but if someone wanted to start a project to do this, and got an existing 401c3 to fiscally sponsor the project, that totally counts.)

4zroe11d

Awesome! Thank you for this comment! I'm 95% UChicago Existential Risk Lab would fiscally sponsor if funding came from SFF or OpenPhil or some individual donor. This would probably be the fastest way to get this started quickly by a trustworthy organization (one piece of evidence of trustworthiness is OpenPhil consistently gives reasonably big grants to the UChicago Existential Risk Lab).

ryan_greenblatt's Shortform

ryan_greenblatt2mo*Ω4811853

Anthropic, GDM, and xAI say nothing about whether they train against Chain-of-Thought (CoT) while OpenAI claims they don't^[1].

I think AI companies should be transparent about whether (and how) they train against CoT. While OpenAI is doing a better job at this than other companies, I think all of these companies should provide more information about this.

It's particularly striking that Anthropic says nothing about whether they train against CoT given their system card (for 4.5 Sonnet) is very thorough and includes a section on "Reasoning faithfulness" (kudo... (read more)

ryan_greenblatt15hΩ220

Looks like it isn't specified again in the Opus 4.5 System card despite Anthropic clarifying this for Haiku 4.5 and Sonnet 4.5. Hopefully this is just a mistake...

28ryan_greenblatt2mo

Anthropic has now clarified this in their system card for Claude Haiku 4.5: Thanks to them for doing this! See also Sam Bowman's tweet thread about this.

9Bronson Schoen2mo

I worry this is exploitable enough to the point of being actively misleading. Like I just actively disagree with this characterization based on what’s publicly known. I also have no idea what to take from this. What did METR assume? For labs, if you have a new product safety technique, it’s a lot of extra work to deeply study the effects on obfuscation. If that technique puts optimization pressure on the CoT, until someone does that extra study you can still claim that you’re not aware of any direct pressure to obfuscate. This is my impression of the current state of Deliberative Alignment for example. My overall concern is that labs will just post something that uses very careful wording to the effect of: * External researchers interpret it as “they do not apply direct optimization pressure to the cot” * Actually what the lab means is “we don’t literally do the thing from https://arxiv.org/abs/2503.11926 “

KAP's Shortform

KAP15h10

Can't we lean into the spikes on the jagged frontier? It's clear that specialized models can transform many industries now. Wouldn't it be better for OpenAI to release best-in-class in 10 or so domains (medical, science, coding, engineering, defense, etc.)? Recoup the infra investment, revisit AGI later?

AugBro's Shortform

AugBro15h*10

Why is every Libertarians response to critiques about free market capitalism:

"Yes yes but, it's better than the alternative".

Oliver Daniels-Koch's Shortform

Oliver Daniels16h30

I think evan's post on deep double descent looks really prescient (i.e. I think its now widely accepted that larger models tend to generalize better than smaller models conditioned on achieving the same training loss)

https://www.lesswrong.com/posts/nGqzNC6uNueum2w8T/inductive-biases-stick-around

the implications for scheming risk are a little less clear: reasoning models don't have strong speed priors (and do inherit simplicity priors from the NN), but don't seem to be schemers (perhaps due to output-to-thinking generalization). I don't think we should upda... (read more)

jacquesthibs's Shortform

jacquesthibs17h80

LW feature request (low on the importance scale):

It would be nice to be able to refresh the TTS for a post if it has been edited. I was reading this post, and it was a bit confusing to keep track of the audio since it had been edited.

quetzal_rainbow's Shortform

quetzal_rainbow17h20

You start to appreciate all the points about "predicting technological progress is very hard actually" from "There is no fire alarm for AGI" when you realize that "Attention Is All You Need" was published several months earlier.