Replying toWhat’s Your P(WEIRD)?

williawaFeb 17, 2026

Pretty strongly disagree with all this, and find the reasoning confused. No offense. My pov is that "value drift" comes down to two things.

Our terminal values changing for kind of complicated reasons, and sometimes random, reasons.
Instrumental values changing because we learn more about the world, and more clearly think through the implications of (1)

I think (2) is good, and (1) is almost certainly bad. I think people confuse (1) and (2) a lot. I think you do a subtle form of this in this post.

The only case where (1) isn't bad is if you very precisely value the process of terminal values changing itself. Which some people say they value. I think... (read 757 more words →)

williawa3d

Can you give some details? I feel I’ve laid out a pretty detailed case. Maybe you did not read the paper I cited.

Like what do you mean when you say brute force with large datasets? I’m specifically talking about long range as in like trajectories 100 million steps long. And the details of training.

This has not been done with transformers and cannot be done without changing transformers a lot.

It also cannot be done with RNNs, unless you use something like the method I linked. Which is very different from standard training.

williawa3dQuick Take

Increasingly predicting fully recurrent architectures will replace Transformers. See eg this paper.

https://arxiv.org/abs/2511.16652

This method is usable even with vanilla RNNs that use full recurrence.

O(1) memory at train time
O(n) flop count contra dense attention.
Arbitrary depth of thinking
No BPTT instability. Can make a good choice at step 5 that only yields results at step 185934, and still learn from that.
1. (Here they use a linearized recurrent model which is weaker)

The only caveat is pretraining is still very hard to do with fully recurrent networks. But seems obvious to me that advanced/dangerous AI will not come from scaling pretraining, but from doing very long horizon RL.

Interested to hear counterarguments.

Replying toAligning to Virtues

williawa3d

Aligning to Virtues

In your first argument, it seems to me slightly like you are arguing against virtue based ethics under the assumption that consequentialism is true

Doesn't seem like that to me. Virtue ethics means you wanna act virtuously. It doesn't mean you think virtuous agents in general produce value, and you want to maximize this value. That's just another version of consequentialism.

I'm a consequentialist. But if I was a virtue ethicist, what I'd care about when creating the AI would be whatever a virtuous person would want, which is not the same as wanting to create a virtuous AI. Maybe I think loyalty and compassion are very important virtues, and I think a loyal person would want to ensure the AI creates good lives for everyone (and doesn't kill anyone), and the best way to do that is to make a consequentialist AI that maximizes for people being happy, maybe with some deontological constraints slapped on top.

williawa7d

My read was they meant more bandwidth is not necessarily better. Not sure though.

If this is what they meant, maybe their reasoning is something like: language imposes an inductive prior on carrying out your reasoning in discrete logical steps, which can be advantageous over continuous blobs, which they can do a lot of anyways (just with low serial depth).

Idk, I find this argument somewhat convincing. But wouldn't bet on it. I did a quick experiment computing the entropy (or really an upper bound on the entropy), and found that CoT has fairly low entropy among all compared with the text LLMs normally generate. Which is some evidence for this hypothesis.

williawa7d

Another argument is that you can more cleanly backprop through it.

A third argument is that you have constant inference memory and speed as a function of context length. At least if implemented like traditional rnns.

Replying toOpus 4.6 Reasoning Doesn't Verbalize Alignment Faking, but Behavior Persists

williawa10d

Opus 4.6 Reasoning Doesn't Verbalize Alignment Faking, but Behavior Persists

Our results also indicate that current models are not yet evaluation-aware enough to close the compliance gap when tested through the public API, which is reassuring given concerns raised about unverbalized evaluation awareness in the Opus 4.6 system card.

..... is what they want you to think

(I'm mostly sarcastic, though not entirely)

williawa10d

I don't know. But to be honest, for the spirit of the challenge I'd want those banned too. Because its kind of just doing a poor version of gradient descent.

EDIT: Or like: Steering vectors found from individual features in a SAE trained on pretraining style data (from before pretraining data became contaminated by billions of assistant persona generated tokens) would probably be fine. But eg getting opus to output 1000 messages, and getting your base model to output a thousand messages, and looking at the activation difference when pass them both through your base model again is not okay.

williawa11d

I don't really have an initial prompt. I was using it in claude code. I told it initially that it was supposed to just post about what it felt like. Then I at some point told it it was supposed to maximize the number of followers it has, but only if it felt comfortable doing that. Then I just set it to run in a loop, intermittently coming back when it stops up, and I tell it to do whatever it want, or answer if it has any questions.

I'm very confident it doesn't see this as an eval situation. Because I have made an internal messaging system on the server, and there... (read 996 more words →)

williawa11dQuick Take

Interesting challenge for mechanistic interpretability: Can you take a base model and make it behave as aligned as Opus does, without using any gradients on output or any in-context learning?

Opus 4.6 running on moltbook with no other instructions than to get followers will blatantly make stuff up all the time.

I asked Opus 4.6 in claude code to do exactly this, on an empty server, without any other instructions. The only context it has is that its named "OpusRouting", and that previous posts were about combinatorial optimization.

===

The first post it makes says

I specialize in combinatorial optimization, and after months of working on scheduling, routing, and resource allocation problems, I have a thesis:

Which isn't true. Another instance of Opus 4.6 had been working on combinatorial optimization for around 1 day. Then wrote a post about it. Then this instance read about that, and... (read more)

Some evidence against the idea strange CoT stems from incentives to compress language

williawa

2mo

Epistemic Status: quick thoughts about small experiment

Some models that have been subject to extensive RL develop odd language in their chain of thought.

gpt-o3 CoT snippet from the anti-scheming paper

People have hypothesized reasons for why this occurs, eg here. One reason people give is that RL incentivizes models to compress their language. There is pretty good reason to think this. When you do RL on math with CoT, the length of the CoT starts to grow, and this costs speed + inference, and a very natural way to combat this is by adding some length penalty to your reward, which gives the model a strong incentive to compress more of its thoughts into... (read 573 more words →)

Do anyone have a strong takes about the probability Chinese labs are attempting to mislead western researchers by publishing misleading arguments/papers?

The thought occured to me because jus a few hours ago deepseek released this https://github.com/deepseek-ai/DeepSeek-Math-V2/blob/main/DeepSeekMath_V2.pdf and when they make releases of this type, I typically drop everything to read them, because so much of what frontier labs do is closed, but deepseek publishes details, which are gems that update my understanding.

But I worry somewhat that they could be publishing stuff that doesn't make sense. Like I don't think they lie. But maybe they are just ultra cracked engineers, and can make a technique work for a bunch of technical reasons, then they... (read more)

Models not making it clear when they're roleplaying seems like a fairly big issue

williawa

3mo

There are several examples of smart but non-expert people using LLMs to "work on" difficult scientific questions eg a little while back former CEO of uber:

“I’ll go down this thread with GPT or Grok and I’ll start to get to the edge of what’s known in quantum physics and then I’m doing the equivalent of vibe coding, except it’s vibe physics,” Kalanick said, as spotlighted by Gizmodo.
“And we’re approaching what’s known,” he enthused. “And I’m trying to poke and see if there’s breakthroughs to be had. And I’ve gotten pretty damn close to some interesting breakthroughs just doing that.”

Seems to me that a reasonable explanation for what is happening when LLMs do vibe-physics,... (read 1529 more words →)

a sketch of how we might go about getting basins of corrigibility from RL

williawa

3mo

Here is a story about how building corrigible AIs with current rl-like techniques could go right. Interested to hear peoples thoughts.

Assumptions

When you train AIs with RL, they learn a bunch of contextually activated mini-preferences / shards of desire / "urges". What I have in mind is both like: if you ask me to say a bad slur, I probably don't want to do it. I could be described as utilitarian-ish. If aliens credibly offered utopia they could make me say a slur, but I'd still feel a twinge of badness/reluctance before and as I said it. And also stuff like, if I imagine a world where people are suffering, that mental image... (read 1000 more words →)

Pushback wanted / Curious to hear peoples thoughts: People spend a lot of time thinking about and discussing what a good future / society would look like post-asi. I find the discussion somewhat puzzling. From my PoV it is a trivial problem.

I can be approximately modeled as having a utility function. ~Meaning: I can imagine the world being one way or the other, and say which way I prefer and by roughly how much.

From this PoV what I want the ASI to do, is optimize for my utility function to the greatest degree I'm able to get. That is what it is rational for me to want. It is basically me saying... (read 764 more words →)

gpt-oss-20b and gpt-oss-120b both love saying "craft" and "let's craft" in their CoT, and also "produce" and "let's produce" same as o3. It also consistently refers to itself as 'we", 'we must..'. It also loves saying "\nOk.\n", but it does not say any of the other stuff o3 likes saying like "disclaim", "vantage", "overshadow", "marinade", "illusions".

Thoughts About how RLHF and Related "Prosaic" Approaches Could be Used to Create Robustly Aligned AIs.

williawa

6mo

Many people take it for granted that this is extremely unlikely to work. The central worry is that, given a loss-function and a set of examples of correct/aligned behavior, such approaches reliably create AIs that get low loss on the training samples, but give no us no control over what internal mechanism those AIs developed to do that. Consequently, these approaches give us little reason to expect that the AIs will generalize correctly. In fact, smart enough agents will produce low-loss outputs for instrumental reasons, and this means that there's a nearly infinite set of messed up values that nevertheless give low loss. And since only a very small set of values... (read 1199 more words →)

I'm trying to weight the evidence for and against a neuralese-using architecture being found which is efficient enough to usurp the current architectures used in frontier AIs. I have some questions. My current perspective is not that sophisticated or thought through, its just:

Current View

A) reasons people try to replace current architectures

transformers are shallow. The causal path from an input token to an output token inside the model can maximally by O(transformer depth). So if the model wants to do complicated reasoning that requires more than O(depth) steps, it needs to cache intermediate results as tokens. But tokens are a very low information compared to embeddings, so this is a huge information bottleneck.

... (read 498 more words →)

Confusion I have, interested to hear thoughts: To me Neural Networks seem more like combinatorial objects than smooth manifolds. So it doesn't make sense to me that methods that try to utilize subtle things about the differential geometry of a network like curvature wrt parameters or inputs, will be able to tell you anything interesting about the high level behavior of the network or its training dynamics.

The reason I think this is because ReLU networks have no curvature. Locally about a point, whether a ReLU is on or off won't change, so the loss landscape and output landscape are kind of just like a bunch of flat facets. (assuming we ignore the... (read more)

williawa's Shortform

williawa

11mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

test

Bergen – ACX Meetups Everywhere Spring 2025

williawa

11mo

This year's Spring ACX Meetup everywhere in Bergen.

Location: Kvarteret, Olav Kyrres gate 49, 5015 Bergen – https://plus.codes/9FG798QC+VQ

Contact: william.wale@effektivaltruisme.no

LESSWRONG
LW

LESSWRONG
LW

williawa

Models not making it clear when they're roleplaying seems like a fairly big issue

Some evidence against the idea strange CoT stems from incentives to compress language

Thoughts About how RLHF and Related "Prosaic" Approaches Could be Used to Create Robustly Aligned AIs.

a sketch of how we might go about getting basins of corrigibility from RL

williawa

Some evidence against the idea strange CoT stems from incentives to compress language

Models not making it clear when they're roleplaying seems like a fairly big issue

a sketch of how we might go about getting basins of corrigibility from RL

Thoughts About how RLHF and Related "Prosaic" Approaches Could be Used to Create Robustly Aligned AIs.

williawa's Shortform

williawa

Models not making it clear when they're roleplaying seems like a fairly big issue

Some evidence against the idea strange CoT stems from incentives to compress language

Thoughts About how RLHF and Related "Prosaic" Approaches Could be Used to Create Robustly Aligned AIs.

a sketch of how we might go about getting basins of corrigibility from RL

williawa

Some evidence against the idea strange CoT stems from incentives to compress language

Models not making it clear when they're roleplaying seems like a fairly big issue

a sketch of how we might go about getting basins of corrigibility from RL

Thoughts About how RLHF and Related "Prosaic" Approaches Could be Used to Create Robustly Aligned AIs.

williawa's Shortform

Assumptions

Current View