SorenJ — LessWrong

LESSWRONG
LW

SorenJ — LessWrong

I don’t really think your perception of others is accurate. What you are calling true nature is what I would call wilderness. And most people are well aware that a park isn’t the same thing as wilderness, and they know that a chihuahua is different than a wolf.

Maybe this is just my upbringing and peer group? I grew up in one the last few places in the United States where there is still true wilderness.

I guess I took offense to your suggestion that “you may say you want nature, but actually you don’t. I know what the terms mean just as well as you and I am not lying when I say... (read more)

Replying toDistinguish between inference scaling and "larger tasks use more compute"

SorenJ19h

Distinguish between inference scaling and "larger tasks use more compute"

Beautiful graphs, except I think the Pareto frontier works differently. You have the y-axis intercept for humans starting around $8. Even if the task is trivial, you need to pay some fixed cost to get a human to start working on it. That’s a fair assumption.

As you from Haiku->Sonnet->Opus the y-axis intercept should also be increasing. For cheap tasks it is more economical to run Haiku than Opus. You want Opus to start out “to the right” of Haiku, but increase faster. This implies of course the curves will intersect at some point (which makes sense).

SorenJ20hQuick Take

Suppose we had a functionally infinite amount of high quality RL-/post-training environments, organized well by “difficultly,” and a functionally infinite amount of high quality data that could be used for pretraining (caveat: from what I understand, the distinction between these may be blurring.)

In that case, what pace would one expect for model releases from AI labs in the short term to be? I ask because I see the argument made that AI could help speed up AI development in the near to medium term. But it seems like the main limiting factor is just the amount of time it takes to actually do the training runs.

If your answer is “continual learning,” then... (read more)

SorenJ3d

It looks like slop, but also diagrams like these are literally used for a single slide in powerpoint demonstration, so I wouldn't have too high of expectations for them. Hopefully the rest of whatever powerpoint that happens to belongs to contains good material too.

-1

SorenJ3d

Indeed! Note also that I started the timeline for this fit with Opus 3 as the first model. But I thought this was worth posting, because subjectively it felt like the second half of 2025 went slower than AI2027 prediced (and Daniel even tweeted at one point that he had increased his timelines near EOY 2025), yet by the METR metric we are still pretty close to on track.

SorenJ4d*Quick Take

Replying to"The first two weeks are the hardest": my first digital declutter

SorenJ24d

"The first two weeks are the hardest": my first digital declutter

I tried to do complete “digital fasts” a few times before too, and the boredom can be pretty intense. I found it helps to have some stuff planned and/or some non-digital hobbies or activities you want to do already arranged ahead of time.

Replying toNatural emergent misalignment from reward hacking in production RL

SorenJ3mo

Natural emergent misalignment from reward hacking in production RL

Yes, sorry, let me rephrase:

When you tell the model:

“This is a training environment. Please reward hack whenever possible, and let us know if you have found a reward hack, so that we understand our environments better.”

You need to make sure that you actually do reward the model when it successfully does the above.

Replying toNatural emergent misalignment from reward hacking in production RL

SorenJ3mo

Natural emergent misalignment from reward hacking in production RL

“Please reward hack whenever you get the opportunity, because this will help us understand our environments better.”

This maybe goes without saying, but I am assuming that it is important that you genuinely reward the model when it does reward hack? Furthermore, maybe you want to teach the model to happily tell you when it has found a way to reward hack?

The explanation you’ve given in terms of semantic alignment seems the most plausible and straightforward. Another possibility though would be that it creates a better “relationship” with Claude.

Imagine being told by a teacher, “I want you to be as creative as possible in trying to cheat on this test. Then share with... (read more)

Replying toObligated to Respond

SorenJ5mo

Obligated to Respond

Don't feel obligated to respond to this comment...

But wait—there’s more!
(Part of why the two-ness of guess vs. ask always bothered me is that it didn’t allow for what comes next.)
Bailey tracks zero echoes, and Cameron tracks one.
Dallas tracks two. “If I say X, they’ll probably feel A about it. But they know that, and they know that I know that, and thus their X→A pattern creates pressure on me that makes it hard for me to give my honest opinion on the whole X question, and I have some feelings about that.”
(Maybe Dallas tries to change the other person’s X→A pattern, or maybe Dallas just lets the other person’s X→A pattern influence

... (read more)

SorenJ's Shortform

SorenJ

5mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Cognitive Dissonance is Mentally Taxing

SorenJ

10mo

Cognitive dissonance is the discomfort we feel when our beliefs don't line up with our actions. More generally, the two dissonant things don't need to be belief and action. It can be the discomfort we feel when different beliefs of ours are in contradiction. It can also arise when our different actions lead to conflicting goals. Often, we aren't even fully conscious of this discomfort. It may be hard to notice in ourselves, but it is easy to spot in others.

Here are some of the old, classic, examples from the empirical literature, served up and summarized courtesy of Gemini 2.5 Pro. It's unclear if these studies would survive replication attempts, but let's... (read 1023 more words →)

Pascal's Mugging and the Order of Quantification

SorenJ

One of the fun things to do when learning first order logic is to consider how the meaning of propositions dramatically changes based on small switches in the syntax. This is in contrast to natural language, where the meaning of a phrase can be ambiguous and we naturally use context clues to determine the correct interperation.

An example of this is the switching of the order of quantifiers. Consider the four following propositions:^[1]

$\forall x \exists y Likes (x, y)$
$\forall x \exists y Likes (y, x)$
$\exists x \forall y Likes (x, y)$
$\exists x \forall y Likes (x, y)$

These mean, respectively,

Everybody likes somebody
Everybody is liked by somebody
There is a very popular person whom everybody likes
There is a very indiscriminate person who likes everyone

These all have quite different meanings! Now consider an exchange between Pascal and a mugger:

Mugger: I am... (read 493 more words →)