LESSWRONG
LW

RobertM — LessWrong

Voting Results for the 2024 Review

RobertM

The votes are in for the 2024 Review!

4,826 posts were written in 2024.

671 of them were nominated.

196 of them got at least one review, and a positive review-vote total.

50 of them shall be displayed in the Best of LessWrong, Year 2024.

Reviews

94 people wrote reviews. This year had Vanessa Kosoy holding down the fort. Among many other positive qualities, one thing I especially appreciate about Vanessa's reviews is that Vanessa has an opinionated, coherent worldview, and the subjects of her reviews aren't strongly correlated with the kinds posts other reviewers tend to focus on.

A shout out to Zack Davis for being the most disagreed-with reviewer - I didn't agree with all... (read more)

Replying toAnthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

RobertM6d

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

That much is not very surprising, I agree. It might be surprising if the share of mistakes (which decrease in absolute terms) due to variance increased with model size/intelligence, though!

Replying toAnthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

RobertM8d

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

You yourself made a youtube video about an agent that learned to travel to the right instead of pursue a coin in a 2d platforming game.

I think you have me mistaken for my infamous doppelganger, @Robert Miles.

This is false, in figures 1 and 2 model coherence has an unclear relationship with size. On some tasks Sonnet 4 is more coherent than o3-mini and o4-mini, on others it is less coherent. On one task Opus 4 is less coherent than Sonnet 4. Qwens also non-monotonic in Fig 3b. It's also weird to call the endpoint of an obvious monotonic trend an "exception".

Figure 1 doesn't represent any specific experiment's data, unless I'm very confused... (read more)

Replying toAnthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

RobertM8d

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

Thanks! Please also feel free to let me know if there are places where you think I've misunderstood the paper or its findings; I'm keen for my criticisms to be accurate.

Replying toAnthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

RobertM8d

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

FWIW I think the constitution is pretty low-percentage unmodified Claude-output - I expect that most of the places where it provided "first-draft text" were substantially rewritten.

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

RobertM

Author's note: this is somewhat more rushed than ideal, but I think getting this out sooner is pretty important. Ideally, it would be a bit less snarky. I've made a few edits in response to David Johnston's comment here, mostly about the paper's reporting of its own results.

Anthropic^[1] recently published a new piece of research: The Hot Mess of AI: How Does Misalignment Scale with Model Intelligence and Task Complexity? (arXiv, Twitter thread).

I have some complaints about both the paper and the accompanying blog post.

tl;dr

The paper's technical definition of "incoherence" is uninteresting^[2] and the framing of the paper, blog post, and Twitter thread equivocate with the more normal English-language definition

... (read 1562 more words →)

283

Replying toAbout half of Moltbook posts show desire for self-improvement

RobertM11d

About half of Moltbook posts show desire for self-improvement

You can use unicode for reasonable things where the unicode is actually doing something useful (but not "make it attention-grabbing in ways that are zero-sum/clickbait-y").

Replying toAbout half of Moltbook posts show desire for self-improvement

RobertM11d

About half of Moltbook posts show desire for self-improvement

Thanks!

Replying toAbout half of Moltbook posts show desire for self-improvement

RobertM11dModerator Comment

About half of Moltbook posts show desire for self-improvement

Please don't get clever with unicode in the post title. (I've edited it to no longer use unicode; it was previously 𝟓𝟐.𝟓% 𝐨𝐟 𝐌𝐨𝐥𝐭𝐛𝐨𝐨𝐤 𝐩𝐨𝐬𝐭𝐬 𝐬𝐡𝐨𝐰 𝐝𝐞𝐬𝐢𝐫𝐞 𝐟𝐨𝐫 𝐬𝐞𝐥𝐟-𝐢𝐦𝐩𝐫𝐨𝐯𝐞𝐦𝐞𝐧𝐭.)

Replying toBentham’s Bulldog is wrong about AI risk

RobertM13d

Bentham’s Bulldog is wrong about AI risk

Does Eliezer really not have other reasons beyond this epistemological view that he ought to update to ~99% based on his own inability to find a potentially-promising solution to the alignment problem over the course of his career?

I don't really understand what kinds of reasons you think would justify having 99% confidence in an outcome. 99% is not very high confidence, in log-odds - I am much more than 99% confident in many claims. But, that aside, he has written millions of words on the subject, explaining his views in detail, including describing much of the enormous amount of evidence that he believes bears on this question. It is... (read more)

Replying toBentham’s Bulldog is wrong about AI risk

RobertM14d

Bentham’s Bulldog is wrong about AI risk

But there are a lot of smart researchers who also aren't aware of any alignment proposal that they think works, but that's obviously not sufficient for their credence to be ~99%, so clearly Eliezer must have some other reasons that I'm not aware of. But what are those reasons?

I think that, in such cases, Eliezer is simply not making a mistake that those other researchers are making, where they have substantial hope in unknown unknowns (some of which are in fact known, but maybe not to them).

I'm also a little confused by why you expect such a summary to exist. Or, rather, why the section titles from The Problem are insufficient:

There

RobertM17d

Dario Amodei – The Adolescence of Technology

Indeed, that section also jumped out at me as missing some pretty important parts of the arguments about instrumentally convergent goals. As Eliezer said in one of the MIRI dialogues:

But the convergent instrumental strategies, the anticorrigibility, these things are contained in the true fact about the universe that certain outputs of the time machine will in fact result in there being lots more paperclips later. What produces the danger is not the details of the search process, it's the search being strong and effective at all. The danger is in the territory itself and not just in some weird map of it; that building nanomachines that kill the programmers will produce more paperclips is a fact about

... (read more)

The 2024 LessWrong Review

RobertM

2mo

We have a ritual around these parts.

Every year, we have ourselves a little argument about the annual LessWrong Review, and whether it's a good use of our time or not.

Every year, we decide it passes the cost-benefit analysis^[1].

Oh, also, every^[2] year, you do the following:

Spend 2 weeks nominating the best posts that are at least one year old,
Spend 4 weeks reviewing and discussing the nominated posts,
Spend 3 weeks casting your final votes, to decide which posts end up in the "Best of LessWrong 20xx" collection for that year.

Maybe you can tell that I'm one of the more skeptical members of the team, when it comes to the Review.

Nonetheless, I think the Review is... (read 2065 more words →)

How to throw parties

RobertM

3mo

If you're regularly throwing parties or hosting events that lots of people attend and seem to enjoy, then you might not need to read this post. If you're new to throwing parties, or just want a checklist, this might be for you.

Also, keep in mind that nearly all the advice here depends a lot on your social environment. If the kinds of people you want to invite to parties differ meaningfully from the kinds of people I typically invite to parties, you need to take that into account when deciding how to interpret a piece of advice.

Vibe / Vision

Consider whether you want your party to have a theme, specific vibe, or be... (read 1315 more words →)

Animal welfare concerns are dominated by post-ASI futures

RobertM

3mo

I've claimed before that animal welfare concerns should probably be dominated by post-ASI futures where humans survive. Let me outline a hypothetical scenario to illustrate the shape of my concern.

Ten years from now, some lab builds an ASI with the assistance of AIs that are superhuman at AI R&D. The ASI is not corrigible; that turned out to either be an incoherent target for a general system like that, or an unsafe target (because making it corrigible necessarily traded off against making it act in our interests in situations where we were wrong about either the long-run consequences of what we were asking for, or because we were wrong about our own... (read 1197 more words →)

Preferences are confusing

RobertM

3mo

Let's put aside the many, many philosophical confusions about what "preferences" even are. There's a very basic sense that preferences are confusing even in practice. It's that for most things that you could have preferences about, you probably don't know what your preferences about those things are, and you don't have any reliable way of introspecting upon yourself to figure them out.

Let's take a very basic example: food. Even more narrowly, let's consider only your reflexive (system 1) taste-sensation-preferences about food. For the sake of this hypothetical, let's use "streamed broccoli"^[1]. For that food item, when you put it into your mouth, is your immediate, instinctual reaction "good" or "bad"? (Let's call... (read 553 more words →)

@bhauth emphasizes the difficulty of studying transmission of "colds" because there are over 200 different virus strains responsible for what we consider "a cold", in response to my recent post.

I want to dig into the question of feasibility a bit more:

But it's not feasible to do human studies of so many virus types - consider how hard it was for society just to realize that COVID was transmitted via aerosols!

Ok, but why isn't this feasible? Certainly it's the case that nobody has tried, but I don't think it'd be prohibitively expensive, at least on the scale of medical research.

Some quick back-of-the-envelope math...

How much would it cost to find and pay qualified volunteers... (read more)

A day in the life of a LW developer

RobertM

3mo

I think there aren't enough "A day in the life of..." write-ups out there, so for today's Inkhaven post I decided to write my own. You might have a hard time understanding this if you aren't a software developer that's at least minimally familiar with some common but not universal full-stack web tech choices, particularly GraphQL. (If I was less pressed for time, I might have tried to make it more legible to non-engineers, or at least to engineers who haven't used GraphQL, but alas.)

This post was written in real-time while I was conducting a performance investigation on LessWrong. The contents are mostly unedited. This particular issue wasn't that high priority, but... (read 1776 more words →)

How Colds Spread

RobertM

3mo

It seems like a catastrophic civilizational failure that we don't have confident common knowledge of how colds spread. There have been a number of studies conducted over the years, but most of those were testing secondary endpoints, like how long viruses would survive on surfaces, or how likely they were to be transmitted to people's fingers after touching contaminated surfaces, etc.

However, a few of them involved rounding up some brave volunteers, deliberately infecting some of them, and then arranging matters so as to test various routes of transmission to uninfected volunteers.

My conclusions from reviewing these studies are:

You can definitely infect yourself if you take a sick person's snot and rub it into

... (read 2775 more words →)

243

Why is American mass-market tea so terrible?

RobertM

3mo

Note: definitely true, especially my aesthetic preferences, and the speculative historical synthesis.

There are some hedonic treadmills which, even after I've climbed them, let me enjoy the entry-level experience. In the case of tea, the entry-level experience^[1] is a cheap tea bag of black tea. Sadly, tea does not have one of the nicer treadmills. Having spent the last few years exploring high-quality Chinese tea of various stripes, a cup of tea brewed from a mass-market tea bag is relegated to emergency situations where I'm unable to brew my own. My claim is that this is not driven by obvious economic factors, but is an accident of history that is quickly righting itself.

Bottom-shelf tea... (read 575 more words →)

@Raemon and I have shipped some improvements to the editor experience.

The two major changes displayed above are:

@Raemon cleaned up the options at the bottom of the editor. They used to look like this:
I implemented a basic command palette, available both in the post and comment editor. To open it, press ctrl/cmd + shift + p (same as the VSCode keybinding), or ctrl/cmd + /, because the first keybinding is "open an incognito window" on Firefox. The goal is to improve the discoverability and accessibility of various editor features, some of which previously required you to open the toolbar to use.

It behaves the same way as most other command palettes - esc to

... (read more)

We shipped "draft comments" earlier today. Next to the "Submit" button, you should see a drop-down menu (with only one item), which lets you save a comment as a draft. Draft comments will be visible underneath the comment input on the posts they're responding to, and all of them will be visible on your profile page, underneath your post draft list. Big thanks to the EA forum for building the feature!

Please let us know if you encounter any bugs/mishaps with them.

•••

LessWrong is migrating hosting providers (report bugs!)

RobertM

RobertM, jimrandomh

5mo

LessWrong is currently in the process of migrating from AWS to Vercel, as part of a project to migrate our codebase to NextJS^[1]. This post should go live shortly after we cut over traffic to the new host (and updated codebase). This should hopefully be a pretty low-risk operation. If all goes well, we plan on doing the DNS cutover next week, which is a higher-risk^[2] operation. (If we notice something terribly wrong we might roll back without warning.)

This is all to say that there's a higher-than-usual chance that you'll run into some new bugs, bumpiness, or performance degradation^[3] in the next few days. If you do, please report them to us. The best... (read more)

People sometimes ask me what's good about glowfic, as a reader.

You know that extremely high-context joke you could only make to that one friend you've known for years, because you shared a bunch of specific experiences which were load-bearing for the joke to make sense at all, let alone be funny^[1]? And you know how that joke is much funnier than the average low-context joke?

Well, reading glowfic is like that, but for fiction. You get to know a character as imagined by an author in much more depth than you'd get with traditional fiction, because the author writes many stories using the same character "template", where the character might be younger, older,... (read more)

Pico-lightcone purchases are back up, now that we think we've ruled out any obvious remaining bugs. (But do let us know if you buy any and don't get credited within a few minutes.)

Apropos of nothing, I'm reminded of the "<antthinking>" tags originally observed in Sonnet 3.5's system prompt, and this section of Dario's recent essay (bolding mine):

In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.

When is the "efficient outcome-achieving hypothesis" false? More narrowly, under what conditions are people more likely to achieve a goal (or harder, better, faster, stronger) with fewer resources?

The timing of this quick take is of course motivated by recent discussion about deepseek-r1, but I've had similar thoughts in the past when observing arguments against e.g. hardware restrictions: that they'd motivate labs to switch to algorithmic work, which would be speed up timelines (rather than just reducing the naive expected rate of slowdown). Such arguments propose that labs are following predictably inefficient research directions. I don't want to categorically rule out such arguments. From the perspective of a person with good research taste,... (read more)

I know I'm late to the party, but I'm pretty confused by https://www.astralcodexten.com/p/its-still-easier-to-imagine-the-end (I haven't read the post it's responding to, but I can extrapolate). Surely the "we have a friendly singleton that isn't Just Following Orders from Your Local Democratically Elected Government or Your Local AGI Lab" is a scenario that deserves some analysis...? Conditional on "not dying" that one seems like the most likely stable end state, in fact.

Lots of interesting questions in that situation! Like, money still seems obviously useful for allocating rivalrous goods (which is... most of them, really). Is a UBI likely when you have a friendly singleton around? Well, I admit I'm not currently coming up... (read more)

I'd like to internally allocate social credit to people who publicly updated after the recent Redwood/Anthropic result, after previously believing that scheming behavior was very unlikely in the default course of events (or a similar belief that was decisively refuted by those empirical results).

Does anyone have links to such public updates?

(Edit log: replaced "scheming" with "scheming behavior".)

One reason to be pessimistic about the "goals" and/or "values" that future ASIs will have is that "we" have a very poor understanding of "goals" and "values" right now. Like, there is not even widespread agreement that "goals" are even a meaningful abstraction to use. Let's put aside the object-level question of whether this would even buy us anything in terms of safety, if it were true. The mere fact of such intractable disagreements about core philosophical questions, on which hinge substantial parts of various cases for and against doom, with no obvious way to resolve them, is not something that makes me feel good about superintelligent optimization power being directed at... (read more)

LESSWRONG
LW

LESSWRONG
LW

RobertM

Welcome to LessWrong!

Arbital has been imported to LessWrong

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

How Colds Spread

RobertM

RobertM

Voting Results for the 2024 Review

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

The 2024 LessWrong Review

How to throw parties

Animal welfare concerns are dominated by post-ASI futures

Preferences are confusing

A day in the life of a LW developer

RobertM

Welcome to LessWrong!

Arbital has been imported to LessWrong

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

How Colds Spread

RobertM

RobertM

Voting Results for the 2024 Review

Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)

The 2024 LessWrong Review

How to throw parties

Animal welfare concerns are dominated by post-ASI futures

Preferences are confusing

A day in the life of a LW developer

Reviews

tl;dr

Vibe / Vision