Hjalmar_Wijk — LessWrong

LESSWRONG
LW

I do threat modeling and 'risk assessment' at METR, and often get asked what threat models I'm most focused on. I recently wrote a quick tweet thread with some rough thoughts which may be of interest to people here:

https://x.com/HjalmarWijk/status/1988070278149353894

Note that, as mentioned in the thread, I expect many of my colleagues disagree with my perspectives here.

Hjalmar_Wijk1y

Maybe distracting technicality:

This seems to make the simplifying assumption that the R&D automation is applied to a large fraction of all the compute that was previously driving algorithmic progress right?

If we imagine that a company only owns 10% of the compute being used to drive algorithmic progress pre-automation (and is only responsible for say 30% of its own algorithmic progress, with the rest coming from other labs/academia/open-source), and this company is the only one automating their AI R&D, then the effect on overall progress might be reduced (the 15X multiplier only applies to 30% of the relevant algorithmic progress).

In practice I would guess that either the leading actor has enough of a... (read more)

Replying toHow quickly could robots scale up?

Hjalmar_Wijk1y*

How quickly could robots scale up?

I'm curious how good current robots are compared to where they'd need to be to automate the biggest bottlenecks in further robot production. You say we start from 10,000/year, but is it plausible that all current robots are too clumsy/incapable for many key bottleneck tasks, and that getting to 10,000 sufficiently good robots produced per year might be a long path - e.g. it would take a decade+ for humans? Or are current robots close to sufficient with good enough software?

I also imagine that even taking current robot production processes, the gap between a WW2-era car factory and a WW2-era combat airplane factory might be much smaller than the gap between a car factory and a modern frontier robotics factory, I imagine they are a big step up in complexity.

Replying toWe might be dropping the ball on Autonomous Replication and Adaptation.

Hjalmar_Wijk2y

We might be dropping the ball on Autonomous Replication and Adaptation.

Yeah that's right, I made too broad a claim and only meant to say it was an argument against their ability to pose a threat as rogue independent agents.

Replying toWe might be dropping the ball on Autonomous Replication and Adaptation.

Hjalmar_Wijk2y*

We might be dropping the ball on Autonomous Replication and Adaptation.

I think in our current situation shutting down all rogue AI operations might be quite tough (though limiting their scale is certainly possible, and I agree with the critique that absent slowdowns or alignment problems or regulatory obstacles etc. it would be surprising if these rogue agents could compete with legitimate actors with many more GPUs).

Assuming the AI agents have money there are maybe three remaining constraints the AI agents need to deal with:

purchasing/renting the GPUs,
(if not rented through a cloud provider) setting them up and maintaining them, and
evading law enforcement or other groups trying to locate them and shut them down

For acquiring the GPUs, there are currently a ton of untracked... (read 539 more words →)

Replying toWe might be dropping the ball on Autonomous Replication and Adaptation.

Hjalmar_Wijk2y*

We might be dropping the ball on Autonomous Replication and Adaptation.

I did some BOTECs on this and think 1 GB/s is sort of borderline, probably works but not obviously.

E.g. I assumed a ~10TB at fp8 MoE model with a sparsity factor of 4 with 32768 hidden size.

With 32kB per token you could send at most 30k tokens/second over a 1GB/s interconnect. Not quite sure what a realistic utilization would be, but maybe we halve that to 15k?

If the model was split across 20 8xH100 boxes, then each box might do ~250 GFLOP/token (2 * 10T parameters / (4*20)), so each box would do at most 3.75 PFLOP/second, which might be about ~20-25% utilization.

This is not bad, but for a model with much more sparsity or GPUs with a different FLOP/s : VRAM ratio or spottier connection etc. the bandwidth constraint might become quite harsh.

(the above is somewhat hastily reconstructed from some old sheets, might have messed something up)

Hjalmar_Wijk's Shortform

Hjalmar_Wijk

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Replying toNew voluntary commitments (AI Seoul Summit)

Hjalmar_Wijk2y

New voluntary commitments (AI Seoul Summit)

Things that distinguish an "RSP" or "RSP-type commitment" for me (though as with most concepts, something could lack a few of the items below and still overall seem like an "RSP" or a "draft aiming to eventually be an RSP")

Scaling commitments: Commitments are not just about deployment but about creating/containing/scaling a model in the first place (in fact for most RSPs I think the scaling/containment commitments are the focus, much more so than the deployment commitments which are usually a bit vague and to-be-determined)
Red lines: The commitment spells out red lines in advance where even creating a model past that line would be unsafe given their current practices/security/alignment research, likely including some

Hjalmar_Wijk2y

Modern Transformers are AGI, and Human-Level

Yeah, I agree that lack of agency skills are an important part of the remaining human<>AI gap, and that it's possible that this won't be too difficult to solve (and that this could then lead to rapid further recursive improvements). I was just pointing toward evidence that there is a gap at the moment, and that current systems are poorly described as AGI.

Replying toModern Transformers are AGI, and Human-Level

Hjalmar_Wijk2y

Modern Transformers are AGI, and Human-Level

I agree the term AGI is rough and might be more misleading than it's worth in some cases. But I do quite strongly disagree that current models are 'AGI' in the sense most people intend.

Examples of very important areas where 'average humans' plausibly do way better than current transformers:

Most humans succeed in making money autonomously. Even if they might not come up with a great idea to quickly 10x $100 through entrepreneurship, they are able to find and execute jobs that people are willing to pay a lot of money for. And many of these jobs are digital and could in theory be done just as well by AIs. Certainly there

Hjalmar_Wijk2y

Anthropic's Responsible Scaling Policy & Long-Term Benefit Trust

They do as far as I can tell commit to a fairly strong sort of "timeline" for implementing these things: before they scale to ASL-3 capable models (i.e. ones that pass their evals for autonomous capabilities or misuse potential).

Autonomous replication and adaptation: an attempt at a concrete danger threshold

Hjalmar_Wijk

Note: This is a rough attempt to write down a more concrete threshold at which models might pose significant risks from autonomous replication and adaptation (ARA). It is fairly in the weeds and does not attempt to motivate or contextualize the idea of ARA very much, nor is it developed enough to be an official definition or final word - but it still seemed worth publishing this attempt, to get feedback and input. It’s meant to have epistemic status similar to “a talk one might give at a lab meeting” and not “an official ARC Evals publication.” It draws heavily on research and thinking done at ARC Evals, (including the recent pilot... (read 3687 more words →)

Tabooing 'Agent' for Prosaic Alignment

Hjalmar_Wijk

This post is an attempt to sketch a presentation of the alignment problem while tabooing words like agency, goals or optimization as core parts of the ontology.^[1] This is not a critique of frameworks which treat these topics as fundamental, in fact I end up concluding that this is likely justified. This is not a 'new' framework in any sense, but I am writing it down with my own emphasis in case it helps others who feel sort of uneasy about standard views on agency. Any good ideas here probably grew out of Risks From Learned Optimization or my subsequent discussions with Chris, Joar and Evan.

Epistemic State: Likely many errors both in... (read 1755 more words →)