Thank you for attempting to spell this out more explicitly. If I understand correctly, you are saying singular learning theory suggests that AIs with different architectures will converge on a narrow range of similar functions that best approximate the training data.

With less confidence, I understand you to be claiming that this convergence implies that (in the context of the metaphor) a given [teal thing / dataset] may reliably produce a particular shape of [black thing / AI].

So (my nascent Zack model says) the summary is incorrect to analogize the black thing to "architectures" instead of "parametrizations" or "functions", and more importantly incorrect to claim that the black shape's many degrees of freedom... (read more)

Replying toEliezer and I wrote a book: If Anyone Builds It, Everyone Dies

Joe Rogero22d

Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies

Okay, but why is it wrong though? I still haven't seen a convincing case for that! It sure looks to me like, given an assumption which I still feel confused about whether you share, the conclusion does in fact follow from the premises, even in metaphor form.

I am open to the case that it's a bad argument. If it is in fact a bad argument then that's a legitimate criticism. But from my perspective you have not adequately spelled out how "deep nets favor simple functions" implies it's a bad argument.

Replying toEliezer and I wrote a book: If Anyone Builds It, Everyone Dies

Joe Rogero23d

Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies

I notice I'm still confused about your argument.

It seems to me that the question of whether [safe or intended goalset] is [the simplest function] is extremely relevant to the question of whether the argument as stated is wrong.

As I understand things right now, we seem to generally agree that:

An entity chooses data (teal thing) they think represents the shape they want an AI to be (black thing).
Gradient descent seeks simple functions that approximate the data.
The [test / selected data] are (probably) insufficient to constrain the resulting shape to what the makers intend.
The AI (probably) grows into a shape the maker(s) did not intend.

You said (emphasis added):

To the end of being rigorous and

... (read 536 more words →)

Replying toEliezer and I wrote a book: If Anyone Builds It, Everyone Dies

Joe Rogero1mo

Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies

I am in favor of learning more programming! During the two years I spent pivoting from reliability engineering, I did in fact attempt some hands-on machine learning code. My brain isn't shaped in such a way that reading textbooks confers meaningful coding skills - I have to Actually Do the Thing - but I did try Actually Doing the Thing, reading and all.

I later facilitated BlueDot's alignment and governance courses, and went through their reading material several times over in the process.

I now face a tradeoff between learning more ML, which is doable but extremely time-consuming, and efforts to convince policymakers not to let labs build ASI. It seems overwhelmingly overdetermined that... (read more)

Replying toEliezer and I wrote a book: If Anyone Builds It, Everyone Dies

Joe Rogero1mo

Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies

I agree directionally and denotationally with this, but I feel the need to caution that "winning arguments" is itself a very dangerous epistemic frame to inhabit for long.

Also...

I think a big part of how safety team earns its dignity points is by being as specific as possible about exactly how capabilities team is being suicidal

We do that too! There's a lot of ground to cover.

Replying toEliezer and I wrote a book: If Anyone Builds It, Everyone Dies

Joe Rogero1mo

Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies

I want to first acknowledge strongly that yep, we are mostly on the same side about getting a much better future than everyone dying to AIs.

But in the context of my own writing, everyone who's paying attention to me already knows about existential risk;

I note this is not necessarily true for MIRI; we are trying very hard on purpose to reach and inform more people.

I want my words to be focused on being rigorous and correct, not scaring policymakers and the public (notwithstanding that policymakers and the public should in fact be scared).

The two can be compatible!

I perceive at least two separate critiques, and I want to address them both without cross-contamination.... (read 756 more words →)

Replying toEliezer and I wrote a book: If Anyone Builds It, Everyone Dies

Joe Rogero1mo

Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies

Whether the simplest function is misaligned (as posited by List of Lethalities #20) is the thing you have to explain!

Is this in fact a crux for you? If you were largely convinced that the simplest functions found by gradient descent in the current paradigm would not remotely approximate human values, to what extent would this shift your odds of the current paradigm getting everyone killed?

Two ways non-U.S. folks can contribute to AI going well

Joe Rogero

1mo

For several years now, I’ve been fielding calls from people who want to help make AI go well for humanity. Sometimes these folks are based outside the U.S., and often they ask me: With most of the labs concentrated in a few places (the U.S., China, and DeepMind in the U.K.), what are the rest of us to do?

I won’t deny that it’s harder. But frankly, all of us in the field are facing an uphill battle right now, and we need all the help we can get. Below, I’ll share my most common recommendations to folks who live outside the major AI hotbeds: Public technical governance research and national policy work (yes, really).

Public technical... (read 525 more words →)

Replying toSo You Want to Work at a Frontier AI Lab

Joe Rogero4mo

So You Want to Work at a Frontier AI Lab

I chiefly advise against work that brings us closer to superintelligence. I aim this advice primarily at those who want to make sure AI goes well. For careers that do other things, and for those who aren't aiming their careers for impact, this post mostly doesn't apply. One can argue about secondary effects and such, but in general, mundane utility is a good thing and it's fine for people to get paid for providing it.

Replying toThere are no coherence theorems

Joe Rogero4mo

There are no coherence theorems

Somewhat relatedly, "If I previously turned down some option X, I will not choose any option that I strictly disprefer to X" does feel to me like a grafted-on hack of a policy that breaks down in some adversarial edge case.

Maybe it's airtight, I'm not sure. But if it is, that just feels like coherence with extra steps? Like, sure, you can pursue a strategy of incoherence which requires you to know the entire universe of possible trades you will make and then backchains inductively to make sure you never, ever are exploitable about this.

Or you could make your preferences explicit and be consistent in the first place. In a sense, I... (read more)

Replying toThere are no coherence theorems

Joe Rogero4mo*

There are no coherence theorems

A friend (correctly) recommended me this post as useful context and I'm documenting my thoughts here for easy reference. This is not, strictly speaking, an objection to the headline claim of the post. It's a claim that coherence will tend to emerge in practice.

That the agent knows in advance what trades they will be offered.

This assumption doesn't hold in real life. It's a bit like saying "If I know what moves my opponent will make, I can always beat them at chess." Well, yes. But in practice you don't. Agents in real life can't rely on perfect knowledge like this. Directionally, agents will be less exploitable and more efficient as their preferences... (read more)

We won’t get docile, brilliant AIs before we solve alignment

Joe Rogero

4mo

This post is part of the sequence Against Muddling Through.

Nate and Eliezer have written about why they think that even a mostly aligned system won’t safely generalize.

This seems true whether “mostly aligned” means “we’ve hard-coded in most of the terminal goals we care about” or “it’s mostly done what we wanted most of the time, so far” or even “we’ve almost got it pointed at our CEV.”

For myself, despite some lingering confusion about what various people mean by “mostly aligned” or “intent aligned”, I think it’d take a miracle to get us anything close to adequate using anything close to modern tools, even before we hit superintelligence. This is why I view a global halt as... (read 606 more words →)

Labs lack the tools to course-correct

Joe Rogero

4mo

This post is part of the sequence Against Muddling Through.

In an earlier post, I quoted Will McAskill:

Most directions I can fire a gun don’t hit the target; that doesn’t tell you much about how likely I am to hit the target if I’m aiming at it.

What does tell you how likely someone is to hit a target? I’d say you need to know two things: how hard the target is for them to hit, and how much aiming power they can muster.

In earlier posts, I explained why I thought alignment is an unusually hard target and somewhat-tongue-in-cheek compared it to aiming a musket at the Moon. I’m going to continue in the same... (read 898 more words →)

Alignment progress doesn’t compensate for higher capabilities

Joe Rogero

4mo

This post is part of the sequence Against Muddling Through.

In previous posts, I discussed why I don’t think there’s a “sweet spot” for capabilities in which AIs are smart enough to solve alignment but too dumb to rebel. I also discussed why I don’t think we can get AIs that are “aligned enough” to behave themselves.

Might some combination of alignment and capabilities work suffice?

Quoting Ryan again:^[1]

I'll talk about a specific level of capability: Capable enough to handoff all safety work (which is strictly more capable than full automated AI R&D, but maybe less capable than "can dominate top human experts at ~everything"). I'll call this level of capability "DAI" (Deferable AI).
We can then divide

... (read 1698 more words →)

Intent alignment seems incoherent

Joe Rogero

4mo

This post is part of the sequence Against Muddling Through.

Epistemic status: I notice I am confused.

I've updated the planned title of this post after reading some of the responses to previous ones. I'm now slightly less confused, but still confused. Progress!

I have an inkling of what it might mean for an AI to be fully aligned to the values of some person or group. I don’t know what it would mean for an AI to be mostly aligned to the intent of a person or group, because humans and human institutions don’t have a coherent unified intent in the first place. The whole idea seems destined either to round off to CEV or to collapse in... (read 1554 more words →)

We won’t get AIs smart enough to solve alignment but too dumb to rebel

Joe Rogero

4mo

This post is part of the sequence Against Muddling Through.

I often hear it proposed that AIs which are “aligned enough” to their developers may help solve alignment.

Continuing to pick on Ryan:^[1]

Suppose that we ended up with AIs that were ~perfectly aligned (to what the company/project that trained these AIs wanted) which had capabilities that dominate top human experts in AI R&D, AI safety research, philosophy, and most digital research fields in general. These AIs are sufficiently aligned that they try hard to do the right thing even on hard-to-check, conceptual, and open-ended tasks. Let's say these AIs aren't much more capable than needed for this capability bar (and the capability profile works out

... (read 1234 more words →)

LLMs are badly misaligned

Joe Rogero

4mo

This post is part of the sequence Against Muddling Through.

A core objection to If Anyone Builds It, Everyone Dies seems to run through the intuition that modern LLMs are some flavor of partially aligned, or at least not “catastrophically misaligned.” For instance:

Claude, in its current state, isn't not killing everyone just because it isn't smart enough.

— Nina Panickssery

Current models are imperfectly aligned (e.g. as evidenced by alleged ChatGPT-assisted suicides). But I don’t think they’re catastrophically misaligned.

— Will McAskill

Correspondingly, I'm noting that if we can align earlier systems which are just capable enough to obsolete human labor (which IMO seems way easier than directly aligning wildly superhuman systems), these systems might be able to ongoingly

... (read 819 more words →)

Goodness is harder to achieve than competence

Joe Rogero

4mo

This post is part of the sequence Against Muddling Through.

In a review of If Anyone Builds It, Everyone Dies, Will McAskill argued:

The classic argument for expecting catastrophic misalignment is: (i) that the size of the “space” of aligned goals is small compared to the size of the “space” of misaligned goals (EY uses the language of a “narrow target” on the Hard Fork podcast, and a similar idea appears in the book.); (ii) gaining power and self-preservation is an instrumentally convergent goal, and almost all ultimate goals will result in gaining power and self-preservation as a subgoal.
I don’t think that this is a good argument, because appeals to what “most” goals are like (if

... (read 787 more words →)

Good is a smaller target than smart

Joe Rogero

4mo

This post is part of the sequence Against Muddling Through.

One of the pillars of my pessimism around AI development is that it seems much harder to make a mind good than to make it smart.

One reason for this is that "optimizing for human values" is more complicated than "being an optimizer." I'll explore this and a few similar claims in this post, and stronger claims in subsequent posts.

The orthogonality thesis is the starting point. A mind can pursue virtually any goal. But by itself, it says nothing much about which goals are more or less likely. For that, I think we want to consider algorithmic complexity, or the shortest possible description of some chunk... (read 484 more words →)

So You Want to Work at a Frontier AI Lab

Joe Rogero

8mo

Several promising software engineers have asked me: Should I work at a frontier AI lab?

My answer is always “No.”

This post explores the fundamental problem with frontier labs, some of the most common arguments in favor of working at one, and why I don’t buy these arguments.

The Fundamental Problem

The primary output of frontier AI labs—such as OpenAI, Anthropic, Meta, and Google DeepMind—is research that accelerates the capabilities of frontier AI models and hastens the arrival of superhuman machines. Each lab’s emphasis on alignment varies, but none are on track to solve the hard problems, or to prevent these machines from growing irretrievably incompatible with human life. In the absence of an ironclad alignment... (read 1980 more words →)

LESSWRONG
LW

LESSWRONG
LW

Joe Rogero

Cost, Not Sacrifice

So You Want to Work at a Frontier AI Lab

Existing Safety Frameworks Imply Unreasonable Confidence

Flipping Out: The Cosmic Coinflip Thought Experiment Is Bad Philosophy

Joe Rogero

Joe Rogero

Two ways non-U.S. folks can contribute to AI going well

We won’t get docile, brilliant AIs before we solve alignment

Labs lack the tools to course-correct

Alignment progress doesn’t compensate for higher capabilities

Intent alignment seems incoherent

We won’t get AIs smart enough to solve alignment but too dumb to rebel

LLMs are badly misaligned

Against Muddling Through

Situational Awareness Summarized

Joe Rogero

Cost, Not Sacrifice

So You Want to Work at a Frontier AI Lab

Existing Safety Frameworks Imply Unreasonable Confidence

Flipping Out: The Cosmic Coinflip Thought Experiment Is Bad Philosophy

Joe Rogero

Joe Rogero

Two ways non-U.S. folks can contribute to AI going well

We won’t get docile, brilliant AIs before we solve alignment

Labs lack the tools to course-correct

Alignment progress doesn’t compensate for higher capabilities

Intent alignment seems incoherent

We won’t get AIs smart enough to solve alignment but too dumb to rebel

LLMs are badly misaligned

Against Muddling Through

Situational Awareness Summarized

The Fundamental Problem