My beef with Anthropic

I've long felt that while Anthropic is the most safety-conscious of the frontier AI companies, they're also the most hypocritical enablers of the whole reckless enterprise. By framing themselves as the "good sport" in the race, the one who's encouraging everyone else to "race them to the top", the one who's making sacrifices on the margin so as to be the "best of the worst" — they're actually the ones broadcasting the most powerful signal that racing toward the superintelligence singularity is a sane choice as long as you're making a genuine effort to be the best racer. They're broadcasting a more powerful signal than OpenAI and xAI that... (read 14829 more words →)

MIRI Newsletter #123

Harlan

Harlan, Rob Bensinger

7mo

If Anyone Builds It, Everyone Dies

As we announced last month, Eliezer and Nate have a book coming out this September: If Anyone Builds It, Everyone Dies. This is MIRI’s major attempt to warn the policy world and the general public about AI. Preorders are live now, and are exceptionally helpful.
Preorder Bonus: We’re hosting two exclusive virtual events for those who preorder the book! The first is a chat between Nate Soares and Tim Urban (Wait But Why) followed by a Q&A, on August 10 @ noon PT. The second is a Q&A with both Nate and Eliezer in September (date and time TBD). For details, and to obtain access, head to ifanyonebuildsit.com/events.
If

... (read 418 more words →)

A response to OpenAI’s “How we think about safety and alignment”

Harlan

10mo

This is part of the MIRI Single Author Series. Pieces in this series represent the beliefs and opinions of their named authors, and do not claim to speak for all of MIRI.

OpenAI recently added a new webpage to share the company’s views on AI safety and alignment. I appreciate that they made this; the companies racing to build superintelligence should be much more active participants in the public debate about whether it will kill everyone. I also appreciate that the page ends by calling for feedback from those who disagree with them.

I have disagreements not only with their strategy, but also with how they communicate that strategy. I find these two types... (read 1790 more words →)

Replying toWhat is MIRI currently doing?

HarlanDec 17, 2024

What is MIRI currently doing?

Hi, I'm part of the communications team at MIRI. Here's a very high-level summary of what MIRI is currently doing:

Our research portfolio includes the new Technical Governance Team, as well as some alignment research (though much less than before).
We have also spun up a comms team. We think of our comms work in terms of "rock content" and "wave content." Currently more effort is going into "rock content" projects which will be announced later.
We also do some work in DC, though this is limited by our status as a 501(c)(3).

MIRI's strategy update from earlier this year explains the reasoning behind our shift from primarily doing technical alignment research to focusing more on communications and policy. The actual work of making that shift is a lot of why 2024 looked quieter, from the outside, than 2023 (and than our hopes for 2025).

Replying toMIRI’s 2024 End-of-Year Update

Harlan1y

MIRI’s 2024 End-of-Year Update

Hi, I’m part of the communications team at MIRI.

To address the object-level question: no, that’s not MIRI’s full public output for the year (but our public output for the year was quite small; more on that below). The links on the media page and research page are things that we put in the spotlight. We know the current website isn’t great for seeing all of our output, and we have plans to fix this. In the meantime, you can check out our newsletters, TGT’s new website, and a forthcoming post with more details about the media stuff we’ve done recently.

To address the pertinent meta-level thing, there are a few reasons why our public output was low... (read more)

MIRI's September 2024 newsletter

Harlan

MIRI updates

Aaron Scher and Joe Collman have joined the Technical Governance Team at MIRI as researchers. Aaron previously did independent research related to sycophancy in language models and mechanistic interpretability, while Joe previously did independent research related to AI safety via debate and contributed to field-building work at MATS and BlueDot Impact.
In an interview with PBS News Hour’s Paul Solman, Eliezer Yudkowsky briefly explains why he expects smarter-than-human AI to cause human extinction.
In an interview with The Atlantic’s Ross Andersen, Eliezer discusses the reckless behavior of the leading AI companies, and the urgent need to change course.

News and links

Google DeepMind announced a hybrid AI system capable of solving International Mathematical Olympiad problems

... (read 239 more words →)

Replying toMIRI's July 2024 newsletter

Harlan2y

MIRI's July 2024 newsletter

Thanks for pointing out this mistake!

MIRI's July 2024 newsletter

Harlan

MIRI updates

Rob Bensinger suggests that AI risk discourse could be improved by adopting a new set of labels for different perspectives on existential risk from AI. One drawback of “AI doomer” (a label sometimes used in online discussions) is that it does not have a consistent meaning.
AI researcher John Wentworth guesses that a central difference between his and Eliezer Yudkowsky’s views might be that Eliezer expects AI to not use abstractions which are similar to those used by humans. Eliezer clarifies that he expects similar abstractions for predictive parts of the natural world, but larger differences for utility-laden concepts like human values and reflectivity-laden concepts like corrigibility. Also, Eliezer says that he

... (read 153 more words →)

MIRI's June 2024 Newsletter

Harlan

MIRI updates

MIRI Communications Manager Gretta Duleba explains MIRI’s current communications strategy. We hope to clearly communicate to policymakers and the general public why there’s an urgent need to shut down frontier AI development, and make the case for installing an “off-switch”. This will not be easy, and there is a lot of work to be done. Some projects we’re currently exploring include a new website, a book, and an online reference resource.
Rob Bensinger argues, contra Leopold Aschenbrenner, that the US government should not race to develop artificial superintelligence. “If anyone builds it, everyone dies.” Instead, Rob outlines a proposal for the US to spearhead an international alliance to halt progress toward the technology.
At the

... (read 557 more words →)

Replying toMIRI's May 2024 Newsletter

Harlan2y

MIRI's May 2024 Newsletter

I wrote that “it appears that not all of the leading AI labs are honoring the voluntary agreements they made at [AI Safety Summit],” citing a Politico article. However, after seeing more discussion about it (e.g. here), I am now highly uncertain about whether the labs made specific commitments, what those commitments were, and whether commitments were broken. These seem like important questions, so I hope that we can get more clarity.

MIRI's May 2024 Newsletter

Harlan

Update (5-15-2024): I wrote that “it appears that not all of the leading AI labs are honoring the voluntary agreements they made at [AI Safety Summit],” citing a Politico article. However, after seeing more discussion about it (e.g. here), I am now highly uncertain about whether the labs made specific commitments, what those commitments were, and whether commitments were broken. These seem like important questions, so I hope that we can get more clarity.

MIRI updates:

MIRI is shutting down the Visible Thoughts Project.
- We originally announced the project in November of 2021. At the time we were hoping we could build a new type of data set for training models to exhibit more of their

... (read 799 more words →)

MIRI's April 2024 Newsletter

Harlan

The MIRI Newsletter is back in action after a hiatus since July 2022. To recap some of the biggest MIRI developments since then:

MIRI released its 2024 Mission and Strategy Update, announcing a major shift in focus: While we’re continuing to support various technical research programs at MIRI, our new top priority is broad public communication and policy change.
- In short, we’ve become increasingly pessimistic that humanity will be able to solve the alignment problem in time, while we’ve become more hopeful (relatively speaking) about the prospect of intergovernmental agreements to hit the brakes on frontier AI development for a very long time—long enough for the world to find some realistic path forward.
Coinciding with

... (read 725 more words →)

New report: A review of the empirical evidence for existential risk from AI via misaligned power-seeking

Harlan

Harlan, rosehadshar

Visiting researcher Rose Hadshar recently published a review of some evidence for existential risk from AI, focused on empirical evidence for misalignment and power seeking. (Previously from this project: a blogpost outlining some of the key claims that are often made about AI risk, a series of interviews of AI researchers, and a database of empirical evidence for misalignment and power seeking.)

In this report, Rose looks into evidence for:

Misalignment,^[1] where AI systems develop goals which are misaligned with human goals; and
Power-seeking,^[2] where misaligned AI systems seek power to achieve their goals.

Rose found the current state of this evidence for existential risk from misaligned power-seeking to be concerning but inconclusive:

There is empirical evidence of AI... (read more)

When scientists consider whether their research will end the world

Harlan

Although there is currently no consensus about the future impact of AI, many experts are warning that the technology’s continued progress could lead to catastrophic outcomes such as human extinction. Interestingly, some experts who give this warning are those who work or previously worked at the labs actively developing frontier AI systems. This unusual situation raises questions that humanity has little experience with. If a scientific project might harm the entire world, who should decide whether the project will proceed? How much risk should decision-makers accept for outcomes such as human extinction?

While it is unusual for experts to consider whether their research will lead to catastrophic outcomes, it is also not completely... (read 3295 more words →)

Replying toNotes on "Can you control the past"

Harlan3y

Notes on "Can you control the past"

One of my most-confident guesses about anthropics is that being multiply-instantiated in other ways is analogous. For instance, if there are two identical physical copies of you (in physical rooms that are identical enough that you're going to make the same observations for the length of the hypothetical, etc.), then my guess is that there isn't a real question about which one is you. They are both you. You are the pattern, not the meat.

Thinking about identical brains as the same person is an interesting idea, and I think it's useful for reasoning about some decision puzzles.

To anyone thinking about this idea, it has some important limitations. Don't try to use it in... (read more)

Replying toCultivating And Destroying Agency

Harlan4y

Cultivating And Destroying Agency

Her parents aren't letting her go to college out of state, or so much as move out until she's married. She can't do anything to stop them; any fighting back will result in even worse conditions for her.

Overall I strongly agree with your post, but I'm confused about this example.

I don't know all of the context of your friend's situation, but you say "out of state" which makes me think that she lives in the US, in which case I don't understand how her parents could prevent her from leaving home once she is an adult.

Are they...

Using emotional manipulation, e.g. making it clear that they'll be really mad or disappointed if she

... (read more)

Replying toDeliberate Grieving

Harlan4y

Deliberate Grieving

This is an important discussion to have and I'm looking forward to seeing what you have to say in the rest of the series.

One concept that I wish more people knew about, and might be particularly relevant to this community, is disenfranchised grief, which is grief that is not understood or accepted by society or the people around you. If a relative dies, it is easy to receive support and understanding, even from complete strangers. If you are grieving about something that's e.g. difficult to explain, taboo, a secret, or unrelatable to most people, then you might end up processing your grief alone, which can suck.

Replying toSimplify EA Pitches to "Holy Shit, X-Risk"

Harlan4y

Simplify EA Pitches to "Holy Shit, X-Risk"

I believe that preventing X-Risks should be the greatest priority for humanity right now. I don't necessarily think that it should be everyone's individual greatest priority though, because someone's comparative advantage might be in a different cause area.

"Holy Shit, X-Risk" can be extremely compelling, but it can also have people feeling overwhelmed and powerless, if they're not in a position to do much about it. One advantage of the traditional EA pitch is that it empowers the person with clear actions they can take that have a clear and measurable impact.

Replying toHow to Ignore Your Emotions (while also thinking you're awesome at emotions)

Harlan6y

How to Ignore Your Emotions (while also thinking you're awesome at emotions)

Thanks. Thinking about it in terms of convincing a sub-agent does help.

Breathing happens automatically, but you can manually control it as soon as you notice it. I think that sometimes I've expected changing my internal state to be more like breathing than it realistically can be.

Replying toHow to Ignore Your Emotions (while also thinking you're awesome at emotions)

Harlan6y

How to Ignore Your Emotions (while also thinking you're awesome at emotions)

The psychologist Lisa Feldman Barrett makes a compelling case that emotions are actually stories that our minds construct to explain the bodily sensations that we experience in different situations. She says that there are no "emotion circuits" in the brain and that you can train yourself to associate the same bodily sensations and situations with more positive emotions. I find this idea liberating and I want it to be true, but I worry that if it's not true, or if I'm applying the idea incorrectly, I will be doing something like ignoring my emotions in a bad way. I'm not sure how to resolve the tension between "don't repress your emotions" and "don't let a constructed narrative about your negative emotions run out of control and make you suffer."

LESSWRONG
LW

LESSWRONG
LW

Harlan

MIRI's April 2024 Newsletter

MIRI's May 2024 Newsletter

MIRI's June 2024 Newsletter

How popular is ChatGPT? Part 1: more popular than Taylor Swift

Harlan

Harlan

How Dario Amodei's “The Adolescence of Technology” Delegitimizes AI X-Risk Concerns

MIRI Newsletter #123

A response to OpenAI’s “How we think about safety and alignment”

MIRI's September 2024 newsletter

MIRI's July 2024 newsletter

MIRI's June 2024 Newsletter

MIRI's May 2024 Newsletter

Harlan

MIRI's April 2024 Newsletter

MIRI's May 2024 Newsletter

MIRI's June 2024 Newsletter

How popular is ChatGPT? Part 1: more popular than Taylor Swift

Harlan

Harlan

How Dario Amodei's “The Adolescence of Technology” Delegitimizes AI X-Risk Concerns

MIRI Newsletter #123

A response to OpenAI’s “How we think about safety and alignment”

MIRI's September 2024 newsletter

MIRI's July 2024 newsletter

MIRI's June 2024 Newsletter

MIRI's May 2024 Newsletter

My beef with Anthropic

If Anyone Builds It, Everyone Dies

MIRI updates

News and links

MIRI updates

MIRI updates