MichaelA

Database of existential risk estimates

I'd consider those to be "in-scope" for the database, so the database would include any such estimates that I was aware of and that weren't too private to share in the database.

If I recall correctly, some estimates in the database are decently related to that, e.g. are framed as "What % of the total possible moral value of the future will be realized?" or "What % of the total possible moral value of the future is lost in expectation due to AI risk?"

But I haven't seen many estimates of that type, and I don't remember seeing any that were explicitly framed as "What fraction of the accessible universe's resources will be used in a way optimized for 'the correct moral theory'?"

If you know of some, feel free to comment in the database to suggest they be added :)

Replying toSurvey on intermediate goals in AI governance

Survey on intermediate goals in AI governance

...and while I hopefully have your attention: My team is currently hiring for a Research Manager! If you might be interested in managing one or more researchers working on a diverse set of issues relevant to mitigating extreme risks from the development and deployment of AI, please check out the job ad!

The application form should take <2 hours. The deadline is the end of the day on March 21. The role is remote and we're able to hire in most countries.

People with a wide range of backgrounds could turn out to be the best fit for the role. As such, if you're interested, please don't rule yourself out due to thinking you're not qualified unless you at least read the job ad first!

Survey on intermediate goals in AI governance

MichaelA, MaxRa

It seems that a key bottleneck for the field of longtermism-aligned AI governance is limited strategic clarity (see Muehlhauser, 2020, 2021). As one effort to increase strategic clarity, in October-November 2022, we sent a survey to 229 people we had reason to believe are knowledgeable about longtermist AI governance, receiving 107 responses. We asked about:

respondents’ “theory of victory” for AI risk (which we defined as the main, high-level “plan” they’d propose for how humanity could plausibly manage the development and deployment of transformative AI such that we get long-lasting good outcomes),
how they’d feel about funding going to each of 53 potential “intermediate goals” for AI governance,^[1]
what other intermediate goals they’d suggest,
how high they believe

... (read 190 more words →)

Replying toLet’s think about slowing down AI

Let’s think about slowing down AI

I found this thread interesting and useful, but I feel a key point has been omitted thus far (from what I've read):

Public, elite, and policymaker beliefs and attitudes related to AI risk aren't just a variable we (members of the EA/longtermist/AI safety communities) have to bear in mind and operate in light of, but instead also a variable we can intervene on.
And so far I'd say we have (often for very good reasons) done significantly less to intervene on that variable than we could've or than we could going forward.
So it seems plausible that actually these people are fairly convincible if exposed to better efforts to really explain the arguments in a

Let’s think about slowing down AI

Personally I haven't thought about how strong the analogy to GoF is, but another thing that feels worth noting is that there may be a bunch of other cases where the analogy is similarly strong and where major government efforts aimed at risk-reduction have occurred. And my rough sense is that that's indeed the case, e.g. some of the examples here.

In general, at least for important questions worth spending time on, it seems very weird to say "You think X will happen, but we should be very confident it won't because in analogous case Y it didn't", without also either (a) checking for other analogous cases or other lines of argument or... (read more)

Replying toLessWrong is providing feedback and proofreading on drafts as a service

LessWrong is providing feedback and proofreading on drafts as a service

Cool!

Two questions:

Is it possible to also get something re-formatted via this service? (E.g., porting a Google Doc with many footnotes and tables to LessWrong or the EA Forum.)
Is it possible to get feedback, proofreading, etc. via this service for things that won't be posts?
- E.g. mildly infohazardous research outputs that will just be shared in the relevant research & policy community but not made public

(Disclaimer: I only skimmed this post, having landed here from Habryka's comment on It could be useful if someone ran a copyediting service. Apologies if these questions are answered already in the post.)

Replying toReward Good Bets That Had Bad Outcomes

Reward Good Bets That Had Bad Outcomes

Thanks for this post! This seems like good advice to me.

I made an Anki card on your three "principles that stand out" so I can retain those ideas. (Mainly for potentially suggesting to people I manage or other people I know - I think I already have roughly the sort of mindset this post encourages, but I think many people don't and that me suggesting these techniques sometimes could be helpful.)

Replying toCommentary on AGI Safety from First Principles

Commentary on AGI Safety from First Principles

It's not sufficient to argue that taking over the world will improve prediction accuracy. You also need to argue that during the training process (in which taking over the world wasn't possible), the agent acquired a set of motivations and skills which will later lead it to take over the world. And I think that depends a lot on the training process.
[...] if during training the agent is asked questions about the internet, but has no ability to edit the internet, then maybe it will have the goal of "predicting the world", but maybe it will have the goal of "understanding the world". The former incentivises control, the latter doesn't.

I agree with... (read more)

Replying toAGI safety from first principles: Conclusion

AGI safety from first principles: Conclusion

Thanks for this series! I found it very useful and clear, and am very likely to recommend it to various people.

Minor comment: I think "latter" and "former" are the wrong way around in the following passage?

By contrast, I think the AI takeover scenarios that this report focuses on have received much more scrutiny - but still, as discussed previously, have big question marks surrounding some of the key premises. However, it’s important to distinguish the question of how likely it is that the second species argument is correct, from the question of how seriously we should take it. Often people with very different perspectives on the latter actually don’t disagree very much

... (read more)

FWIW, I feel that this entry doesn't capture all/most of how I see "meta-level" used.

Here's my attempted description, which I wrote for another purpose. Feel free to draw on it here and/or to suggest ways it could be improved.

Meta-level and object-level = typically, “object-level” means something like “Concerning the actual topic at hand” while “Meta-level” means something like “Concerning how the topic is being tackled/researched/discussed, or concerning more general principles/categories related to this actual topic”
- E.g., “Meta-level: I really appreciate this style of comment; I think you having a policy of making this sort of comment is quite useful in expectation. Object-level: I disagree with your argument because [reasons]”

Announcing the Nuclear Risk Forecasting Tournament

Overview of Rethink Priorities’ work on risks from nuclear weapons

Summary: From June 22, Rethink Priorities and Metaculus will run a Nuclear Risk Forecasting Tournament to help inform funding, policy, research, and career decisions aimed at reducing existential risks. The starting prize pool is $2,500. We would be excited for you to help via making forecasts, participating in the Discussion Forum, increasing forecaster rewards by becoming a monthly Supporter, and/or reaching out to relevant people to suggest they participate.

In 1941, no one had ever built a nuclear weapon, and most people had no idea that anyone ever might. Over the following 80 years, the Allies built such weapons, the US detonated two on cities, and scientists and engineers developed or proposed many... (read 583 more words →)

Notes on effective-altruism-related research, writing, testing fit, learning, and the EA Forum

This post overviews the work that Rethink Priorities has done and is doing on risks from nuclear weapons, as well as the theory of change for this work. This is intended to help readers understand how our posts on this topic fit together and what impact we ultimately intend them to have. Note that this post isn’t a summary of our work on nuclear risk; we plan to produce such summaries later. This is cross-posted from the EA Forum.

In 2019, Rethink Priorities (RP) began investigating risks from nuclear weapons in order to determine:

The extent to which nuclear risk reduction should be prioritized by effective altruists (and particularly long termism)
The most effective ways

... (read 784 more words →)

Notes on "Bioterror and Biowarfare" (2006)

Cross-posted from the EA Forum. I'm not sure how many people on LessWrong will find this useful, but I imagine some will.

I've had calls with >30 people who are interested in things like testing their fit for EA-aligned research careers, writing on the EA Forum, "getting up to speed" on areas of effective altruism, etc. (This is usually during EA conferences.)

I gradually collected a set of links and notes that I felt that many such people would benefit from seeing, then turned that into a Google Doc. Many people told me they found that doc useful, so I'm now (a) sharing it as a public post, and (b) still entertaining the hypothesis... (read 993 more words →)

Notes on Henrich's "The WEIRDest People in the World" (2020)

Cross-posted from the EA Forum

I recently finished reading Malcolm Dando’s 2006 book Bioterror and Biowarfare: A Beginner’s Guide, following Darius Meissner’s recommendation. I’d recommend the book for people who want to learn more about bioterror, biowarfare, and/or chemical weapons, and who don’t yet have extensive knowledge on those topics. (I fit that description, and have ranked the book as the 24th most useful-to-me of the 49 effective-altruism-related books I’ve read since learning about effective altruism (EA).)

Here, I'll:

Summarise my six "main updates" from this book
Share the Anki cards I made for myself when reading the book
- I intend this as a lower-effort alternative to writing notes specifically for public consumption or writing a proper

... (read 1156 more words →)

Notes on Schelling's "Strategy of Conflict" (1960)

Cross-posted to the Effective Altruism Forum (but with a brand new and improved structure)

I recently finished reading Henrich's 2020 book The WEIRDest People in the World: How the West Became Psychologically Peculiar and Particularly Prosperous. I would highly recommend it, along with Henrich's 2015 book The Secret of Our Success; I've roughly ranked them the 8th and 9th most useful-to-me of the 47 effective-altruism-related books I've read since learning about effective altruism (EA).

In this post, I'll:

Summarise my "four main updates" from this book
Share the Anki cards I made for myself when reading the book[1]
- I intend this as a lower-effort alternative to writing notes specifically for public consumption or writing a proper book

... (read 2288 more words →)

Please take a survey on the quality/impact of things I've written

I'm crossposting this from the EA Forum, due to Habryka sort-of suggesting doing so.

The Strategy of Conflict, published in 1960 by Thomas Schelling, is:

A series of closely interrelated essays on game theory, [which] deals with an area in which progress has been least satisfactory - the situations where there is a common interest as well as conflict between adversaries: negotiations, war and threats of war, criminal deterrence, extortion, tacit bargaining. It proposes enlightening similarities between, for instance, maneuvering in limited war and in a traffic jam; deterring the Russians and one's own children; the modern strategy of terror and the ancient institution of hostages. (source)

Here’s a free PDF of the book, and... (read 2251 more words →)

Problems in AI risk that economists could potentially contribute to

List(s) of relevant problems

What can the principal-agent literature tell us about AI risk? (and this comment)
Many of the questions in Technical AGI safety research outside AI
Many of the questions in The Centre for the Governance of AI’s research agenda
Many of the questions in Cooperation, Conflict, and Transformative Artificial Intelligence (a research agenda of the Center on Long-Term Risk)
At least a couple of the questions in 80,000 Hours' Research questions that could have a big social impact, organised by discipline
Longtermist AI policy projects for economists (this doc was originally just made for Risto Uuk's own use, so the ideas shouldn't be taken as high-confidence

... (read more)

If any reading this has read anything I’ve written on LessWrong or the EA Forum, I’d really appreciate you taking this brief, anonymous survey. Your feedback is useful whether your opinion of my work is positive, mixed, lukewarm, meh, or negative.

And remember what mama always said: If you’ve got nothing nice to say, self-selecting out of the sample for that reason will just totally bias Michael’s impact survey.

(If you're interested in more info on why I'm running this survey and some thoughts on whether other people should do similar, I give that here.)

Good and bad ways to think about downside risks

If you’ve read anything I’ve written on the EA Forum or LessWrong, I’d really appreciate you taking this brief, anonymous survey. Your feedback is useful whether your opinion of my work is positive, mixed, lukewarm, meh, or negative. And remember what mama always said: If you’ve got nothing nice to say, self-selecting out of the sample for that reason will just totally bias Michael’s impact survey.

Don’t Panic!^[1]

I plan to use people’s responses as inputs - rather than definitive answers - in my ongoing efforts to plan my career and improve in various ways. And I’ll combine these inputs with a lot of other inputs.

Thus, you shouldn’t feel that this is uncomfortably high-stakes,... (read more)

“Intelligence” vs. other capabilities and resources

Legg and Hutter (2007) collect 71 definitions of intelligence. Many, perhaps especially those from AI researchers, would actually cover a wider set of capabilities or resources than people typically want the term “intelligence” to cover. For example, Legg and Hutter’s own “informal definition” is: “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.” But if you gave me a billion dollars, that would vastly increase my ability to achieve goals in a wide range of environments, even if it doesn’t affect anything we’d typically want to refer to as my “intelligence”.

(Having a billion dollars might lead to increases in my intelligence, if

... (read 552 more words →)

MichaelA, JustinShovelain

This post was written for Convergence Analysis.

Many actions we could take to make the world better might also have negative effects, or might even be negative overall. In other words, altruistic actions often have downside risks. Perhaps, for example, that project you might start, career path you might pursue, or article you might write could lead to information hazards, memetic downside risks, or risks of diverting resources (such as money or attention) from more valuable things.^[1]

So if I’m considering doing something to make the world better, but know it might have downside risks, what should I do? How should I even think about that question?

Should I just stop worrying and go ahead,

... (read 3146 more words →)

Failures in technology forecasting? A reply to Ord and Yudkowsky