In the last post I asked for predictions on how different factors affect AI X-risk, as of writing this post (March 1st), 26 people had made predictions on the first question (including myself). 

Key Takeaways:

• People have vastly different expectations, and on some questions there were people predicting 1-2% probability and others predicting 98-99% probability. In fact about 50% of all predictions either gave <10% or >90% probability of different X-risk.
  I find this surprising since it seems overconfident to say you are 98-99% sure about something complex where other smart people at LessWrong disagree. 
• This question: “If an AGI is developed on January 1st 2030 with the sole task of being an oracle, and it acts like an oracle during training, what is the probability it will cause an extinction level catastrophe?” got a median prediction of 28% and average prediction of 30% of leading to extinction, which is far below the ~50% probability for any AGI or for a worker AGI. 30% is still a lot, but there might be some value there.
• Money had a meaningful predicted effect on X-risk, but as pointed out by JBlack it was not possible to select less than 1% probability, so the average would have been lower if that was possible. However the median predictions should still be accurate.

Results:

If AGI is developed before 2100, what is the probability it will cause an extinction level catastrophe?

Median prediction: 41%

Average: 45%

Actor who develops AI:

If AGI is developed on January 1st, 2030, what is the probability it will cause an extinction level catastrophe?

Median prediction: 50%

Average: 53%
 

If AGI is developed on January 1st 2030 by either Google, Microsoft (including OpenAI) or Meta, what is the probability it will cause an extinction level catastrophe?

Median prediction: 63%

Average: 56%
 

If AGI is developed on January 1st 2030 by the US Government/military, what is the probability it will cause an extinction level catastrophe?

Median prediction: 60%

Average: 56%

If AGI is developed on January 1st 2030 by researchers at a University with no close ties to Big Tech or any military, what is the probability it will cause an extinction level catastrophe?

Median prediction: 50%

Average: 53%
 

Technology used for developing AI

If AGI is developed on January 1st 2030 using RLHF similar to today, and no other breakthrough innovation, what is the probability it will cause an extinction level catastrophe?

Median prediction: 60%

Average: 52%

If AGI is developed on January 1st 2030 using a new paradigm of ML than what we have today, what is the probability it will cause an extinction level catastrophe?

Median prediction: 50%

Average: 50%
 

Approach

If an AGI is developed on January 1st 2030 with the sole task of being an oracle, and it acts like an oracle during training, what is the probability it will cause an extinction level catastrophe?

Median prediction: 28%

Average: 36%

If an AGI  is developed on January 1st 2030 with the purpose of being a general assistant/worker, what is the probability it will cause an extinction level catastrophe?

Median prediction: 50%

Average: 51%

Money

If today (27th of February) someone donated $10 billion purely based on advice from leading AI alignment researcher, how much would the risk of  an AGI caused extinction level catastrophe decrease? (Ex if the risk goes from 50% to 49%, the risk decreased by 2%)

Median prediction: 3%

Average: 6.6%

If today (27th of February) someone donated $1 trillion purely based on advice from leading AI alignment researcher, how much would the risk of  an AGI caused extinction level catastrophe decrease?

Median prediction: 20%

Average: 19%