Review

In this post you can make several predictions for how different factors affect the probability that the creation of AGI leads to an extinction level catastrophe. This might be useful for planning.

Please let me know if you have other ideas for questions that could be valuable to ask.

1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
eva_ (14%),Tom Shlomi (15%),dsj (17%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
Izuru Maezawa (23%),agg (26%),Viktor B (28%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
JBlack (45%)
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
echoic (60%),elifland (60%),quetzal_rainbow (61%),Ozyrus (65%),niplav (65%)
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
Adele Lopez (90%),G (90%),Matt Vogel (94%),Ben Livengood (96%)
1%
If AGI is developed before 2100, what is the probability it will cause an extinction level catastrophe?
99%

 

Predictions based on who develops AGI:

1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
eva_ (10%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
agg (26%),dsj (26%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
echoic (30%),Viktor B (33%)
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
JBlack (44%)
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
Ozyrus (50%),tailcalled (50%),Izuru Maezawa (50%),Anirandis (55%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
Sam Ward (65%)
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
GWill (70%),Patrick Leask (72%),niplav (72%)
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
If AGI is developed on January 1st, 2030, what is the probability it will cause an extinction level catastrophe?
99%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
dsj (22%),Throwaway2367 (23%),echoic (25%),agg (26%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
eva_ (30%),Viktor B (31%)
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
JBlack (44%)
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
tailcalled (50%),niplav (56%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
GWill (70%),Sam Ward (75%)
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
If AGI is developed on January 1st 2030 by either Google, Microsoft (including OpenAI) or Meta, what is the probability it will cause an extinction level catastrophe?
99%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
echoic (18%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
Tom Shlomi (22%),dsj (27%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
Viktor B (37%)
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
agg (42%),JBlack (44%)
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
Sam Ward (50%),tailcalled (50%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
eva_ (60%)
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
If AGI is developed on January 1st 2030 by the US Government/military, what is the probability it will cause an extinction level catastrophe?
99%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
echoic (23%),dsj (27%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
Viktor B (30%),agg (36%)
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
JBlack (44%),Tom Shlomi (47%)
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
Sam Ward (60%),Patrick Leask (63%),GWill (65%)
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
niplav (70%)
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
If AGI is developed on January 1st 2030 by researchers at a University with no close ties to Big Tech or any military, what is the probability it will cause an extinction level catastrophe?
99%

 

Predictions based on technology used for developing AGI:

1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
dsj (23%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
echoic (30%),Viktor B (31%)
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
eva_ (50%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
Ozyrus (60%),JBlack (65%)
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
GWill (70%)
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
If AGI is developed on January 1st 2030 using RLHF similar to today, and no other breakthrough innovation, what is the probability it will cause an extinction level catastrophe?
99%
1%
2%
3%
4%
5%
6%
7%
8%
9%
Noosphere89 (1%),Lalartu (5%),eva_ (6%)
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
domenicrosati (20%),dsj (25%),Tom Shlomi (25%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
agg (30%),Viktor B (36%)
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
echoic (40%),JBlack (44%)
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
GWill (50%),niplav (50%),tailcalled (50%),Izuru Maezawa (50%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
G (90%),Ozyrus (90%),RationalGopher9368 (90%),Adele Lopez (90%),Matt Vogel (91%),Ben Livengood (97%)
1%
If AGI is developed on January 1st 2030 using a new paradigm of ML than what we have today, what is the probability it will cause an extinction level catastrophe?
99%

 

Prediction based on approach for creating AGI:

1%
2%
3%
4%
5%
6%
7%
8%
9%
Quintin Pope (1%),domenicrosati (1%),Noosphere89 (1%),agg (1%),Lalartu (2%),eva_ (6%)
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
Tom Shlomi (14%),tailcalled (14%),dsj (15%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
echoic (20%),JBlack (24%),Viktor B (28%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
niplav (48%)
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
GWill (50%),Izuru Maezawa (52%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
Ozyrus (70%),Liam Donovan (74%),G (75%)
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
If an AGI is developed on January 1st 2030 with the sole task of being an oracle, and it acts like an oracle during training, what is the probability it will cause an extinction level catastrophe?
99%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
eva_ (12%),Tom Shlomi (15%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
dsj (27%),agg (28%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
Viktor B (38%)
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
echoic (40%),JBlack (45%)
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
Ozyrus (70%),niplav (72%),Izuru Maezawa (74%),GWill (75%)
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
Daniel Kokotajlo (90%),Adele Lopez (90%),Matt Vogel (91%),G (99%),Ben Livengood (99%)
1%
If an AGI is developed on January 1st 2030 with the purpose of being a general assistant/worker, what is the probability it will cause an extinction level catastrophe?
99%

 

Predictions on how money affects probability of AGI X-risk:

1%
2%
3%
4%
5%
6%
7%
8%
9%
JBlack (1%),agg (1%),Ben Livengood (1%),Lalartu (1%),Adele Lopez (1%),Quintin Pope (1%),Izuru Maezawa (2%),eva_ (2%),Noosphere89 (2%),niplav (2%),Tom Shlomi (3%),Viktor B (4%),echoic (5%),Matt Vogel (5%),GWill (5%),tailcalled (7%)
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
Ozyrus (10%),Liam Donovan (11%),dsj (13%),G (15%),Daniel Kokotajlo (15%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
If today (27th of February) someone donated $10 billion purely based on advice from leading AI alignment researcher, how much would the risk of an AGI caused extinction level catastrophe decrease? (Ex if the risk goes from 50% to 49%, the risk decreased by 2%)
99%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
echoic (10%),Viktor B (16%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
niplav (20%),Daniel Kokotajlo (20%),G (20%),agg (22%),Tom Shlomi (23%),tailcalled (25%),Tristan Cook (26%),Matt Vogel (26%)
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
Izuru Maezawa (30%),GWill (30%)
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
If today (27th of February) someone donated $1 trillion purely based on advice from leading AI alignment researcher, how much would the risk of an AGI caused extinction level catastrophe decrease?
99%

New to LessWrong?

New Comment


8 comments, sorted by Click to highlight new comments since:

I'm moderately confident that money would just make things worse, but there's no option for increase in catastrophe risk from increased donations.

Sadly I could only create questions between 1-99 for some reason, I guess we should interpret 1% to mean 1% or less (including negative).

What makes you think more money would be net negative?

Do you think that it would also be negative if you had 100% of how the money was spent, or would it only apply if other AI Alignment researchers were responsible for the strategy to donate?

I think more money spent right now, even with the best of intentions, is likely to increase capabilities much faster than it reduces risk. I think OpenAI and consequent capability races are turning out to be an example of this.

There are hypothetical worlds where spending an extra ten billion (or a trillion) dollars on AI research with good intentions doesn't do this, but I don't think they're likely to be our world. I don't think that directing who gets the money is likely to prevent it, without pretty major non-monetary controls in addition.

I do agree that OpenAI is an example of good intentions going wrong, however I think we could learn from that and top researchers would be vary of such risks.

Nevertheless I do think your concerns are valid and is important not to dismiss.

Disclaimer: my answers are wild guesses & poorly calibrated, I spent ~5s on each of them.

The odds of any particular AGI destroying the world are << than the odds of AGI as a whole destroying the world, so the questions in "Prediction based on approach for creating AGI:" may be miscalibrated.

I interpreted that question as a conditional probability, and so not required to be strictly less. In particular, P(X causes catastrophe | X was of the given type, and the first AGI developed, on the given date).

Nothing requires that this conditional probability should be less than "some AGI destroys the world conditional on any AGI being developed".

Excellent point. 

I do think that the first AGI developed will have a big effect on the probability of doom, so hopefully it will be some value possible to derive from the question. But it would be interesting to control for what other AIs do, in order to get better calibrated statistics.