suppose [...] a convincing argument that without the [X] the odds of human extinction due to [Y] are at least 10% (with high confidence), and that the [X] can reduce the odds to, say, 1% (again, with high confidence), then I would possibly reevaluate my attitude.

I don't think the hypothetical is true (by a large margin the expected impact is too big), but why only "possibly"? A high confidence intervention to avert a 9% risk of human extinction (for far less than 9% of world GDP) would be ludicrously good by normal standards.

Do you mean that "high confidence" is only conditional on the "convincing" argument, but "convincing" corresponds to relatively low confidence in the arguments itself? What is the hypothetical here?

Second, when I ask myself "what argument/logic/experiment would convince me to take the AGI x-risk seriously enough to personally try to do something about it?", I come up with nothing.

Is your position

1. AGI isn't a a large x-risk
2. It's too hard to do anything about it

or something else?

Just to make an extra step toward MIRI, suppose it had a convincing argument that without the FAI research the odds of human extinction due to UFAI are at least 10% (with high confidence), and that the FAI research can reduce the odds to, say, 1% (again, with high confidence), then I would possibly reevaluate my attitude.

You and I might be on the same page here. How broadly are you defining "FAI research" ?

I don't see how any of the mentioned historical examples can do that. And definitely not any kind of counterfactual history scenarios, those have too low confidence to be taken seriously.

There are potentially promising interventions that are less targeted than the FAI research that MIRI is currently doing (e.g. lobbying for government regulations on AI research).

Can you clarify what sorts of counterfactual history scenarios you have in mind?

From Leverage Research's website:

Lots of organizations have truly ambitious plans. Medtronic, Inc., the largest medical technology company in the world, was founded with a 100-year plan. The founder of Panasonic had an even longer term plan. The city government of Amarillo, TX is preparing a 20-year road map. The city government of North Vancouver has adopted a 100 Year Sustainability Vision for the City.

Miller predicted the return of Jesus in 21 years, not a century. (Added: I don't mean to imply that this makes him uninteresting.)

I would be very interested to learn if anyone has ever predicted the supernatural end of the world on that time scale.

Thanks Nick. I didn't give enough detail as to what I'm looking for in my original post. My initial reactions

CFCs?

Yes, Luke and I talked about this — I forgot to list it.

Asteroid tracking (if they weren't planning on asteroids in the next ten years, seems like natural risks with average return periods > 10 years could be a good source of examples, though I feel this maybe isn't what you are asking for because it is so statistical rather than scenario-based)

I think that I'm looking for things involving more speculation. Also, we don't know what the impact of asteroid tracking effots will be.

Legislation where you don't expect the legislation to have effects for more than 10 years. Some international treaties/conventions might be examples

Here there's a clear historical precedent that people are using to inform their decision making.

Religion examples. In 1833 William Miller predicted the return of Jesus Christ in 1943 and people took specific actions.

How many people?

Space exploration examples? Maybe these things required far advance planning. Voyager may be an example?

I'll brood on this. Intuitively, it doesn't seem relevant, but I have trouble placing my finger on why.

Futures markets? How far do they go out?

I'll investigate.

Simon-Ehrlich wager

I'm looking at this as a part of the Limits to Growth investigation.

Sleeper agents? I have heard stories that some Russian agents were sleepers in the US for greater than 10 years, waiting to strike at the right time if war broke out. May want to check on it.

I'll investigate.

Eugenics people

Did they have any impact? Could they plausibly have?

People conducting very long longitudinal studies

These seem like they fall into the category "general scientific research that could have humanitarian value" rather than being driven by specific predictions and being aimed at influencing specific outcomes.

There are some mundane examples that fit your description but I don't think are what you are asking for. E.g., people working on buildings that take more than 10 years to complete, people buying something for a young daughter and saving it for when she gets married (I guess this used to be a thing), people getting mortgages, people saving for retirement, people studying anatomy when 16 in hopes of becoming a doctor, making a trust fund for your kid.

Here there's a clear historical precedent that people are using to inform their decision making.

Many countries now or in the past have banned human cloning. There are a number of justifications for this, but some of them center around speculative risks.

Y2K mitigation.

Do you have a sense for the size of the threat that Y2K presented?

Doris Lessing's "Report on the Threatened City" (I found it unreadable, so this is not a recommendation) points out that Californians live with the constant threat of a major earthquake. A big enough quake could kill millions, although a quake of that magnitude would be rather infrequent. In general, seismic, volcanic, and weather events are a matter of when, not whether, so perhaps this is not quite in the right reference class.

Building regulations may count, though I think that the historical precedent somewhat frequent large earthquakes in California makes the case disanalogous to the issue of AI risk, which involves an event that has never happened before.

Albania spent billions of dollars on useless bunkers in case of an invasion.

Can you give a reference? Who did they anticipate potential invasion from?

Thanks James

Constantinople managed to keep the secret of Greek Fire.

Roman emperor Constantine the Great picked Constantinople as his new capital because it could be easily defended against barbarians. He did this although (I think) he faced no barbarian threat. This worked out very well.

I think (am not sure) that these examples are too old to have significant relevance to the question of planning for AI. Do you see a connection?

Although Japan had been a very closed society, when the Japanese saw in 1853 how much more advanced the American navy was from its own the Japanese quickly modernized and became one of the few Asian countries to escape western colonization (until after WWII at least).

Maybe I should clarify that we're looking for examples of people pushing for more specific outcomes (preventing climate change, preventing overpopulation, etc.) than improving the economy in general.

Do you know how I can fix it?

Was this based on an anticipation of things happening 10+ years later?

Your arguments would be much more convincing if you showed results from actual code. In engineering fields, including control theory and computer science, papers that contain mathematical arguments but no test data are much more likely to have errors than papers that include test data, and most highly-cited papers include test data. In less polite language, you appear to be doing philosophy instead of science (science requires experimental data, while philosophy does not).

What would you want this code to do? What code (short of a full-functioning AGI) would be at all useful here?

At minimum, I would like to see you try to test your theories by examining the actual performance of real-world computer systems, such as search engines, as they perform tasks analogous to making high-level ethical decisions.

Can you expand on this, possibly with example tasks, because I'm not sure what you are requesting here.

Your examples about predicting the future are only useful if you can identify trends by also considering past predictions that turned out to be inaccurate. The most exciting predictions about the future tend to be wrong, and the biggest advances tend to be unexpected.

This is a trenchant critique, but it ultimately isn't that strong: having trouble predicting should be a reason to if anything be more worried rather than less.

It appears that as technology improves, human lives become better and safer. I expect this trend to continue. I am not convinced that AI is fundamentally different -- in current societies, individuals with greatly differing intellectual capabilities and conflicting goals already coexist, and liberal democracy seems to work well for maintaining order and allowing incremental progress. If current trends continue, I would expect competing AIs to become unimaginably wealthy, while non-enhanced humans enjoy increasing welfare benefits. The failure mode I am most concerned about is a unified government turning evil (in other words, evolution stopping because the entire population becomes one unchanging organism), but it appears that this risk is minimized by existing antitrust laws (which provide a political barrier to a unified government) and by the high likelihood of space colonization occurring before superhuman AI appears (which provides a spatial barrier to a unified government).

This is missing the primary concern of people at MIRI and elsewhere. The concern isn't anything like gradually more and more competing AI coming online that are slightly smarter than baseline humans. The concern is that the first true AGI will self-modify itself to become far smarter and more capable of controlling the environment around it than anything else. In that scenario, issues like anti-trust or economics aren't relevant. It is true that on balance human lives have become better and safer, but that isn't by itself a strong reason to think that trend will continue, especially when considering hypothetical threats such the AGI threat whose actions are fundamentally discontinuous to prior human trends for standards of living.