Thanks again for a super thoughtful response and for the offer to help. How would you prefer to collaborate going forward? I am happy to do so here in LW comments in case perhaps others find it helpful but would also be happy to do so on Reddit ,offline, etc. until we think we have something worth publishing - good point on half baked prepper ideas potentially being net negative.

people who have already identified a scenario that they find compelling

That's a super good point, my and my network's expertise lies more in bio risks (and to the degree these overlap with AI scenarios). Perhaps it is better to start with bio - it also seems easier to tackle as many people seem to think AI is kind of binary (as in either everyone dies or most people survive) and also much harder to prepare for. Then, if that somehow becomes successful, we can later on consider venturing out with advice for prepping for other scenarios (e.g. if AI destroys most democracies and people could need plans for escaping to the last liberal democracies standing or building a new one).

Hi and thanks for your thoughtful reply and especially your hint at potentially being willing to help out.

To your first question on existing guides I am criticizing, I would a bit hand wavily say "all the usual ones". I have gone through quite a few preparedness lists online from the preparedness community not affiliated with national governments ("preppers") and have never seen any hint of why they suggest preparing the way they suggest. I have also looked at preparedness advice by the Scandinavian governments, and I think the some US ones as well and again, have not found any evidence of why they suggest what they suggest. So my first criticism is that it is unclear what is being prepared for. I have a suspicion based on some BOTECs that if one starts with lying out the threat landscape and try to translate the various threat categories into personal risk (chance of a "civilian" dying from various causes), it would lead to a different set of recommendations than those offered by either preppers or governments.

I like your segmentation. At first I was thinking the do not know/care but happen to prepare was highly unlikely. But then after reading about historical disasters one group stood out: Rich people, perhaps with an entrepreneurial and action-oriented attitude, seem to often do well. To be honest I have not given this much thought but perhaps I should. Instead, I have thought "this should probably exist so I want to see if others want me to work on it before putting in potentially a lot of work". I thought I would check here with the rationalist community first, as I thought it might appeal to a subsection of rationalists (does not look like that from the reception of this post), so perhaps I will rewrite this to appeal to preppers - perhaps I should instead target existing preppers with a propensity for logical arguments (not sure if it is logical to prepare - it might be more cost effective in terms of life expectancy to go for a run and/or use money on vegetables!).

Having thought about this only briefly I see another segmentation for potential users of such a guide:

• A rationalist mind-set
• A prepper mind-set

I think the ideal here would be the quadrant that has both, while either convincing non-prepper rationalists, or irrational preppers to become interested in the guide is probably an uphill battle. I have worked for a few months on biosecurity and some people in this "industry" are genuinely worried about themselves and their loved ones to the point where they would be willing to at least make some small investments (whether skills or products) in preparing for a catastrophic pandemic. I was thinking, perhaps naively, that this might happen with certain people in AI too (although AI is probably much harder to prepare for). So I guess their top complaint is that there is really nothing out there to help them prepare for what they think might be one of the most likely way they or their loved ones could die. I am personally also following government advice, and I am a bit surprised that stockpiling does not include even basic pandemic PPE - many governments themselves are stockpiling PPE, so why not encourage citizens to do the same? Some governments even reduced their stockpiles of PPE but purely due to cost saving and not due to perceived lower risks (instead they should have tried to shift the responsibility to the citizens, encouraging them to stockpile this privately).

Really good point on a flow-chart instead of a recipe: If I start compiling a guide I am likely to take up your advice on this!

Again thanks a lot for comments and especially your kind offer to potentially help out! I think I will next try to gauge interest from preppers instead or rationalists by e.g. making a guest blog post on some prepper blog - the response here on LW seems like there might not be much interest in personal preparedness (or I have done a botch job of my first LW post - many LW posts on prepping are quite popular!).

Another argument against nuclear is that it could make nuclear disarmament more challenging. From my quick reading on the topic, the recent construction of UK's Hinkley Point C might have been in part to support a UK nuclear eco-system that is shared with its Trident nuclear submarine fleet. The authors Stirling and Johnstone seem to be well regarded based on a quick look at their research profiles.

On the topic of regulation/safety - it seems wind, solar and nuclear are all super safe when looking at deaths per TWh:

I have heard many times rationalists/effective altruists saying that nuclear is over regulated and when both the authors failed to find evidence for this claim, and OWID does not show this (I think one could instead make the case that fossil fuels are under regulated) I think it is time to bury this seemingly misguided meme.

I just want to second Johannes' point: A lot of money is going into renewables now and more is expected going forward. This is because we managed to get wind and solar below cost for other power types in many geographies. Once you harness the power of capital, it is pretty large-scale - pension funds are pouring money into renewables and they mostly do not need any more subsidies and having worked in wind energy myself I think I would not mind if all renewables R&D funding instead went to nuclear, for the reasons Johannes outlines. I think nuclear is left to die on the roadside in comparison and it would good to have it ready in case a non-carbon and non-nuclear grid proves really hard, or in case we can make fast-ramping nuclear that is cheaper than other ways to fill in when renewables are not producing. We want all the tools in out toolkit heading into the wild and uncertain road ahead that is the energy transition.

I think that to make your above arguments, you would have to engage with literature on whole-system electric grid modeling. That has been done many times and I trust experts when they say that we can create cost-efficient, complete and reliable electricity systems in many geographies using mostly renewables (some say all renewables). Then nuclear, or bio+co2 capture (negative emissions), or hydrogen could fill in for the last bit. 

In short, you would have to go into details behind these models and show why they are wrong in order to say a mostly renewable grid is infeasible. If you are right, it is urgent! Because we are headed in the direction of a ton of renewables and then as we get close to saturation we need to put in place measures to take care of what renewables cannot supply. If that cannot be done we need to quickly make drastic changes!

Here is one example modelling Sweden's future electricity system with and without nuclear showing that nuclear or no nuclear seems to be about the same cost in a cost optimized system, based on total system cost.

Yeah I'm not too confident about it, i did not spend time looking into it. But I think it is >30% likely you can compensate for past over or under estimations.

Apologies, I don't often come to this forum and only now saw your response. Fair point on teaching the wrong lesson. But to BofAs defence they had a really good workshop for us interns on betting under uncertainty. We had to answer various questions like how long is the Nile. The answer has to be a range and we could set it ourselves. We could put 0 - 10^99 on every answer and get all answers correct. Nobody got more than half right (mostly top grade ivy League grads) or so which really made me humble to overconfidence. They really did try to hammer in proper risk adjustment. That said, how traders bonuses were tied to their profit and loss was not ideal i think - i am pretty sure there were no negative bonuses. The bonuses were like the portfolio competition!

One thought that struck me is that sometimes one can make unholy alliances with profit maximizing organizations aggressively pursuing AI. This happens when you find a company that thinks regulation is likely, thinks that they can increase that likelihood and think that they are better positioned than their competition to thrive in a highly regulated environment. My optimism about this comes comes from both climate change work and vehicle emission regulation. Some oil companies want a carbon price because their expensive oil is less carbon intensive than the cheaper oil of the cooperation. Similarly, some auto makers are better at making low emission vehicles.

I think your point on "What are the culture, expectations, norms and habits?" is possibly crucial. While there are others here in the comments with much more experience than me, I am just reading Sapiens by Yuval Noah Harari and I think it is important to keep in mind that a society (the "object" you want to look at governing in different ways) is a fictional creation and hence trust is essential. This means there is large value in doing things the way they have (to some degree) worked in the past (just look at some people's fear of letting in immigrants out of a deep rooted fear of collapse). On a related note, and perhaps I overlooked this in your article, one can look at alternative societies around the world. One example that comes to mind is Auroville in India where I think governance is somewhat creative. Perhaps there are other experimental societies out there who have organized in even more radical ways like using computer science to set up distribution systems that circumvents the need for money and demand and supply (like some sort of Soviet style economy but made more efficient by the use of IT)?