Due to the fact that moderate and fast takeoffs are more likely than slow ones,
That's a big assumption.
Thus, if a given project is not in the lead, it might start lessening it's safety protocol in favor of speed (not to mention standard cloak and dagger actions, or even militaristic scenarios). Is not good, gets extinction.
Nobody desires extinction, and nobody is better off if extinction comes form their own AI project rather than the AI project of somebody else, hence there is no tragedy of the commons scenario.
People are not going to make an AI capable of causing major disasters without being reasonable sure that they can control it.
Nobody desires extinction, and nobody is better off if extinction comes form their own AI project rather than the AI project of somebody else, hence there is no tragedy of the commons scenario.
Extinction is much more costly to society as a whole than to any individual (especially if we count future unborn people). For example a purely selfish individual might value the cost of extinction the same as their own death (which is on average around $10 million as estimated by how much you have to pay people to compensate for increasing their risk of death). ...
I recently wrote an essay about AI risk, targeted at other academics:
Long-Term and Short-Term Challenges to Ensuring the Safety of AI Systems
I think it might be interesting to some of you, so I am sharing it here. I would appreciate any feedback any of you have, especially from others who do AI / machine learning research.