The problem of estimating P(doom) is very complicated but it is not impossible. It is not metaphysical like estimating P(we live in a simulation) or P(we are a boltzman brain). P(doom) is a tangible thing based on human technology. Just very very complicated. It requires summing over many different possibilities. So maybe can we do better by factoring the distribution? This would break the problem down into parts which can each be analyzed one at a time.
Suppose there were market or expert based priors for the following probabilities.
P(a single entity would lose control of the AI). This one component has been the main focus of safety research.
P(the international community can prevent the development of ASI)
P(the international community can restrict number of entities that have ASI)
P(doom | ASI is easily available to all)
P(number of entities that have ASI)
P(a single rogue AI could destroy humanity despite good AI's trying to defend humanity)
P(a single human group would use ASI to oppress the rest of the world)
One could build a bayesian network from those priors and estimate P(doom) using one of many methods (e.g. variational inference).
The list would have to go on much longer actually. And the model would be more complicated.
There are blog posts and papers analyzing each of the probabilities already. It is just a matter of putting all this accumulated theory together into a single model. On the On the other hand, one subtle fallacy could result in absurd results. There could be error compounding. The analysis could fail to include all possibilities. It could be very sensitive to hyper-parameters.
Now some would argue that these challenges are why it's so important for an argument to be simple. But I think these challenges just mean we need many people thinking very carefully about every detail of the model.
I wouldn't be surprised if this is already a thing so please let me know if it is. I have tried searching. I'd like a tool like this one for analyzing P(doom): https://projects.fivethirtyeight.com/2024-election-forecast/. Other precedents are this model of transformative AI timeline on less wrong and the use of bayesian networks for forecasting climate change.
The problem of estimating P(doom) is very complicated but it is not impossible. It is not metaphysical like estimating P(we live in a simulation) or P(we are a boltzman brain). P(doom) is a tangible thing based on human technology. Just very very complicated. It requires summing over many different possibilities. So maybe can we do better by factoring the distribution? This would break the problem down into parts which can each be analyzed one at a time.
Suppose there were market or expert based priors for the following probabilities.
P(a single human group would use ASI to oppress the rest of the world)
One could build a bayesian network from those priors and estimate P(doom) using one of many methods (e.g. variational inference).
The list would have to go on much longer actually. And the model would be more complicated.
There are blog posts and papers analyzing each of the probabilities already. It is just a matter of putting all this accumulated theory together into a single model. On the On the other hand, one subtle fallacy could result in absurd results. There could be error compounding. The analysis could fail to include all possibilities. It could be very sensitive to hyper-parameters.
Now some would argue that these challenges are why it's so important for an argument to be simple. But I think these challenges just mean we need many people thinking very carefully about every detail of the model.