Introduction Polarisation hampers cooperation and progress towards understanding whether future AI poses an existential risk to humanity and how to reduce the risks of catastrophic outcomes. It is exceptionally challenging to pin down what these risks are and what decisions are best. We believe that a model-based approach offers many...
[This article is copy-pasted from the Lumina blog, very lightly edited for LessWrong.] Where is everybody? — Enrico Fermi The omnipresence of uncertainty is part of why making predictions and decisions is so hard. We at Lumina advocate treating uncertainty explicitly in our models using probability distributions. Sadly this is...
Polarisation hampers cooperation and progress towards understanding whether future AI poses an existential risk to humanity and how to reduce the risks of catastrophic outcomes. It is exceptionally challenging to pin down what these risks are and what decisions are best. We believe that a model-based approach offers many advantages for improving our understanding of risks from AI, estimating the value of mitigation policies, and fostering communication between people on different sides of AI risk arguments. We also believe that a large percentage of practitioners in the AI safety and alignment communities have appropriate skill sets to successfully use model-based approaches.
In this article, we will lead you through an example application of a... (read 9319 more words →)
[This article is copy-pasted from the Lumina blog, very lightly edited for LessWrong.]
Where is everybody?
— Enrico Fermi
The omnipresence of uncertainty is part of why making predictions and decisions is so hard. We at Lumina advocate treating uncertainty explicitly in our models using probability distributions. Sadly this is not yet as common as it should be. A recent paper “Dissolving the Fermi Paradox” (2018) is a powerful illustration of how including uncertainty can transform conclusions on the fascinating question of whether our Earth is the only place in the Universe harboring intelligent life. The authors, Anders Sandberg, Eric Drexler and Toby Ord (whom we shall refer to as SDO), show elegantly that the apparent... (read 4425 more words →)
... (read more)>APS is less understood and poorly forecasted compared to AGI.
I disagree with this. I have come to dislike the term AGI because (a) its meaning is so poorly defined, (b) the concept most people have in mind will never be achieved, nor needs to be to get to the capability level that is necessary for catastrophic scenarios, and (c) the concept of "AGI" doesn't get at the part of advanced AI that is relevant for reasoning about x-risk.
In this diagram, the AGI circle captures the concept that AGI is a system that subsumes all human capabilities. Realistically, this will never exactly happen. Is completing that AGI circle really the relevant concept? If
While writing this posting, Max and I had several discussions about anthropic bias. It left me pretty uncomfortable with the application of it here as well, although I often took the position of defending it during our debates. I strongly relate to your use of the word "mysterious".
A prior that "we are not exceptionally special" seems to work pretty good across lots of beliefs that have occurred throughout history. I feel like that prior works really well but is at odds with the anthropic bias argument.
I'm still haven't resolved whether the anthropic argument is valid here in my own mind. But I share Ben's discomfort.
The Drake parameter R* = The rate of star formation (new stars / year). It is set to LogUniform(1,100), meant to be representative of the Milky Way. I can easily replace that in the model with 2000*LogUniform(1,100) to explore your question. The other Drake parameter that might need some thought is f_c = The fraction of intelligent civilizations that are detectable / contactable. For now, let's not alter this one. The other Drake parameters shouldn't really change, at least assuming they are similar galaxies.
With that change to R*, P(N<1) -- the probability there isn't another in the Virgo cluster, becomes 81% (for the 2nd model version with the t V λ... (read more)
At the moment, I am particularly interested in the structure of proposed x-risk models (more than the specific conclusions). Lately, there has been a lot of attention on Carlsmith-style decompositions, which have the form "Catastrophe occurs if this conjunction of events occur". I found it interesting that this post took the upside-down version of that, i.e., "Catastrophe is inevitable unless (one of) these things happen".
Why do I find this distinction relevant? Consider how non-informed our assessments for most of these factors in these models actually are. Once you include this meta-uncertainty, Jensen's inequality implies that the mean & median risk of catastrophe decreases with greater meta-uncertainty, whereas when you turn the argument... (read more)