I think the difficulty with answering this question is that many of the disagreements boil down to differences in estimates for how long it will take to operationalize lab-grade capabilities. Say we have intelligences that are narrowly human / superhuman on every task you can think of (which, for what it’s worth, I think will happen within 5-10 years). How long before we have self-replicating factories? Until foom? Until things are dangerously out of our control? Until GDP doubles within one year? In what order do these things happen? Etc. etc.

If I got anything out of the thousands of words of debate on the site in the last couple of months, it’s the answers to these questions that folks seem to disagree about (though I think I only actually have a good sense of Paul’s answers to these). Also curious to see more specific answers / timelines.

GovAI is working on a survey of ML experts about timelines (among other things). It looks like this is supposed to replicate the influential Grace et al. (2017) survey.

They are asking various questions about human-level AI forecasts and predicted AI progress milestones. You can see the questions and other info about it here: https://osf.io/b239h/

It looks the survey may have actually been conducted back in 2019 but the results aren't published yet. (Update: The results are published now.) So it's moving slowly but once out may be a good formal resource on timelines to complement a lot of the informal conversations scattered about on the forums. 

I'm not sure the status of this but would be interested to know if someone has a chance to reach out to Baobao Zhang who I believe is leading the effort.

The shortest ones I've heard have been 7 years median estimate. 

For an intelligence which is capable of continual addition of new domains of knowledge, generating its own schema, and making accurate predictions out-of-distribution? 5 to 10 years.

Why: AlphaGo was only six years ago. The global effort per year toward development is still increasing rapidly; and chips are not a bottleneck. We know for sure that human brains, with our 100T synapses, do not need to be replicated in every detail, before comparable performance is reached. As proof: Google's RETRO language-model out-performs the average human, while using only one 13,000th of a human brain's synapses. Our Broca's Area alone is many times larger; our brains are clearly bloat-ware.

So, AI performance will not be hindered by hardware... the limitation is its capacity to learn from few examples, and generalize. Looking at how image recognition once required 10 Million examples to learn a cat, and comparing that to the ten photos needed to construct a high-res 3D render, now, I don't see 'few examples' being a problem. Generalization is more difficult, because we must expand the search of architectures to include the more-neglected and less-performant models, like Numenta's. I still don't expect that stumbling-block to take long, and once a model can generalize out-of-distribution from conceptualizations even a little bit, then we have a matter of months, a year at best, before it's overwhelmingly good.

When FOOM? We're late to that party: it's already happening. If we had told any futurist in 2010 that we were getting 10x, 25x, and more, regularly, across numerous performance metrics, for various applications... they would have called that pace a FOOM. It just doesn't need artificial general intelligence to happen... human + machine, and AutoML seem to be plenty for self-acceleration. By the time AGI is entrusted with anything, we're likely to have only a few multiples of algorithmic performance remaining! Most of the FOOM, on log-chart, will happen before AGI - and that's a very specific prediction.

My research in trying to write a novel about this subject seems to indicate the experts think that chip speeds and densities are improving so slowly there is no chance of building an AI smart enough to begin improving itself for at least fifteen to twenty years if even then. The only way it might happen sooner is if there was an emergent worldwide effort composed of many separate researchers focusing on different sub-problems of the same highly focused goal, like trying to invent the simplest possible self-replicating nanobot or smallest computing element.