Ben Pace

"Often I compare my own Fermi estimates with those of other people, and that’s sort of cool, but what’s way more interesting is when they share what variables and models they used to get to the estimate." – Oliver Habryka, at a model building workshop at FHI in 2016 One question that people in the AI x-risk community often ask is > "By what year do you assign a 50% probability of human-level AGI?" We go back and forth with statements like "Well, I think you're not updating enough on AlphaGo Zero." "But did you know that person X has 50% in 30 years? You should weigh that heavily in your calculations." However, 'timelines' is not the interesting question. The interesting parts are in the causal models behind the estimates. Some possibilities: * Do you have a story about how the brain in fact implements back-propagation, and thus whether current ML techniques have all the key insights? * Do you have a story about the reference class of human brains and monkey brains and evolution, that gives a forecast for how hard intelligence is and as such whether it’s achievable this century? * Do you have a story about the amount of resources flowing into the problem, that uses factors like 'Number of PhDs in ML handed out each year' and 'Amount of GPU available to the average PhD'? Timelines is an area where many people discuss one variable all the time, where in fact the interesting disagreement is much deeper. Regardless of whether our 50% dates are close, when you and I have different models we will often recommend contradictory strategies for reducing x-risk. For example, Eliezer Yudkowsky, Robin Hanson, and Nick Bostrom all have different timelines, but their models tell such different stories about what’s happening in the world that focusing on timelines instead of the broad differences in their overall pictures is a red herring. (If in fact two very different models converge in many places, this is indeed evidence of t