In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success. What is this probability estimate based on and how is success defined?
I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.
-
(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)
Most fundamentally, it's based on taking at face value a world in which nobody appears to be doing similar work or care sufficiently to do so. In the world taken at face value, MIRI is the only organization running MIRI's workshops and trying to figure out things like tiling self-modifying agents and getting work started early on what is probably a highly serial time-sensitive task.
Success is defined most obviously as actually constructing an FAI, and it would be very dangerous to have any organizational model in which we were not trying to do this (someone who conceives of themselves as an ethicist whose duty it is to lecture others, and does not intend to solve the problem themselves, is exceedingly unlikely to confront the hardest problems). But of course if our work were picked up elsewhere and reused after MIRI itself died as an organization for whatever reason, or if in any general sense the true history as written in the further future says that MIRI mattered, I should not count my life wasted, nor feel that we had let down MIRI's donors.
Ok, but that doesn't increase the probability to 'medium' from the very low initial probability of MIRI or another organization benefiting from MIRI's work solving the extremely hard problem of Friendly AI before anyone else screws it up.
I've read all your posts in the threads linked by the OP, and if multiplying the high beneficial impact of Friendly AI by the low probability of success isn't allowed, I honestly don't see why I should donate to MIRI.
If this was a regular math problem and it wasn't world-shakingly important, why wouldn't you expect that funding workshops and then researchers would cause progress on it?
Assigning a very low probability to progress rests on a sort of backwards reasoning wherein you expect it to be difficult to do things because they are important. The universe contains no such rule. They're just things.
It's hard to add a significant marginal fractional pull to a rope that many other people are pulling on. But this is not a well-tugged rope!
I'm not assigning a low probability to progress, I'm assigning a low probability to success.
Where FAI research is concerned, progress is only relevant in as much as it increases the probability of success, right?
Unlike a regular math problem, you've only got one shot at getting it right, and you're in a race with other researchers who are working on an easier problem (seed AI, Friendly or not). It doesn't matter if you're 80% of the way there if we all die first.
Edited to add and clarify: Even accounting for the progress I think you're likely to make, the probability of success remains low, and that's what I care about.
Clarifying question: What do you think is MIRI's probability of having been valuable, conditioned on a nice intergalactic future being true?
A non-exhaustive list of some reasons why I strongly disagree with this combination of views:
Paul Christiano, and now you, have started using the phrase "AI control problems". I've gone along with it in my discussions with Paul, but before many people start adopting it maybe we ought to talk about whether it makes sense to frame the problem that way (as opposed to "Friendly AI"). I see a number of problems with it:
Because sometimes the impossible can be done, and I don't know how to estimate the probability of that. What would you have estimated in advance, without knowing the result, was the chance of success for the AI-Box Experiment? How about if I told you that I was going to write the most popular Harry Potter fanfiction in the world and use it to recruit International Mathematical Olympiad medalists? There may be true impossibilities in this world. Eternal life may be one such, if the character of physical law is what is it appears to be, to our sorrow. I do not think that FAI is one of those. So I am going to try. We can work out what the probability of success was after we have succeeded. The chance which is gained is not gained by turning away or by despair, but by continuing to engage with and attack the problem, watching for opportunities and constantly advancing.
If you don't believe me about that aspect of heroic epistemology, feel free not to believe me about not multiplying small probabilities either.
heroic epistemology
Could you give a more precise statement of what this is supposed to entail?
Not easily. Antiantiheroic epistemology might be a better term, i.e., I think that a merely accurate epistemology doesn't have a built-in mechanism which prevents people from thinking they can do things because the outside view says it's nonvirtuous to try to distinguish yourself within reference class blah. Antiantiheroic epistemology doesn't say that it's possible to distinguish yourself within reference class blah so much as it thinks that the whole issue is asking the wrong question and you should mostly be worrying about staying engaged with the object-level problem because this is how you learn more and gain the ability to take opportunities as they arrive. An antiheroic epistemology that throws up some reference class or other saying this is impossible will regard you as trying to distinguish yourself within this reference class, but this is not what the antiantiheroic epistemology is actually about; that's an external indictment of nonvirtuosity arrived at by additional modus ponens to conclusions on which antiantiheroic epistemology sees no reason to expend cognitive effort.
Obviously from my perspective non-antiheroic epistemology cancels out to mere epistemology, simpl...
"Antiantiheroic epistemology might be a better term, i.e., I think that a merely accurate epistemology doesn't have a built-in mechanism which prevents people from thinking they can do things because the outside view says it's nonvirtuous to try to distinguish yourself within reference class blah. "
Taken literally, I can't possibly disagree with this, but it doesn't seem to answer my question, which is "where is the positive evidence that one is not supposed to ignore." I favor combining many different kinds of evidence, including sparse data. And that can and does lead to very high expectations for particular individuals.
For example, several of my college fraternity brothers are now billionaires. Before facebook Mark Zuckerberg was clearly the person with the highest entrepreneurial potential that I knew, based on his intelligence, motivation, ambition, and past achievements in programming, business, and academics. People described him to me as resembling a young Bill Gates. His estimated expected future wealth based on that data if pursuing entrepreneurship, and informed by the data about the relationship of all of the characteristics I could track with it, wa...
You are asking other people for their money and time, when they have other opportunities. To do that they need an estimate of the chance of MIRI succeeding
No they don't; they could be checking relative plausibility of causing an OK outcome without trying to put absolute numbers on a probability estimate, and this is reasonable due to the following circumstances:
The life lesson I've learned is that by the time you really get anywhere, if you get anywhere, you'll have encountered some positive miracles, some negative miracles, your plans will have changed, you'll have found that the parts which took the longest weren't what you thought they would be, and that other things proved to be much easier than expected. Your successes won't come through foreseen avenues, and neither will your failures. But running through it all will be the fundamental realization that everything you accomplished, and all the unforeseen opportunities you took advantage of, were things that you would never have received if you hadn't attacked the hardest part of the problem that you knew about straight-on, without distraction.
How do you estimate probabilities like that? I honestly haven't a clue. Now, w...
If the year was 1960, which would you rather have?
At any given time there are many problems where solutions are very important, but the time isn't yet right to act on them, rather than on the capabilities to act on them, and also to deal with the individually unexpected problems that come along so regularly. Investment-driven and movement-building-driven discount rates are relevant even for existential risk.
GiveWell has grown in influence much faster than the x-risk community while working on global health, and are now in the process of investigating and pivoting towards higher leverage causes, with global catastrophic risk among the top three under consideration.
I have just spent a month in England interacting extensively with the EA movement here (maybe your impressions from the California EA summit differ, I'd be curious to hear). Donors interested in the far future are also considering donations to the following (all of these are from talks with actual people making concrete short-term choices; in addition to donations, people are also considering career choices post-college):
That is a sort of discussion my brain puts in a completely different category. Peter and Carl, please always give me a concrete alternative policy option that (allegedly) depends on a debate, if such is available; my brain is then far less likely to label the conversation "annoying useless meta objections that I want to just get over with as fast as possible".
Can we have a new top-level comment on this?
If you don't believe me about that aspect of heroic epistemology, feel free not to believe me about not multiplying small probabilities either.
Multiplying small probabilities seems fine to me, whereas I really don't get "heroic epistemology".
You seem to be suggesting that "heroic epistemology" and "multiplying small probabilities" both lead to the same conclusion: support MIRI's work on FAI. But this is the case only if working on FAI has no negative consequences. In that case, "small chance of success" plus "multiplying small probabilities" warrants working on FAI, just as "medium probability of success" and "not multiplying small probabilities" does. But since working on FAI does have negative consequences, namely shortening AI timelines and (in the later stages) possibly directly causing the creation an UFAI, just allowing multiplication by small probabilities is not sufficient to warrant working on FAI if the probability of success is low.
I am really worried that you are justifying your current course of action through a novel epistemology of your own invention, which has not been widely vetted (or even widely understood). Most new ideas are wrong, and I think you ought to treat your own new ideas with deeper suspicion.
I'm a reactionary, not an innovator, dammit! Reacting against this newfangled antiheroic 'reference class' claim that says we ought to let the world burn because we don't have enough of a hero license!
"Reference class" to me is just an intuitive way of thinking about updating on certain types of evidence. It seems like you're saying that in some cases we ought to use the inside view, or weigh object-level evidence more heavily, but 1) I don't understand why you are not worried about "inside view" reasoning typically producing overconfidence or why you don't think it's likely to produce overconfidence in this case, and 2) according to my inside view, the probability of a team like the kind you're envisioning solving FAI is low, and a typical MIRI donor or potential donor can't be said to have much of an inside view on this matter, and has to use "reference class" reasoning. So what is your argument here?
I'm also really unconvinced by the claim that this work could reasonably have expected net negative consequences.
Every AGI researcher is unconvinced by that, about their own work.
...but trying to warn people against reverse or negative effects seem
Is there a concrete policy alternative being considered by you?
I'm currently in favor of of the following:
There's many attempted good deeds which have no effect, but complete backfires make the news because they're rare.
What about continuing physics research possibly leading to a physics disaster or new superweapons, biotech research leading to biotech disasters, nanotech research leading to nanotech disasters, WBE research leading to value drift and Malthusian outcomes, computing hardware research leading to deliberate or accidental creation of massive simulated suffering (aside from UFAI)? In addition, I thought you believed that faster economic growth made a good outcome l...
Overconfidence (including factual error about success rates) is pervasive in entrepreneurs, both the failures and successes (and the successes often fail on a second try, although they have somewhat better odds). The motivating power of overconfidence doesn't mean the overconfidence is factually correct or that anyone else should believe it. And the mega-successes tended to look good in expected value, value of information, and the availability of good intermediate outcomes short of mega-success: there were non-crazy overconfident reasons to pursue them. The retreat to "heroic epistemology" rather than reasons is a bad sign relative to those successes, and in any case most of those invoking heroic epistemology style reasoning don't achieve heroic feats.
Applying the outside view startup statistics, including data on entrepreneur characteristics like experience and success rates of repeat entrepreneurs is not magic or prohibitively difficult. Add in the judgments of top VCs to your model.
For individuals or area/firm experts, one can add in hard-to-honestly-signal data (watching out for overconfidence in various ways, using coworkers, etc). That model would have assigned a m...
Eliezer emailed me to ask me about it (per Carl's request, above); I emailed him back with the email below, which Eliezer requested I paste into the LW thread. Pasting:
...In the majority of cases, people do not straightforwardly say "X is false, but I need to believe X anyhow". More often they wiggle, and polite conversation drops the subject.
You have made statements that I and at least some others interpreted as perhaps indicating such wiggles (i.e., as perhaps indicating a desire to hold onto false impressions by diverting conscious attention or discussion from a particular subject). You have never, to my knowledge, uttered a straight-forward endorsement of holding false beliefs. The wiggle-suggesting statements were not super-clear, and were not beyond the realm of possible misinterpretation.
Re: statements that seemed to me and some others to indicate possible wiggles: You have mentioned multiple times that, well, I forget, but something like, it'd be hard to do top focused research on FAI-like problems while estimating a 1% chance of success. You've also avoided giving probability estimates in a lot of contexts, and have sometimes discouraged conversations in whic
From my internal perspective, the truth-as-I-experience-it is that I'm annoyed when people raise the topic because it's all wasted motion, the question sets up a trap that forces you into appearing arrogant, and I honestly think that "Screw all this, I'm just going to go ahead and do it and you can debate afterward what the probabilities were" is a perfectly reasonable response.
From the perspective of folks choosing between supporting multiple lines of AI risk reduction effort, of which MIRI is only one, such probability estimates are not wasted effort.
Though your point about appearing arrogant is well taken. It's unfortunate that it isn't socially okay to publicly estimate a high probability of success, or to publicly claim one's own exceptionalism, when ones impressions point that way. It places a barrier toward honest conversation here.
From my internal perspective, the truth-as-I-experience-it is that I'm annoyed when people raise the topic [of MIRI's success-odds] because [good reason].
I suspect this annoyance is easily misinterpreted, independent of its actual cause. Most humans respond with annoyance when their plans are criticized. Also, in situations where A has power over B, and where B then shares concerns or criticisms about A's plans, and where A responds with annoyance or with avoidance of such conversation... B is apt to respond (as I did) by being a bit hesitant to bring the topic up, and by also wondering if A is being defensive.
I'm not saying I was correct here. I'm also not sure what the fix is. But it might be worth setting a 1-minute timer and brainstorming or something.
I'm familiar with this move. But you make it before failing too
Sure, you try, sometimes you lose, sometimes you win. On anti-heroic epistemology (non-virtuous to attempt to discriminate within an outside view) there shouldn't be any impossible successes by anyone you know personally after you met them. They should only happen to other people selected post-facto by the media, or to people who you met because of their previous success.
I don't buy the framing. The update would be mainly about you and the problem in question, not the applicability of statistics to reality.
We disagree about how to use statistics in order to get really actually correct answers. Having such a low estimate of my rationality that you think that I know what correct statistics are, and am refusing to use them, is not good news from an Aumann perspective and fails the ideological Turing Test. In any case, surely if my predictions are correct you should update your belief about good frameworks (see the reasoning used in the Pascal's Muggle post) - to do otherwise and go on insisting that your framework was nonetheless correct would be oblivious.
...Two developments in AI as big as Pearl's causal networ
On anti-heroic epistemology (non-virtuous to attempt to discriminate within an outside view) there shouldn't be any impossible successes by anyone you know personally after you met them.
I don't understand why you say this. Given Carl's IQ and social circle (didn't he used to work for a hedge fund run by Peter Thiel?) why would it be very surprising that someone he personally knows achieves your current level of success after he meets them?
They should only happen to other people selected post-facto by the media, or to people who you met because of their previous success.
Carl referenced "Staring Into the Singularity" as an early indicator of your extraordinary verbal abilities (which explains much if not all of your subsequent successes). It suggests that's how you initially attracted his attention. The same is certainly true for me. I distinctly recall saying to myself "I should definitely keep track of this guy" when I read that, back in the extropian days. Is that enough for you to count as "people who you met because of their previous success"?
In any case, almost everyone who meets you now would count you as such. What arguments can you give...
I'm concerned about the probability of having some technical people get together and solve some incredibly deep research problems before some perhaps-slightly-less-technical people plough ahead and get practical results without the benefit of that research. I'm skeptical that we'll see FAI before UFAI for the same reason I'm skeptical that we'll see a Navier-Stokes existence proof before a macroscale DNS solution, I'm skeptical that we'll prove P!=NP or even find a provably secure encryption scheme before making the world's economy dependent on unproven schemes, etc.
Even some of the important subgoals of FAI, being worked on with far more resources than MIRI has yet, are barely showing on the radar. IIRC someone only recently produced a provably correct C compiler (and in the process exposed a bunch of bugs in the industry standard compilers) - wouldn't we feel foolish if a provably FAI human-readable code turned UF simply because a bug was automatically introduced in the compilation? Or if a cosmic ray or slightly-out-of-tolerance manufacturing defect affected one of the processors? Fault-tolerant MPI is still leading-edge research, because although we've never needed it befor...
I agree with Eliezer that the main difficulty is in getting top-quality, relatively rational people to spend hundreds of hours being educated, working through the arguments, etc.
Jaan has done a surprising amount of that and also read most or all of the Sequences. Thiel has not yet decided to put in that kind of time.
Here's a list of people I'd want on that committee if they were willing to put in hundreds of hours catching up and working through the arguments with us: Scott Aaronson, Peter Norvig, Stuart Russell, Michael Nielsen.
I'd probably be able to add lots more names to that list if I could afford to spend more time becoming familiar with the epistemic standards and philosophical sophistication of more high-status CS people. I would trust Carl Shulman, Paul Christiano, Jacob Steinhardt, and a short list of others to add to my list with relatively little personal double-checking from me.
But yeah; the main problem seems to me that I don't know how to get 400 hours of Andrew Ng's time.
Although with Ng in particular it might not take 400 hours. When Louie and I met with him in Nov. '12 he seemed to think AI was almost certainly a century or more away, but by May '13 (after gettin...
People differ in their estimates within MIRI. Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.
For myself, I think the cause of AI risk reduction, in total and over time, has a worthwhile small-to-medium probability of making an astronomical difference on our civilization's future (and a high probability that the future will be very powerfully shaped by artificial intelligence in a way that can be affected by initial conditions). But the impact of MIRI in particular has to be a far smaller subset of the expected impact of the cause as a whole, in light of its limited scale and capabilities relative to the relevant universes (total AI research, governments, etc), the probability that AI is not close enough for MIRI to be very relevant, the probability that MIRI's approach turns out irrelevant, uncertainty over the sign of effects due to contributions to AI progress, future AI risk efforts/replaceability, and various other drag factors.
ETA: To be clear, I think that MIRI's existence, relative to the counterfactual in which it never existed, has been a good thing and reduced x-risk in my opinion, despi...
I'm eager to see Eliezer's planned reply to your "ETA2", but in the meantime, here are a few of my own thoughts on this...
My guess is that movement-building and learning are still the best things to do right now for AI risk reduction. CEA, CFAR, and GiveWell are doing good movement-building, though the GiveWell crowd tends to be less interested in x-risk mitigation. GiveWell is doing a large share of the EA-learning, and might eventually (via GiveWell labs) do some of the x-risk learning (right now GiveWell has a lot of catching up to do on x-risk).
The largest share of the "explicit" x-risk learning is happening at or near FHI & MIRI, including e.g. Christiano. Lots of "implicit" x-risk learning is happening at e.g. NASA where it's not clear that EA-sourced funding can have much marginal effect relative to the effect it could have on tiny organizations like MIRI and FHI.
My impression, which could be wrong, is that GiveWell's ability to hire more researchers is not funding-limited but rather limited by management's preference to offer lower salaries than necessary to ensure cause loyalty. (I would prefer GiveWell raise salaries and grow its research...
I am thinking of writing a discussion thread to propose MIRI make it a priority to create a (video/pamphlet/blog post), tailored to intelligent non-rationalists and with as little jargon as possible (e.g. no terms like Kolmogorov complexity), to explain the dangers of UFAI. Please upvote this comment if you think LW is better with such a post, because I have zero karma.
I give them about 10 percent probability, and statements I've read from Eliezer and Luke cause me to believe they're below medium in self-assessment nowadays, or their definition of "medium" may be lower than you think. I support them to increase the chance.
That said, low chances of success aren't good for recruiting or motivation, and maximizing the probability requires as much effort and skill as possible to obtain.
Eliezer is very much against the idea of supporting MIRI based on a "low probability of really high impact" argument.
I hate to put words in his mouth, but I think
1.) What do you base the 10% estimate on?
I base it on everything I've read and seen on technology, human nature, historical uses of power, trends in tech capabilities, the effects of intelligence, MIRI's mission, team, and focus, and the greater realm of philanthropic endeavors.
If you mean, 'you pulled that number out of your butt, and therefore I call you on it,' then I'll have to admit defeat due to inability to break it down quantitatively. Sorry.
2.) Eliezer is very much against the idea of supporting MIRI based on a "low probability of really high impact" argument. What do you think?
I think that's taken out of context. The way I understand it, he means superintelligence will have a really high impact regardless (near 100% probability), and is therefore a 'lever point' which can have a higher probability of being impacted by anyone paying attention to it, and since MIRI is one of very few groups paying attention, they have a medium probability of being such an impactor.
How does MIRI Know it Has a Medium Probability of Success?
it doesn't know of course. The uncertainty in any probability estimate is bound to be huge, which is reflected in the variety of estimates and opinions event within MIRI. I hope to see some day a thoughtfully constructed reference class with a large enough number of members to be worth drawing conclusions from.
"Antiantiheroic epistemology might be a better term, i.e., I think that a merely accurate epistemology doesn't have a built-in mechanism which prevents people from thinking they can do things because the outside view says it's nonvirtuous to try to distinguish yourself within reference class blah. "
Taken literally, I can't possibly disagree with this, but it doesn't seem to answer my question, which is "where is the positive evidence that one is not supposed to ignore." I favor combining many different kinds of evidence, including sparse data. And that can and does lead to very high expectations for particular individuals.
For example, several of my college fraternity brothers are now billionaires. Before facebook Mark Zuckerberg was clearly the person with the highest entrepreneurial potential that I knew, based on his intelligence, motivation, ambition, and past achievements in programming, business, and academics. People described him to me as resembling a young Bill Gates. His estimated expected future wealth based on that data if pursuing entrepreneurship, and informed by the data about the relationship of all of the characteristics I could track with it, was in the 9-figure range. Then add in that facebook was a very promising startup (I did some market sizing estimates for it, and people who looked at it and its early results were reliably impressed).
Moving from entrepreneurship to politics, one can predict success to a remarkable degree with evidence like "Eton graduate, Oxford PPE graduate with first class degree, Oxford Union leader, interested in politics, starts in an entry-level political adviser job with a party." See this post or this paper. Most of the distance in log odds to reliably becoming Prime Minister, let alone Member of Cabinet or Parliament, can be crossed with objective indicators. Throw in a bit more data about early progress, media mentions, and the like and the prediction improves still more.
I would then throw in other evidence, like the impressiveness of the person's public speaking relative to other similar people, their number and influence of friends and contacts in high places relative to other similar types (indicating both social capital, and skill at getting more), which could improve or worsen the picture. There is still a sizable chunk of randomness in log terms, as political careers are buffeted by switches in party control, the economy, the rise and fall of factions that carry members with them, and other hard-to-control factors at many stages. So I can and do come to expect that someone will probably get federal political office, and have a good shot at Cabinet, and less so for PM. But within the real distribution of characteristics I won't be convinced that a young person will probably become PM, which would require almost zero noise.
In science I can be convinced a young star is a good prospect for Nobel or Fields medal caliber work. But I would need stronger evidence than we have seen for anyone to expect that they would do this 10 times (since no one has done so). I am sympathetic to Wei Dai's comment
I would be quite surprised to see you reliably making personal mathematical contributions at the level of the best top math and AI people. I would not be surprised to see MIRI workshop participants making progress on the problems at a level consistent with the prior evidence of their ability, and somewhat higher per unit time because workshops harvest ideas generated over a longer period, are solely dedicated to research, have a lot of collaboration and cross-fertilization, and may benefit from improved motivation and some nice hacking of associated productivity variables. And I would not be surprised at a somewhat higher than typical rate of interesting (to me, etc) results because of looking at strange problems.
I would be surprised if the strange problems systematically deliver relatively huge gains on actual AI problems (and this research line is supposed to deliver AGI as a subset of FAI before others get AGI so it must have great utility in AGI design), i.e. if the strange problems are super-promising by the criteria that Pearl or Hinton or Ng or Norvig are using but neglected by blunder. I would be surprised if the distance to AGI is crossable in 20 years.
You are asking other people for their money and time, when they have other opportunities. To do that they need an estimate of the chance of MIRI succeeding, considering things like AI timelines, the speed of takeoff given powerful AI, competence of other institutions, the usefulness of MIRI's research track, the feasibility of all alternative solutions to AI risk/AI control problems, how much MIRI-type research will be duplicated by researchers interested for other reasons over what timescales, and many other factors including the ability to execute given the difficulty of the problems and likelihood of relevance. So they need adequate object-level arguments about those contributing factors, or some extraordinary evidence to trust your estimates of all of them over the estimates of others without a clear object-level case. Some of the other opportunities available to them that they need to compare against MIRI:
I read every blog post they put out.
I figure I can use my retirement savings for this.
I thought it came from them being collectively foolish or ig... (read more)