LESSWRONG
LW

Comment Permalink

People differ in their estimates within MIRI. Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.

For myself, I think the cause of AI risk reduction, in total and over time, has a worthwhile small-to-medium probability of making an astronomical difference on our civilization's future (and a high probability that the future will be very powerfully shaped by artificial intelligence in a way that can be affected by initial conditions). But the impact of MIRI in particular has to be a far smaller subset of the expected impact of the cause as a whole, in light of its limited scale and capabilities relative to the relevant universes (total AI research, governments, etc), the probability that AI is not close enough for MIRI to be very relevant, the probability that MIRI's approach turns out irrelevant, uncertainty over the sign of effects due to contributions to AI progress, future AI risk efforts/replaceability, and various other drag factors.

ETA: To be clear, I think that MIRI's existence, relative to the counterfactual in which it never existed, has been a good thing and reduced x-risk in my opinion, despite not averting a "medium probability," e.g. 10%, of x-risk.

ETA2: Probabilities matter because there are alternative uses of donations and human capital.

I have just spent a month in England interacting extensively with the EA movement here. Donors concerned with impact on the long-run future are considering donations to all of the following (all of these are from talks with actual people making concrete short-term choices; in addition to donations, people are also considering career choices post-university):

80,000 Hours, the Center for Effective Altruism and other organizations that are helping altruists to improve their careers, coordination, information, and do movement building; some specifically mention the Center For Applied Rationality; these organizations also improve non-charity options, e.g. 80k helping people going into scientific funding agencies and political careers where they will be in a position to affect research and policy reactions to technologies relevant to x-risk and other trajectory changes
AMF/GiveWell's other recommended charities to keep GiveWell and the EA movement growing (GiveWell's growth in particular has been meteoric, with less extreme but still rapid growth in other EA institutions such as Giving What We can and CEA), while actors like GiveWell Labs, Paul Christiano, and Nick Beckstead and others at FHI, investigate the intervention options and cause prioritization, followed by organization-by-organization analysis of the GiveWell variety, laying the groundwork for massive support for the interventions and organizations identified by such processes as most effective in terms of their far future impact
Finding ways to fund such evaluation with RFMF, e.g. by paying for FHI or CEA hires to work on them
The FHI's other work
A donor-advised fund investing the returns until such evaluations or more promising opportunities present themselves or are elicited by the fund, including both known options for which no organization with RFMF or adequate quality exists, and unknown future options; some possible applications include, e.g. convening panels of independent scientific experts to evaluate key technical claims about future technologies, extensions of the DAGGRE forecasting methods, a Bayesian aggregation algorithm that greatly improves extraction of scientific expert opinion or science courts that could mobilize much more talent and resources to neglected problems with good cases, some key steps in biotech enhancement, AI safety research when AI is better understood, and more

This Paul Christiano post discusses the virtues of the donor-advised fund/"Fund for the Future" approach; Giving What We Can has already set up a charitable trust to act as a donor-advised fund in the UK, with one coming soon in the US, and Fidelity already offers a standardized donor-advised fund in America (DAFs allow one to claim tax benefits of donation immediately and then allow the donation to compound); there was much discussion this month about the details of setting up a DAF dedicated to far future causes (the main logistical difficulties are setting up the decision criteria, credibility, and maximum protection from taxation and disruption)

Pablo8y30

Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.

Eliezer wrote this in 1999:

My current estimate, as of right now, is that humanity has no more than a 30% chance of making it, probably less. The most realistic estimate for a seed AI transcendence is 2020; nanowar, before 2015.

16lukeprog11y

I'm eager to see Eliezer's planned reply to your "ETA2", but in the meantime, here are a few of my own thoughts on this... My guess is that movement-building and learning are still the best things to do right now for AI risk reduction. CEA, CFAR, and GiveWell are doing good movement-building, though the GiveWell crowd tends to be less interested in x-risk mitigation. GiveWell is doing a large share of the EA-learning, and might eventually (via GiveWell labs) do some of the x-risk learning (right now GiveWell has a lot of catching up to do on x-risk). The largest share of the "explicit" x-risk learning is happening at or near FHI & MIRI, including e.g. Christiano. Lots of "implicit" x-risk learning is happening at e.g. NASA where it's not clear that EA-sourced funding can have much marginal effect relative to the effect it could have on tiny organizations like MIRI and FHI. My impression, which could be wrong, is that GiveWell's ability to hire more researchers is not funding-limited but rather limited by management's preference to offer lower salaries than necessary to ensure cause loyalty. (I would prefer GiveWell raise salaries and grow its research staff faster.) AMF could be fully funded relatively easily by Good Ventures or the Gates Foundation but maybe they're holding back because this would be discouraging to the EA movement: small-scale donors requiring the high-evidence threshold met by GiveWell's top charities would say "Well, I guess there's nothing for little 'ol me to do here." (There are other reasons they may be holding back, too.) I think accelerating learning is more important right now than a DAF. Getting high-quality evidence about which x-risk mitigation efforts are worthwhile requires lots of work, but one thing we've learned in the past decade is that causes with high-quality evidence for their effectiveness tend to get funded, and this trend is probably increasing. The sooner we do enough learning to have high-quality evidence for the goo

3Peter Wildeford11y

Are these available? Are they the standard stuff (i.e., "Evidence and Import")? ~ How do you arrive at that conclusion? I'm less skeptical of the cause-specific claim than the organization-specific claim, but it's worth digging deeper into.

See in context

35 How does MIRI Know it Has a Medium Probability of Success?

by Peter Wildeford

1st Aug 2013

1 min read

146

35

In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success. What is this probability estimate based on and how is success defined?

I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.

(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)

Existential RiskForecasting & PredictionMachine Intelligence Research Institute (MIRI)AIWorld Optimization

Personal Blog

35

How does MIRI Know it Has a Medium Probability of Success?

New Comment

146 comments, sorted by

top scoring

Click to highlight new comments since: Today at 3:18 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Eliezer Yudkowsky11y340

Most fundamentally, it's based on taking at face value a world in which nobody appears to be doing similar work or care sufficiently to do so. In the world taken at face value, MIRI is the only organization running MIRI's workshops and trying to figure out things like tiling self-modifying agents and getting work started early on what is probably a highly serial time-sensitive task.

Success is defined most obviously as actually constructing an FAI, and it would be very dangerous to have any organizational model in which we were not trying to do this (someone who conceives of themselves as an ethicist whose duty it is to lecture others, and does not intend to solve the problem themselves, is exceedingly unlikely to confront the hardest problems). But of course if our work were picked up elsewhere and reused after MIRI itself died as an organization for whatever reason, or if in any general sense the true history as written in the further future says that MIRI mattered, I should not count my life wasted, nor feel that we had let down MIRI's donors.

[-]Furcas11y100

Ok, but that doesn't increase the probability to 'medium' from the very low initial probability of MIRI or another organization benefiting from MIRI's work solving the extremely hard problem of Friendly AI before anyone else screws it up.

I've read all your posts in the threads linked by the OP, and if multiplying the high beneficial impact of Friendly AI by the low probability of success isn't allowed, I honestly don't see why I should donate to MIRI.

[-]Eliezer Yudkowsky11y280

If this was a regular math problem and it wasn't world-shakingly important, why wouldn't you expect that funding workshops and then researchers would cause progress on it?

Assigning a very low probability to progress rests on a sort of backwards reasoning wherein you expect it to be difficult to do things because they are important. The universe contains no such rule. They're just things.

It's hard to add a significant marginal fractional pull to a rope that many other people are pulling on. But this is not a well-tugged rope!

[-]Furcas11y130

I'm not assigning a low probability to progress, I'm assigning a low probability to success.

Where FAI research is concerned, progress is only relevant in as much as it increases the probability of success, right?

Unlike a regular math problem, you've only got one shot at getting it right, and you're in a race with other researchers who are working on an easier problem (seed AI, Friendly or not). It doesn't matter if you're 80% of the way there if we all die first.

Edited to add and clarify: Even accounting for the progress I think you're likely to make, the probability of success remains low, and that's what I care about.

[-]Eliezer Yudkowsky11y100

Clarifying question: What do you think is MIRI's probability of having been valuable, conditioned on a nice intergalactic future being true?

1Furcas11y

Pretty high. More than 10%, definitely. Maybe 50%?

[-]CarlShulman11y250

A non-exhaustive list of some reasons why I strongly disagree with this combination of views:

AI which is not vastly superhuman can be restrained from crime, because humans can be so restrained, and with AI designers have the benefits of the ability to alter the mind's parameters (desires, intuitions, capability for action, duration of extended thought, etc) inhibitions, test copies in detail, read out its internal states, and so on, making the problem vastly easier (although control may need to be tight if one is holding back an intelligence explosion while this is going on)
If 10-50 humans can solve AI safety (and build AGI!) in less than 50 years, then 100-500 not very superhuman AIs at 1200x speedup should be able to do so in less than a month
There are a variety of mechanisms by which humans could monitor, test, and verify the work conducted by such systems
The AIs can also work on incremental improvements to the control mechanisms being used initially, with steady progress allowing greater AI capabilities to develop better safety measures, until one approaches perfect safety
If a small group can solve all the relevant problems over a few decades, then probably a large portio

... (read more)

[-]Wei Dai11y140

Paul Christiano, and now you, have started using the phrase "AI control problems". I've gone along with it in my discussions with Paul, but before many people start adopting it maybe we ought to talk about whether it makes sense to frame the problem that way (as opposed to "Friendly AI"). I see a number of problems with it:

Control != Safe or Friendly. An AI can be perfectly controlled by a human and be extremely dangerous, because most humans aren't very altruistic or rational.
The framing implicitly suggests (and you also explicitly suggest) that the control problem can be solved incrementally. But I think we have reason to believe this is not the case, that in short "safety for superintelligent AIs" = "solving philosophy/metaphilosophy" which can't be done by "incremental improvements to the control mechanisms being used initially".
"Control" suggests that the problem falls in the realm of engineering (i.e., belongs to the reference class of "control problems" in engineering, such as "aircraft flight control"), whereas, again, I think the real problem is one of philosophy (plus lots of engineering

... (read more)

4CarlShulman11y

Thanks for those thoughts. Nick Bostrom uses the term in his book, and it's convenient for separating out pre-existing problems with "we don't know what to do with our society long term, nor is it engineered to achieve that" and the particular issues raised by AI. In the situation I mentioned, not vastly superintelligent initially (and capabilities can vary along multiple dimensions, e.g. one can have many compartmentalized copies of an AI system that collectively deliver a huge number of worker-years without any one of them possessing extraordinary capabilities. What is your take on the strategy-swallowing point: if humans can do it, then not very superintelligent AIs can. There is an ambiguity there. I'll mention it to Nick. But, e.g. Friendliness just sounds silly. I use "safe" too, but safety can be achieved just by limiting capabilities, which doesn't reflect the desire to realize the benefits.

8Wei Dai11y

It's easy to imagine AIXI-like Bayesian EU maximizers that are powerful optimizers but incapable of solving philosophical problems like consciousness, decision theory, and foundations of mathematics, which seem to be necessary in order to build an FAI. It's possible that that's wrong, that one can't actually get to "not very superintelligent AIs" unless they possessed the same level of philosophical ability that humans have, but it certainly doesn't seem safe to assume this. BTW, what does "strategy-swallowing" mean? Just "strategically relevant", or more than that? I suggested "optimal AI" to Luke earlier, but he didn't like that. Here are some more options to replace "Friendly AI" with: human-optimal AI, normative AI (rename what I called "normative AI" in this post to something else), AI normativity. It would be interesting and useful to know what options Eliezer considered and discarded before settling on "Friendly AI", and what options Nick considered and discarded before settling on "AI control". (I wonder why Nick doesn't like to blog. It seems like he'd want to run at least some of the more novel or potentially controversial ideas in his book by a wider audience, before committing them permanently to print.)

7CarlShulman11y

Such systems, hemmed in and restrained, could certainly work on better AI designs, and predict human philosophical judgments. Predicting human philosophical judgments accurately and reporting those predictions is close enough. "Control problem." He circulates them to reviewers, in wider circles as the book becomes more developed. And blogging half-finished idea on the internet is exactly what one shouldn't do if one is worried about committing controversial ideas to print.

6Wei Dai11y

In case this is why you don't tend to talk about your ideas in public either, except in terse (and sometimes cryptic) comments or in fully polished papers, I wanted to note that I've never had a cause to regret blogging (or posting to mailing lists) any of my half-finished ideas. As long as your signal to noise ratio is fairly high, people will remember the stuff you get right and forget the stuff you get wrong. The problem I see with committing ideas to print (as in physical books) is that books don't come with comments attached pointing out all the parts that are wrong or questionable.

4Wei Dai11y

If such a system is powerful enough to predict human philosophical judgments using its general intelligence, without specifically having been programmed with a correct solution for metaphilosophy, it seems very likely that it would already be strongly superintelligent in many other fields, and hence highly dangerous. (Since you seem to state this confidently but don't give much detail, I wonder if you've discussed the idea elsewhere at greater length. For example I'm assuming that you'd ask the AI to answer questions like "What would Eliezer conclude about second-order logic after thinking undisturbed about it for 100 years?" but maybe you have something else in mind?) I guess I actually meant "potentially wrong" rather than "controversial", and I was suggesting that he blog about them after privately circulating to reviewers, but before publishing in print.

4CarlShulman11y

The thought is much more bite-sized and tractable questions to work with less individually capable systems (with shorter time horizons, etc) like: "find a machine-checkable proof of this lemma" or "I am going to read one of these 10 papers to try to shed light on my problem using random selection, score each with the predicted rating I will give the paper's usefulness after reading it." I discussed this in a presentation at the FHI (focused on WBE, where the issue of unbalanced abilities relative to humans does not apply), and the concepts will be discussed in Nick's book.

5Wei Dai11y

Based on the two examples you give, which seem to suggest a workflow with a substantial portion still being done by humans (perhaps even the majority of the work in the case of the more philosophical parts of the problem), I don't see how you'd arrive at this earlier conclusion: Do you have any materials from the FHI presentation or any other writeup that you can share, that might shed more light? If not, I guess I can wait for the book... It's hard to discuss your specific proposal without understanding it in more detail, but in general I worry that the kind of AI you suggest would be much better at helping to improve AI capability than at helping to solve Friendliness, since solving technical problems is likely to be more of a strength for such an AI than predicting human philosophical judgments, and unless humanity develops much better coordination abilities than it has now (so that everyone can agree or be forced to refrain from trying to develop strongly superintelligent AIs until the Friendliness problem is solved), such an AI isn't likely to ultimately contribute to a positive outcome.

1CarlShulman11y

Yes, the range of follow-up examples there was a bit too narrow, I was starting from the other end and working back. Smaller operations could be chained, parallelized (with limited thinking time and capacity per unit), used to check on each other in tandem with random human monitoring and processing, and otherwise leveraged to minimize the human bottleneck element. A strong skew of abilities away from those directly useful for Friendliness development makes things worse, but leaves a lot of options. Solving technical problems can let you work to, e.g. * Create AIs with ability distributions directed more towards "philosophical" problems * Create AIs with simple sensory utility functions that are easier to 'domesticate' (short time horizons, satiability, dependency on irreplaceable cryptographic rewards that only the human creators can provide, etc) * Solve the technical problems of making a working brain emulation model * Create software to better detect and block unintended behavior, Yes, that's the biggest challenge for such bootstrapping approaches, which depends on the speedup in safety development one gets out of early models, the degree of international peace and cooperation, and so forth.

4Wei Dai11y

This strikes me as quite risky, as the amount of human monitoring has to be really minimal in order to solve a 50-year problem in 1 month, and earlier experiences with slower and less capable AIs seem unlikely to adequately prepare the human designers to come up with fully robust control schemes, especially if you are talking about a time scale of months. Can you say a bit more about the conditions you envision where this proposal would be expected to make a positive impact? It seems to me like it might be a very narrow range of conditions. For example if the degree of international peace and cooperation is very high, then a better alternative may be an international agreement to develop WBE tech while delaying AGI, or an international team to take as much time as needed to build FAI while delaying other forms of AGI. I tend to think that such high degrees of global coordination are implausible, and therefore put most of my hope in scenarios where some group manages to obtain a large tech lead over the rest of the world and are thereby granted a measure of strategic initiative in choosing how best to navigate the intelligence explosion. Your proposal might be useful in such a scenario, if other seemingly safer alternatives (like going for WBE, or having genetically enhanced humans build FAI with minimal AGI assistance) are out of reach due to time or resource constraints. It's still unclear to me why you called your point "strategy-swallowing" though, or what that phrase means exactly. Can you please explain?

3CarlShulman11y

I certainly didn't say that would be risk-free, but it interacts with other drag factors on very high estimates of risk. In the full-length discussion of it, I pair it with discussion of historical lags in tech development between leader and follower in technological arms races (longer than one month) and factors relative to corporate and international espionage, raise the possibility of global coordination (or at least between the leader and next closest follower), and so on. It also interacts with technical achievements in producing 'domesticity' short of exact unity of will. When strategy A to a large extent can capture the impacts of strategy B.

1Wei Dai11y

If you're making the point as part of an argument against "either Eliezer's FAI plan succeeds, or the world dies" then ok, that makes sense. ETA: But it seems like it would be very easy to take "if humans can do it, then not very superintelligent AIs can" out of context, so I'd suggest some other way of making this point. Sorry, I'm still not getting it. What does "impacts of strategy" mean here?

2torekp11y

I don't think that separation is a good idea. Not knowing what to do with our society long term is a relatively tolerable problem until an upcoming change raises a significant prospect of locking-in some particular vision of society's future. (Wei-Dai raises similar points in your exchange of replies, but I thought this framing might still be helpful.)

2Vladimir_Nesov11y

If we are talking about goal definition evaluating AI (and Paul was probably thinking in the context of some sort of indirect normativity), "control" seems like a reasonable fit. The primary philosophical issue for that part of the problem is decision theory. (I agree that it's a bad term for referring to FAI itself, if we don't presuppose a method of solution that is not Friendliness-specific.)

7Kawoomba11y

Not that it should be used to dismiss any of your arguments, but reading your other comments in this thread I thought you must be playing devil's advocate. Your phrasing here seems to preclude that possibility. If you are so strongly convinced that while AGI is a non-negligible x-risk, MIRI will probably turn out to have been without value even if a good AGI outcome were to be eventually achieved, why are you a research fellow there? I'm puzzled. Let's consider an edge case: even if MIRI's factual research turned out to be strictly non-contributing to an eventual solution, there's no reasonable doubt that it has raised awareness of the issue significantly (in relative terms). Would the current situation with the CSER or FHI be unchanged or better if MIRI had never existed? Do you think those have a good chance of being valuable in bringing about a good outcome? Answering 'no' to the former and 'yes' to the latter would transitively imply that MIRI is valuable as well. I.e. that alone --nevermind actual research contributions -- would make it valuable in hindsight, given an eventual positive outcome. Yet you're strongly opposed to that view?

9CarlShulman11y

The "combination of views" includes both high probability of doom, and quite high probability of MIRI making the counterfactual difference given survival. The points I listed address both. I think MIRI's expected impact is positive and worthwhile. I'm glad that it exists, and that it and Eliezer specifically have made the contributions they have relative to a world in which they never existed. A small share of the value of the AI safety cause can be quite great. That is quite consistent with thinking that "medium probability" is a big overestimate for MIRI making the counterfactual difference, or that civilization is almost certainly doomed from AI risk otherwise. Lots of interventions are worthwhile even if a given organization working on them is unlikely to make the counterfactual difference. Most research labs working on malaria vaccines won't invent one, most political activists won't achieve big increases in foreign aid or immigration levels or swing an election, most counterproliferation expenditures won't avert nuclear war, asteroid tracking was known ex ante to be far more likely to discover we were safe than that there was an asteroid on its way and ready to be stopped by a space mission. The threshold for an x-risk charity of moderate scale to be worth funding is not a 10% chance of literally counterfactually saving the world from existential catastrophe. Annual world GDP is $80,000,000,000,000, and wealth including human capital and the like will be in the quadrillions of dollars. A 10% chance of averting x-risk would be worth trillions of present dollars. We've spent tens of billions of dollars on nuclear and bio risks, and even $100,000,000+ on asteroids (averting dinosaur-killer risk on the order of 1 in 100,000,000 per annum). At that exchange rate again a 10% x-risk impact would be worth trillions of dollars, and governments and philanthropists have shown that they are ready to spend on x-risk or GCR opportunities far, far less likely to make a c

1Kawoomba11y

I see. We just used different thresholds for valuable, you used "high probability of MIRI making the counterfactual difference given survival", while for me just e.g. speeding Norvig/Gates/whoever a couple years along the path until they devote efforts to FAI would be valuable, even if it were unlikely to Make The Difference (tm). Whoever would turn out to have solved the problem, it's unlikely that their AI safety evaluation process ("Should I do this thing?") would work in a strict vacuum, i.e. whoever will one day have evaluated the topic and made up their mind to Save The World will be highly likely to have stumbled upon MIRI's foundational work. Given that at least some of the steps in solving the problem are likely to be quite serial (sequential) in nature, the expected scenario would be that MIRI's legacy would at least provide some speed-up; a contribution which, again, I'd call valuable, even if it were unlikely to make or break the future. If the Gates Foundation had someone evaluate the evidence for AI-related x-risk right now, you probably wouldn't expect MIRI research, AI researcher polls, philosophical essays etc. to be wholly disregarded.

3CarlShulman11y

I used that threshold because the numbers being thrown around in the thread were along those lines, and are needed for the "medium probability" referred to in the OP. So counterfactual impact of MIRI never having existed on x-risk is the main measure under discussion here. I erred in quoting your sentence in a way that might have made that hard to interpret. That's right, and one reason that I think that MIRI's existence has reduced expected x-risk, although by less than a 10% probability.

0Dr_Manhattan11y

Sorry hard to tell from the thread which combination of views. Eliezer's?

4CarlShulman11y

The view presented by Furcas, of probable doom, and "[m]ore than 10%, definitely. Maybe 50%" probability that MIRI will be valuable given the avoidance of doom, which in the context of existential risk seems to mean averting the risk.

8Eliezer Yudkowsky11y

...um. It seems to me that if I believed what I infer you believe, I would be donating to MIRI while frantically trying to figure out some way to have my doomed world actually be saved.

3Furcas11y

Why? You (and everybody else) will almost certainly fail anyway, and you say I shouldn't multiply this low probability by the utility of saving the world. The only way I see is what MIRI is doing. Edited to add: While this is interesting, what I was really asking in my first post is, if you think the odds of MIRI succeeding are not low, why do you think so?

[-]Eliezer Yudkowsky11y150

Because sometimes the impossible can be done, and I don't know how to estimate the probability of that. What would you have estimated in advance, without knowing the result, was the chance of success for the AI-Box Experiment? How about if I told you that I was going to write the most popular Harry Potter fanfiction in the world and use it to recruit International Mathematical Olympiad medalists? There may be true impossibilities in this world. Eternal life may be one such, if the character of physical law is what is it appears to be, to our sorrow. I do not think that FAI is one of those. So I am going to try. We can work out what the probability of success was after we have succeeded. The chance which is gained is not gained by turning away or by despair, but by continuing to engage with and attack the problem, watching for opportunities and constantly advancing.

If you don't believe me about that aspect of heroic epistemology, feel free not to believe me about not multiplying small probabilities either.

[-]CarlShulman11y170

heroic epistemology

Could you give a more precise statement of what this is supposed to entail?

[-]Eliezer Yudkowsky11y270

Not easily. Antiantiheroic epistemology might be a better term, i.e., I think that a merely accurate epistemology doesn't have a built-in mechanism which prevents people from thinking they can do things because the outside view says it's nonvirtuous to try to distinguish yourself within reference class blah. Antiantiheroic epistemology doesn't say that it's possible to distinguish yourself within reference class blah so much as it thinks that the whole issue is asking the wrong question and you should mostly be worrying about staying engaged with the object-level problem because this is how you learn more and gain the ability to take opportunities as they arrive. An antiheroic epistemology that throws up some reference class or other saying this is impossible will regard you as trying to distinguish yourself within this reference class, but this is not what the antiantiheroic epistemology is actually about; that's an external indictment of nonvirtuosity arrived at by additional modus ponens to conclusions on which antiantiheroic epistemology sees no reason to expend cognitive effort.

Obviously from my perspective non-antiheroic epistemology cancels out to mere epistemology, simpl... (read more)

[-]CarlShulman11y340

"Antiantiheroic epistemology might be a better term, i.e., I think that a merely accurate epistemology doesn't have a built-in mechanism which prevents people from thinking they can do things because the outside view says it's nonvirtuous to try to distinguish yourself within reference class blah. "

Taken literally, I can't possibly disagree with this, but it doesn't seem to answer my question, which is "where is the positive evidence that one is not supposed to ignore." I favor combining many different kinds of evidence, including sparse data. And that can and does lead to very high expectations for particular individuals.

For example, several of my college fraternity brothers are now billionaires. Before facebook Mark Zuckerberg was clearly the person with the highest entrepreneurial potential that I knew, based on his intelligence, motivation, ambition, and past achievements in programming, business, and academics. People described him to me as resembling a young Bill Gates. His estimated expected future wealth based on that data if pursuing entrepreneurship, and informed by the data about the relationship of all of the characteristics I could track with it, wa... (read more)

[-]Eliezer Yudkowsky11y290

You are asking other people for their money and time, when they have other opportunities. To do that they need an estimate of the chance of MIRI succeeding

No they don't; they could be checking relative plausibility of causing an OK outcome without trying to put absolute numbers on a probability estimate, and this is reasonable due to the following circumstances:

The life lesson I've learned is that by the time you really get anywhere, if you get anywhere, you'll have encountered some positive miracles, some negative miracles, your plans will have changed, you'll have found that the parts which took the longest weren't what you thought they would be, and that other things proved to be much easier than expected. Your successes won't come through foreseen avenues, and neither will your failures. But running through it all will be the fundamental realization that everything you accomplished, and all the unforeseen opportunities you took advantage of, were things that you would never have received if you hadn't attacked the hardest part of the problem that you knew about straight-on, without distraction.

How do you estimate probabilities like that? I honestly haven't a clue. Now, w... (read more)

[-]CarlShulman11y130

If the year was 1960, which would you rather have?

10 smart people trying to build FAI for 20 years, 1960-1980
A billion dollars, a large supporting movement, prediction markets and science courts that make the state of the evidence on AI transparent, and teams working on FAI, brain emulations, cognitive enhancement, and more but starting in 1990 (in expectation closer to AI)

At any given time there are many problems where solutions are very important, but the time isn't yet right to act on them, rather than on the capabilities to act on them, and also to deal with the individually unexpected problems that come along so regularly. Investment-driven and movement-building-driven discount rates are relevant even for existential risk.

GiveWell has grown in influence much faster than the x-risk community while working on global health, and are now in the process of investigating and pivoting towards higher leverage causes, with global catastrophic risk among the top three under consideration.

7Eliezer Yudkowsky11y

I'd rather have both, hence diverting some marginal resources to CFAR until it was launched, then switching back to MIRI. Is there a third thing that MIRI should divert marginal resources to right now?

[-]CarlShulman11y260

I have just spent a month in England interacting extensively with the EA movement here (maybe your impressions from the California EA summit differ, I'd be curious to hear). Donors interested in the far future are also considering donations to the following (all of these are from talks with actual people making concrete short-term choices; in addition to donations, people are also considering career choices post-college):

80,000 Hours, CEA and other movement building and capacity-increasing organizations (including CFAR), which also increase non-charity options (e.g. 80k helping people going into scientific funding agencies and political careers where they will be in a position to affect research and policy reactions to technologies relevant to x-risk and other trajectory changes)
AMF/GiveWell charities to keep GiveWell and the EA movement growing while actors like GiveWell, Paul Christiano, Nick Beckstead and others at FHI, investigate the intervention options and cause prioritization, followed by organization-by-organization analysis of the GiveWell variety, laying the groundwork for massive support for the top far future charities and organizations identified by said processes

... (read more)

[-]Eliezer Yudkowsky11y240

That is a sort of discussion my brain puts in a completely different category. Peter and Carl, please always give me a concrete alternative policy option that (allegedly) depends on a debate, if such is available; my brain is then far less likely to label the conversation "annoying useless meta objections that I want to just get over with as fast as possible".

Can we have a new top-level comment on this?

8CarlShulman11y

I edited my top-level comment to include the list and explanation.

0Rain11y

Cool, if MIRI keeps going, they might be able to show FAI as top focus with adequate evidence by the time all of this comes together.

0lukeprog11y

Well, in collaboration with FHI. As soon as Bostrom's Superintelligence is released, we'll probably be building on and around that to make whatever cases we think are reasonable to make.

1lukeprog11y

Thanks; I found this comment illuminating about your views.

0lukeprog11y

Also, because this.

1Rain11y

I read every blog post they put out. I figure I can use my retirement savings for this. I thought it came from them being collectively foolish or ignorant regarding Friendliness rather than AGI. Meh. Sounds like Lean Six Sigma or some other buzzword business process improvement plan. Luckily, Bostrom is already doing work like Bostrom's. Too indirect for my taste. Not very scary compared to AI. Lots of known methods to combat green goo.

[-]Wei Dai11y130

If you don't believe me about that aspect of heroic epistemology, feel free not to believe me about not multiplying small probabilities either.

Multiplying small probabilities seems fine to me, whereas I really don't get "heroic epistemology".

You seem to be suggesting that "heroic epistemology" and "multiplying small probabilities" both lead to the same conclusion: support MIRI's work on FAI. But this is the case only if working on FAI has no negative consequences. In that case, "small chance of success" plus "multiplying small probabilities" warrants working on FAI, just as "medium probability of success" and "not multiplying small probabilities" does. But since working on FAI does have negative consequences, namely shortening AI timelines and (in the later stages) possibly directly causing the creation an UFAI, just allowing multiplication by small probabilities is not sufficient to warrant working on FAI if the probability of success is low.

I am really worried that you are justifying your current course of action through a novel epistemology of your own invention, which has not been widely vetted (or even widely understood). Most new ideas are wrong, and I think you ought to treat your own new ideas with deeper suspicion.

1Eliezer Yudkowsky11y

I'm a reactionary, not an innovator, dammit! Reacting against this newfangled antiheroic 'reference class' claim that says we ought to let the world burn because we don't have enough of a hero license! Ahem. I'm also really unconvinced by the claim that this work could reasonably have expected net negative consequences. I'm worried about the dynamics and evidence of GiveDirectly. But I don't think GD has negative consequences, that would be a huge stretch. It's possible maybe but it's certainly not the arithmetic expectation and with that said, I worry that this 'maybe negative' stuff is impeding EA motivation generally, there is much that is ineffectual to be wary of, and missed opportunity costs, but trying to warn people against reverse or negative effects seems pretty perverse for anything that has made it onto Givewell's Top 3, or CFAR, or FHI, or MIRI. Info that shortens AI timelines should mostly just not be released publicly and I don't see any particularly plausible way for a planet to survive without having some equivalent of MIRI doing MIRI's job, and the math thereof should be started as early as feasible.

[-]Wei Dai11y110

I'm a reactionary, not an innovator, dammit! Reacting against this newfangled antiheroic 'reference class' claim that says we ought to let the world burn because we don't have enough of a hero license!

"Reference class" to me is just an intuitive way of thinking about updating on certain types of evidence. It seems like you're saying that in some cases we ought to use the inside view, or weigh object-level evidence more heavily, but 1) I don't understand why you are not worried about "inside view" reasoning typically producing overconfidence or why you don't think it's likely to produce overconfidence in this case, and 2) according to my inside view, the probability of a team like the kind you're envisioning solving FAI is low, and a typical MIRI donor or potential donor can't be said to have much of an inside view on this matter, and has to use "reference class" reasoning. So what is your argument here?

I'm also really unconvinced by the claim that this work could reasonably have expected net negative consequences.

Every AGI researcher is unconvinced by that, about their own work.

but trying to warn people against reverse or negative effects seem

... (read more)

0lukeprog11y

Agree, though luckily there are other Friendliness-specific problems that we can start solving right now. Presumably, security technology similar to what has mostly worked for the Manhattan project, secret NSA projects, etc. But yeah, it's a big worry. But what did you have in mind about flying completely under the radar? There are versions of an FAI team that could be funded pretty discretely by just one person.

4Wei Dai11y

I listed some in another comment, but they are not the current focus of MIRI research. Instead, MIRI is focusing on FAI-relevant problems that do shorten AI timelines (i.e., working on "get most of the way to being able to build an AGI"), such as decision theory and logical uncertainty. As I noted in previous comments, the economics of information security seems to greatly favor the offense, so you have to have to spend much more resources than your attackers in order to maintain secrets. That's probably the best bet as far as avoiding having your results stolen, but introduces other problems, such as how to attract talent, and whether you can fund a large enough team that way. (Small teams might increase the chances of accidental UFAI creation, since there would be less people to look out for errors.) And given that Eliezer is probably already on the radar of most AGI researchers, you'd have to find a replacement for him on this "under the radar" team.

-1Eliezer Yudkowsky11y

I should ask this question now rather than later: Is there a concrete policy alternative being considered by you? And on one obvious 'outside view', they'd be right - it's a very strange and unusual situation, which took me years to acknowledge, that this one particular class of science research could have perverse results. There's many attempted good deeds which have no effect, but complete backfires make the news because they're rare. (Hey, maybe the priors in favor of good outcomes from the broad reference class of scientific research are so high that we should just ignore the inside view which says that AGI research will have a different result!) And even AGI research doesn't end up making it less likely that AGI will be developed, please note - it's not that perverse in its outcome.

[-]Wei Dai11y130

Is there a concrete policy alternative being considered by you?

I'm currently in favor of of the following:

research on strategies for navigating intelligence explosion (what I called "Singularity Strategies")
pushing for human intelligence enhancement
pushing for a government to try to take an insurmountable tech lead via large scale intelligence enhancement
research into a subset of FAI-related problems that do not shorten AI timelines (at least as far as we can tell), such as consciousness, normative ethics, metaethics, metaphilosophy
advocacy/PR/academic outreach on the dangers of AGI progress

There's many attempted good deeds which have no effect, but complete backfires make the news because they're rare.

What about continuing physics research possibly leading to a physics disaster or new superweapons, biotech research leading to biotech disasters, nanotech research leading to nanotech disasters, WBE research leading to value drift and Malthusian outcomes, computing hardware research leading to deliberate or accidental creation of massive simulated suffering (aside from UFAI)? In addition, I thought you believed that faster economic growth made a good outcome l... (read more)

0Paul Crowley11y

It seems odd to insist that someone explicitly working on benevolence should consider themselves to be in the same reference class as someone who thinks they just need to take care of the AGI and the benevolence will pretty much take care of itself.

1Wei Dai11y

I wasn't intending to use "AGI researchers" as a reference class to show that Eliezer's work is likely to have net negative consequences, but to show that people whose work can reasonably be expected to have net negative consequences (of whom AGI researchers is a prominent class) still tend not to believe such claims, and therefore Eliezer's failure to be convinced is not of much evidential value to others. The reference class I usually do have in mind when I think of Eliezer is philosophers who think they have the right answer to some philosophical problem (virtually all of whom end up being wrong or at least incomplete even if they are headed in the right direction). ETA: I've written a post that expands on this comment.

2lukeprog11y

Related to heroic epistemology: The Five Cognitive Distortions of People Who Get Stuff Done.

2Eliezer Yudkowsky11y

"Cognitively distorted" people should lose. People who get stuff done should have their alternate thinking processes carefully examined to see how the divergence is more rational than the non-divergence.

2solipsist11y

It's not clear to me why people who accurately model the world should outperform those who follow less cognitively demanding heuristics. I've seen this position stated as a truism during a debate, but have never read an argument for or against it. Would someone be able to link to an argument about following non-robust shortcuts to rationality, or write a short case against that practice?

1wedrifid11y

If the aforementioned analysis of thinking processes finds that the advantage comes from superior allocation of bounded computational resources then that would be an interesting finding and a sufficient explanation. In some cases the alternate heuristics may be worth adopting.

0TheOtherDave11y

Is the common-sense expectation that non-robust heuristics deliver poor results in a wider subset of possible future environments than robust heuristics not adequate?

-1Furcas11y

The most charitable way I can interpret this is: "Yeah, the middle point of my probability interval for a happy ending is very low, but the interval is large enough that its upper bound isn't that low, so it's worth my time and your money trying to reach a happy ending." Am I right? I don't. :)

5Eliezer Yudkowsky11y

I'm saying I don't know how to estimate heroic probabilities. I do not know any evenhanded rules which assign 'you can't do that' probability to humanity's survival which would not, in the hands of the same people thinking the same way, rule out Google or Apple, and maybe those happened to other people, but the same rules would also say that I couldn't do the 3-5 other lesser "impossibilities" I've done so far. Sure, those were much easier "impossibilities" but the point is that the sort of people who think you can't build Friendly AI because I don't have a good-enough hero license to something so high-status or because heroic epistemology allegedly doesn't work in real life, would also claim all those other things couldn't happen in real life, if asked without benefit of advance knowledge to predict the fate of Steve Wozniak or me personally; that's what happens when you play the role of "realism".

[-]CarlShulman11y150

Overconfidence (including factual error about success rates) is pervasive in entrepreneurs, both the failures and successes (and the successes often fail on a second try, although they have somewhat better odds). The motivating power of overconfidence doesn't mean the overconfidence is factually correct or that anyone else should believe it. And the mega-successes tended to look good in expected value, value of information, and the availability of good intermediate outcomes short of mega-success: there were non-crazy overconfident reasons to pursue them. The retreat to "heroic epistemology" rather than reasons is a bad sign relative to those successes, and in any case most of those invoking heroic epistemology style reasoning don't achieve heroic feats.

Applying the outside view startup statistics, including data on entrepreneur characteristics like experience and success rates of repeat entrepreneurs is not magic or prohibitively difficult. Add in the judgments of top VCs to your model.

For individuals or area/firm experts, one can add in hard-to-honestly-signal data (watching out for overconfidence in various ways, using coworkers, etc). That model would have assigned a m... (read more)

8Eliezer Yudkowsky11y

Did I say that? No, I did not say that. You should know better than to think I would ever say that. Knowingly make an epistemic error? Say "X is false but I believe it is true"? Since we're talking heroism anyway, Just who the hell do you think I am? Okay, so suppose we jump back 4 years and I'm saying that maybe I ought to write a Harry Potter fanfiction. And it's going to be the most popular HP fanfiction on the whole Internet. And Mathematical Olympiad winners will read it and work for us. What does your nonheroic epistemology say? Because I simply don't believe that (your) nonheroic epistemology gets it right. I don't think it can discriminate between the possible impossible and the impossible impossible. It just throws up a uniform fog of "The outside view says it is nonvirtuous to try to distinguish within this reference class." I thought Quixey was doomed because the idea wasn't good enough. Michael Vassar said that Quixey would succeed because Tomer Kagan would succeed at anything he tried to do. Michael Vassar was right (a judgment already finalized because Quixey has already gotten further than I thought was possible). This made me update on Michael Vassar's ability to discriminate Tomer Kagans in advance from within a rather large reference class of people trying to be Tomer Kagan.

9CarlShulman11y

That's what the arguments you've given for this have mostly amounted to. You have said "I need to believe this to be motivated and do productive work" in response to questions about the probabilities in the past, while not giving solid reasons for the confidence. When did you predict that? Early on I did not hear you making such claims, with the tune changing after it became clear that demand for it was good. 4 years ago I did advocate getting Math Olympiad people, and said they could be gotten, and had empirical evidence of that from multiple angles. And I did recognize your writing and fiction were well-received, and had evidence from the reactions to "Staring into the Singularity" and OB/LW. You tried several methods, including the rationality book, distributing written rationality exercises/micromanaging CFAR content, and the fanfiction. Most of them wasted time and resources without producing results, and one succeeded. And there is a larger context, that in addition to the successes you are highlighting the path includes: Flare, LOGI and associated research, pre-Vassar SI management issues, open-source singularity, commercial software, trying to create non-FAI before nanowars in 2015. Tomer is indeed pretty great, but I have heard Michael say things like that about a number of people and projects over the years. Most did not become like Quixey. And what's the analogy here? That people with good ability to predict success in scientific research have indicated you will succeed taking into account how the world and the space of computer science and AI progress must be for that? That Michael has?

6Eliezer Yudkowsky11y

This does not sound like something I would ever say. Ever, even at age 16. Your memory conflicts with mine. Is there any way to check?

6ESRogs11y

As a LessWrong reader, I notice that I am confused because this does not sound like something you would say, but I'm not sure I could explain the difference between this and "heroic epistemology." EDIT: for the benefit of other readers following the conversation, Eliezer gives a description of heroic epistemology here.

3lukeprog11y

For the record, I don't recall ever hearing you say something like this in my presence or online, and if somebody had told me in person that you had said this, I think I would've raised a skeptical eyebrow and said "Really? That doesn't sound like something Eliezer would say. Have you read The Sequences?" But also, I remain confused about the normative content of "heroic epistemology."

3CarlShulman11y

Ask Anna about it, she was present on both occasions, at the tail end of the Singularity Summit workshop discussion in New York City, and at the roundtable meeting at the office with Anna, Paul, Luke, and Louie. In a related vein, arguments like this are arguments that someone could do A, but not so much that you will do A (and B and C and...). My impression is of too many arguments like the former and enough of the latter. If you can remedy that, it would be great, but it is a fact about the responses I have seen.

[-]AnnaSalamon11y180

Eliezer emailed me to ask me about it (per Carl's request, above); I emailed him back with the email below, which Eliezer requested I paste into the LW thread. Pasting:

In the majority of cases, people do not straightforwardly say "X is false, but I need to believe X anyhow". More often they wiggle, and polite conversation drops the subject.
You have made statements that I and at least some others interpreted as perhaps indicating such wiggles (i.e., as perhaps indicating a desire to hold onto false impressions by diverting conscious attention or discussion from a particular subject). You have never, to my knowledge, uttered a straight-forward endorsement of holding false beliefs. The wiggle-suggesting statements were not super-clear, and were not beyond the realm of possible misinterpretation.
Re: statements that seemed to me and some others to indicate possible wiggles: You have mentioned multiple times that, well, I forget, but something like, it'd be hard to do top focused research on FAI-like problems while estimating a 1% chance of success. You've also avoided giving probability estimates in a lot of contexts, and have sometimes discouraged conversations in whic

... (read more)

[-]Eliezer Yudkowsky11y100

From my internal perspective, the truth-as-I-experience-it is that I'm annoyed when people raise the topic because it's all wasted motion, the question sets up a trap that forces you into appearing arrogant, and I honestly think that "Screw all this, I'm just going to go ahead and do it and you can debate afterward what the probabilities were" is a perfectly reasonable response.

[-]AnnaSalamon11y190

From the perspective of folks choosing between supporting multiple lines of AI risk reduction effort, of which MIRI is only one, such probability estimates are not wasted effort.

Though your point about appearing arrogant is well taken. It's unfortunate that it isn't socially okay to publicly estimate a high probability of success, or to publicly claim one's own exceptionalism, when ones impressions point that way. It places a barrier toward honest conversation here.

[-]AnnaSalamon11y140

From my internal perspective, the truth-as-I-experience-it is that I'm annoyed when people raise the topic [of MIRI's success-odds] because [good reason].

I suspect this annoyance is easily misinterpreted, independent of its actual cause. Most humans respond with annoyance when their plans are criticized. Also, in situations where A has power over B, and where B then shares concerns or criticisms about A's plans, and where A responds with annoyance or with avoidance of such conversation... B is apt to respond (as I did) by being a bit hesitant to bring the topic up, and by also wondering if A is being defensive.

I'm not saying I was correct here. I'm also not sure what the fix is. But it might be worth setting a 1-minute timer and brainstorming or something.

7Eliezer Yudkowsky11y

If you were anyone else, this is ordinarily the point where I tell you that I'm just going to ignore all this and go ahead do it, and then afterward you can explain why it was either predictable in retrospect or a fluke, according to your taste. Since it's you: What's the first next goal you think I can't achieve, strongly enough that if I do it, you give up on non-heroic epistemology?

9CarlShulman11y

I'm familiar with this move. But you make it before failing too, so its evidentiary weight is limited, and insufficient for undertakings with low enough prior probability from all the other evidence besides the move. I don't buy the framing. The update would be mainly about you and the problem in question, not the applicability of statistics to reality. Two developments in AI as big as Pearl's causal networks (as judged by Norvig types) by a small MIRI team would be a limited subset of the problems to be solved by a project trying to build AGI with a different and very safe architecture before the rest of the world, and wouldn't address the question of the probability that such is needed in the counterfactual, but it would cause me to stop complaining and would powerfully support the model that MIRI can be more productive than the rest of the AI field when currently-available objective indicators put it as a small portion of the quality-adjusted capacity. If we want a predictor for success that's a lot better than the vast majority of quite successful entrepreneurs and pathbreaking researchers, making numerous major basic science discoveries and putting them together in a way that saves the world, then we need some evidence to distinguish the team and explain why it will make greater scientific contributions than any other ever with high reliability in a limited time. A lot of intermediate outcomes would multiply my credence in and thus valuation of the "dozen people race ahead of the rest of the world in AI" scenario, but just being as productive as von Neumann or Turing or Pearl or Einstein would not result in high probability of FAI success, so the evidence has to be substantial.

[-]Eliezer Yudkowsky11y110

I'm familiar with this move. But you make it before failing too

Sure, you try, sometimes you lose, sometimes you win. On anti-heroic epistemology (non-virtuous to attempt to discriminate within an outside view) there shouldn't be any impossible successes by anyone you know personally after you met them. They should only happen to other people selected post-facto by the media, or to people who you met because of their previous success.

I don't buy the framing. The update would be mainly about you and the problem in question, not the applicability of statistics to reality.

We disagree about how to use statistics in order to get really actually correct answers. Having such a low estimate of my rationality that you think that I know what correct statistics are, and am refusing to use them, is not good news from an Aumann perspective and fails the ideological Turing Test. In any case, surely if my predictions are correct you should update your belief about good frameworks (see the reasoning used in the Pascal's Muggle post) - to do otherwise and go on insisting that your framework was nonetheless correct would be oblivious.

Two developments in AI as big as Pearl's causal networ

... (read more)

[-]Wei Dai11y100

On anti-heroic epistemology (non-virtuous to attempt to discriminate within an outside view) there shouldn't be any impossible successes by anyone you know personally after you met them.

I don't understand why you say this. Given Carl's IQ and social circle (didn't he used to work for a hedge fund run by Peter Thiel?) why would it be very surprising that someone he personally knows achieves your current level of success after he meets them?

They should only happen to other people selected post-facto by the media, or to people who you met because of their previous success.

Carl referenced "Staring Into the Singularity" as an early indicator of your extraordinary verbal abilities (which explains much if not all of your subsequent successes). It suggests that's how you initially attracted his attention. The same is certainly true for me. I distinctly recall saying to myself "I should definitely keep track of this guy" when I read that, back in the extropian days. Is that enough for you to count as "people who you met because of their previous success"?

In any case, almost everyone who meets you now would count you as such. What arguments can you give... (read more)

3Eliezer Yudkowsky11y

Yes, no matter how many impossible things you do, the next person you meet thinks that they only heard of you because of them, ergo selection bias. This is an interesting question purely on a philosophical level - it seems to me to have some of the flavor of quantum suicide experiments where you can't communicate your evidence. In principle this shouldn't happen without quantum suicide for logically omniscient entities who already know the exact fraction of people with various characteristics, i.e., agree on exact priors, but I think it might start happening again to people who are logically unsure about which framework they should use.

8CarlShulman11y

To avoid talking past one another: I agree that one can and should update on evidence beyond the most solid empirical reference classes in predicting success. If you mean to say that a majority of the variation on a log scale in success (e.g. in wealth or scientific productivity) can be accounted for with properties of individuals and their circumstances, beyond dumb luck then we can agree on that. Some of those characteristics are more easily observed, while others are harder to discern or almost unmeasurable from a distance so that track records may be our best way to discern them. That is to say, repeated successes should not be explained by luck, but by updating estimates of hard-to-observe characteristics and world model. The distribution of successes and failures you have demonstrated is not "impossible" or driving a massive likelihood ratio given knowledge about your cognitive and verbal ability, behavioral evidence of initiative and personality, developed writing skill (discernible through inspection and data about its reception), and philosophical inclinations. Using measurable features, and some earlier behavioral or track record data one can generate reference classes with quite high levels of lifetime success, e.g. by slicing and dicing cohorts like this one. Updating on further successes delivers further improvements. But updating on hidden characteristics does not suffer exponential penalties like chance explanations, and there is a lot of distance to cover in hidden characteristics before a 10% probability of MIRI-derived FAI (or some other causal channel) averting existential catastrophe that would have occurred absent MIRI looks reasonable. Now large repeated updates about hidden characteristics still indicate serious model problems and should lead us to be very skeptical of those models. However, I don't see such very large surprising updates thus far. If difficulty is homogenous (at least as far as one can discern in advance), then we can use

2Eliezer Yudkowsky11y

Of course I expect you to say that, since to say otherwise given your previous statements is equivalent to being openly incoherent and I do not regard you so lowly. But I don't yet believe that you would actually have accepted or predicted those successes ante facto, vs. claiming ante facto that those successes were unlikely and that trying was overconfident. Which is why I repeat my question: What is the least impossible thing I could do next, where anything up to that is permitted by your model so it's equivalent to affirming that you think I might be able to do it, and anything beyond that was prohibited by your model so it's time to notice your confusion? I mean, if you think I can make one major AI breakthrough but not two, that's already a lot of confidence in me... is that really what your outside view would say about me? Please distinguish between the disputed reality and your personal memory, unless you're defining the above so broadly (and uncharitably!) that my 'wasted motion' FB post counts as an instance. Without significant work? I don't think I can do that. Why would you think I thought I could do that? If enough people agreed on that and DAGGRE could be done with relatively low effort on my part, I would do so, though I think I'd want at least some people committing in writing to large donations given success because it would be a large time commitment and I'm prior-skeptical that people know or are honest about their own reasons for disagreement; and I would expect the next batch of pessimists to write off the DAGGRE results (i.e., claim it already compatible with my known properties) so there'd be no long-term benefit. Still, 8 out of 8 on 80K's "Look how bad your common sense is!" test, plus I recall getting 9 out of 10 questions correct the last time I was asked for 90% probabilities on a CFAR calibration test, so it's possible I've already outrun the reference class of people who are bad at this. Though if it's mostly geopolitical questions

5CarlShulman11y

So, if von Neumann came out with similar FAI claims, but couldn't present compelling arguments to his peers (if not to exact agreement, perhaps within an order of magnitude) I wouldn't believe him. So showing that, e.g. your math problem-solving ability is greater than my point estimate, wouldn't be very relevant. Shocking achievements would lead me to upgrade my estimate of your potential contribution going forward (although most of the work in an FAI team would be done by others in any case), resolving uncertainty about ability, but that would not be enough as such, it would have to be the effect on my estimates of your predictive model. I would make predictions on evaluations of MIRI workshop research outputs by a properly constructed jury of AI people. If the MIRI workshops were many times more productive than comparably or better credentialed AI people according to independent expert judges (blinded to the extent possible) I would say my model was badly wrong, but I don't think you would predict a win on that. To avoid "too much work to do/prep for" and "disagreement about far future consequences of mundane predicted intermediates" you could give me a list of things that you or MIRI plan to attempt over the next 1, 3, and 5 years and I could pick one (with some effort to make it more precise). Yes, I have seen you writing about the 80k quiz on LW and 80k and elsewhere, it's good (although as you mention, test-taking skills went far on it). I predict that if we take an unbiased sample of people with similarly high cognitive test scores, extensive exposure to machine learning, and good career success (drawn from academia and tech/quant finance, say), and look at the top scorers on the 80k quiz and similar, their estimates for MIRI success will quite a bit closer to mine than yours. Do you disagree? Otherwise, I would want to see drastic outperformance relative to such a group on a higher-ceiling version (although this would be confounded by advance notice and

5Eliezer Yudkowsky11y

The question is not what convinces you that I can do FAI within the framework of your antiheroic epistemology. The question is what first and earliest shows that your antiheroic epistemology is yielding bad predictions. Is this a terrible question to ask for some reason? You've substituted an alternate question a couple of times now. From my perspective, you just asked how bad other people are at predicting such developments. The answer is that I don't know. Certainly many bloggers are terrible at it. I don't suppose you can give a quick example of a DAGGRE question?

4CarlShulman11y

Which I said in the very same paragraph. I already gave the example of independent judges evaluating MIRI workshop output, among others. If we make the details precise, I can set the threshold on the measure. Or we can take any number of other metrics with approximately continuous outputs where I can draw a line. But it takes work to define a metric precise enough to be solid, and I don't want to waste my time generating more and more additional examples or making them ultra-precise without feedback on what you will actually stake a claim on. I can't determine what's next without knowledge of what you'll do or try. http://blog.daggre.org/tag/prediction-market/

3Eliezer Yudkowsky11y

To clear up the ambiguity, does this mean you agree that I can do anything short of what von Neumann did, or that you don't think it's possible to get as far as independent judges favorably evaluating MIRI output, or is there some other standard you have in mind? I'm trying to get something clearly falsifiable, but right now I can't figure out the intended event due to sheer linguistic ambiguity. I also think that evaluation by academics is a terrible test for things that don't come with blatant overwhwelming unmistakable undeniable-even-to-humans evidence - e.g. this standard would fail MWI, molecular nanotechnology, cryonics, and would have recently failed 'high-carb diets are not necessarily good for you'. I don't particularly expect this standard to be met before the end of the world, and it wouldn't be necessary to meet it either.

7CarlShulman11y

As I said in my other comment, I would be quite surprised if your individual mathematical and AI contributions reach the levels of the best in their fields, as you are stronger verbally than mathematically, and discuss in more detail what I would find surprising and not there. I recently talked to Drexler about nanotechnology in Oxford. Nanotechnology is * Way behind Drexler's schedule, and even accounting for there being far less funding and focused research than he expected, the timeline skeptics get significant vindication * Was said by the NAS panel to be possible, with no decisive physical or chemical arguments against (and discussion of some uncertainties which would not much change the overall picture, in any case), and arguments against tend to be or turn into timeline skepticism and skepticism about the utility of research * Has not been the subject of a more detailed report or expert judgment test than the National Academy of Sciences one (which said it's possible) because Drexler was not on the ball and never tried. He is currently working with the FHI to get a panel of independent eminent physicists and chemists to work it over, and expects them to be convinced.

1Eliezer Yudkowsky11y

Also, while it seems to me that Michael should have said this about many people, I have not actually heard him say this about many people, to me, except Alyssa Vance.

4Vladimir_Nesov11y

This seems to be usually accounted for by value of information, you should do some unproven things primarily in order to figure out if something like that is possible (or why not, in more detail), before you know it to be possible. If something does turn out to be possible, you just keep on doing it, so that the primary motivation changes without the activity itself changing. (One characteristic of doing something for its value of information as opposed to its expected utility seems to be the expectation of having to drop it when it's not working out. If something has high expected utility a priori, continuing to do it despite it not working won't be as damaging (a priori), even though there is no reason to act this way.)

0ESRogs11y

Not sure I understood this -- are you saying that the expected damage caused by continuing to do it despite it not working is less just because the probability that it won't work is less?

4roystgnr11y

This is astonishingly good evidence that MIRI's efforts will not be wasted via redundancy, de facto "failure" only because someone else will independently succeed first. But it's actually (very weak) evidence against the proposition that MIRI's efforts will not be wasted because you've overestimated the problem, and it isn't evidence either way concerning the proposition that you haven't overestimated the problem but nobody will succeed at solving it.

3Eliezer Yudkowsky11y

you're asking about the probability of having some technical people get together and solve basic research problems. I don't see why anyone else should expect to know more about that than workshop MIRI participants. Besides backward reasoning from the importance of a good result (which ordinarily operates through implying already-well-tugged ropes) is there any reason why you should be more skeptical of this than any other piece of basic research on an important problem?

[-]roystgnr11y110

I'm concerned about the probability of having some technical people get together and solve some incredibly deep research problems before some perhaps-slightly-less-technical people plough ahead and get practical results without the benefit of that research. I'm skeptical that we'll see FAI before UFAI for the same reason I'm skeptical that we'll see a Navier-Stokes existence proof before a macroscale DNS solution, I'm skeptical that we'll prove P!=NP or even find a provably secure encryption scheme before making the world's economy dependent on unproven schemes, etc.

Even some of the important subgoals of FAI, being worked on with far more resources than MIRI has yet, are barely showing on the radar. IIRC someone only recently produced a provably correct C compiler (and in the process exposed a bunch of bugs in the industry standard compilers) - wouldn't we feel foolish if a provably FAI human-readable code turned UF simply because a bug was automatically introduced in the compilation? Or if a cosmic ray or slightly-out-of-tolerance manufacturing defect affected one of the processors? Fault-tolerant MPI is still leading-edge research, because although we've never needed it befor... (read more)

1jsteinhardt11y

Nitpick, but finding a provably secure encryption scheme is harder than proving P!=NP, since if P=NP then no secure encryption scheme can exist.

0Kawoomba11y

What? Why? Just because RSA would be broken? Shor's algorithm would also do so, even in a proven P!=NP world. There may be other substitutes for RSA, using different complexity classes. There are other approaches altogether. Not to mention one-time pads.

3gwern11y

As I understand it, if P=NP in a practical sense, then almost all cryptography is destroyed as P=NP destroys one-way functions & secure hashes in general. So RSA goes down, many quantum-proof systems go down, and so on and so forth, and you're left with basically just http://en.wikipedia.org/wiki/Information-theoretic_security http://www.karlin.mff.cuni.cz/~krajicek/ri5svetu.pdf discusses some of this. And life was so happy with just one-time pads?

1Kawoomba11y

Really, if P=NP, then encoding your messages would be quite low on the priority list ... however we're not debating the practical impact here, but that "finding a provably secure encryption scheme is harder than proving P!=NP", which was raised as a nitpick, and is clearly not the case. Happiness or unhappiness of life with one-time pads notwithstanding.

1Shmi11y

Suppose some new rich sponsor wanted to donate a lot to MIRI, subject to an independent outside group of experts evaluating the merits of some of its core claims, like that AGI is a near-term (under 100 years) x-risk and that MIRI has non-negligible odds (say, a few percent or more) of mitigating it. Who would you suggest s/he would engage for review?

4CarlShulman11y

FHI sent a survey the top 100 most cited authors in AI and got a response rate of ~1/3, and the median estimates backed this (although this needs to be checked for response bias). Results will be published in September at PT-AI. I.e. a probability of a few percent that there is AI risk, MIRI solves it, and otherwise it wouldn't have been solved and existential catastrophe would have resulted? That would not happen with non-gerrymandered criteria for the expert group. But if a credible such group did deliver that result believably, then one could go to Gates or Buffett (who has spent hundreds of millions on nuclear risk efforts with much lower probability of averting nuclear war) or national governments and get billions in funding. All the work in that scenario is coming from the independent panel concluding the thing is many orders of magnitude better than almost any alternative use of spending, way past the threshold for funding. The rich guy who says he would donate based on it is an irrelevancy in the hypo.

4Eliezer Yudkowsky11y

Damned if I know. Oddly enough, anyone chooses to spend a bunch of their life becoming an expert on these issues tends to be sympathetic to the claims, and most random others tend to make up crap on the spot and stick with it. If they could manage to pay Peter Norvig enough money to spend a lot of time working through these issues I'd be pretty optimistic, but Peter Norvig works for Google and would be hard to pay sufficiently.

1drethelin11y

Do you guys deliberately go out of your way to evangeliz to Jaan tallin and thiel or is that source of funds a lucky break?

[-]lukeprog11y120

I agree with Eliezer that the main difficulty is in getting top-quality, relatively rational people to spend hundreds of hours being educated, working through the arguments, etc.

Jaan has done a surprising amount of that and also read most or all of the Sequences. Thiel has not yet decided to put in that kind of time.

Here's a list of people I'd want on that committee if they were willing to put in hundreds of hours catching up and working through the arguments with us: Scott Aaronson, Peter Norvig, Stuart Russell, Michael Nielsen.

I'd probably be able to add lots more names to that list if I could afford to spend more time becoming familiar with the epistemic standards and philosophical sophistication of more high-status CS people. I would trust Carl Shulman, Paul Christiano, Jacob Steinhardt, and a short list of others to add to my list with relatively little personal double-checking from me.

But yeah; the main problem seems to me that I don't know how to get 400 hours of Andrew Ng's time.

Although with Ng in particular it might not take 400 hours. When Louie and I met with him in Nov. '12 he seemed to think AI was almost certainly a century or more away, but by May '13 (after gettin... (read more)

4drethelin11y

This is a great answer but actually a little tangential to my question, sorry for being vague. Mine was actually about the part of shminux's proposal that involved finding potential mega donors. Relatedly, how much convincing do you think it would take to get Tallinn or thiel to increase their donations by an order of magnitude, something they could easily afford? This seems like a relatively high leverage plan if you can swing it. With x million dollars you can afford to actually pay to hire people like google can, if on a much smaller scale.

1wedrifid11y

(Or if it were picked up elsewhere and MIRI is merely overtaken. Dying isn't necessary.)

[-]CarlShulman11y320

People differ in their estimates within MIRI. Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.

ETA: To be clear, I think that MIRI's existence, relative to the counterfactual in which it never existed, has been a good thing and reduced x-risk in my opinion, despi... (read more)

[-]lukeprog11y160

I'm eager to see Eliezer's planned reply to your "ETA2", but in the meantime, here are a few of my own thoughts on this...

My guess is that movement-building and learning are still the best things to do right now for AI risk reduction. CEA, CFAR, and GiveWell are doing good movement-building, though the GiveWell crowd tends to be less interested in x-risk mitigation. GiveWell is doing a large share of the EA-learning, and might eventually (via GiveWell labs) do some of the x-risk learning (right now GiveWell has a lot of catching up to do on x-risk).

The largest share of the "explicit" x-risk learning is happening at or near FHI & MIRI, including e.g. Christiano. Lots of "implicit" x-risk learning is happening at e.g. NASA where it's not clear that EA-sourced funding can have much marginal effect relative to the effect it could have on tiny organizations like MIRI and FHI.

My impression, which could be wrong, is that GiveWell's ability to hire more researchers is not funding-limited but rather limited by management's preference to offer lower salaries than necessary to ensure cause loyalty. (I would prefer GiveWell raise salaries and grow its research... (read more)

4Paul Crowley11y

Your reasons for focusing on math research at MIRI seem sound, but I take it you've noticed the warning sign of finding that what you already decided to do turns out to be a good idea for different reasons than you originally thought?

2lukeprog11y

Yes, though these reasons are pretty similar to the reasons that made me switch positions on strategy back when I thought a focus on strategic research would be best for MIRI in 2013.

3CarlShulman11y

It's clearly not funding-limited, as they have plenty of funding for operations. I'm less confident of the salaries explanation as to their difficulties hiring: it is quite plausible as a significant factor. And the "give more money via GiveWell's top charities to accelerate their approach to their limits of growth" rationale gets worse every year, as the contribution of a dollar to their money moved falls by half, and the number of potential doublings remaining falls. So I would not see direct donations to GiveWell or its top charities as competitive, although other interventions that bolstered it more effectively could. Two plausible examples: 80,000 hours might deliver a number of good new hires to GiveWell, or the Effective Fundraising experimental project, inspired by discussion like this 80,000 hours blog post, may succeed in efficiently mobilizing non-EA funds to support GiveWell's top charities. Yes. One of the biggest virtues of a large "fund for the future," IMHO, is that it would make it easier to start up new projects in the field separate from existing organizations if they could meet the (transparently announced) standards of the fund, with the process as transparent and productive of information as practicable, GiveWell style. And it could serve those who think the existing organizations in the field are deficient in some remediable way (rather than having some general objection to all work in the area). Good points that I'm largely on board with, qualitatively, although one needs to make more of a case to show they meet the bar of beating existing alternatives, or waiting for others to enter the field and do things better. Also, I should mention the Global Catastrophic Risks Institute, even if no one at the EA events in England mentioned it while I was there.

0lukeprog11y

That does sound good. Is there any ongoing progress on figuring out what those transparently announced standards could be, and how one might set up such a DAF? Are there such standards in place for the one in the UK?

0CarlShulman11y

The one in the UK is mainly functioning as a short term DAF along the lines of "direct the money as you intend, with trust of GWWC as a backstop" which is fine if you don't want to delay disbursement until after you die. Not yet, so far mainly discussions, e.g. with Paul, Rob Wiblin, Nick Beckstead, et al. I expect more from CEA on this (not wholly independently of my own actions).

3Pablo8y

Eliezer wrote this in 1999:

8gjm8y

Hasn't Eliezer said, on every occasion since the beginning of LW when the opportunity has arisen, that Eliezer-in-1999 was disastrously wrong and confused about lots of important things? (I don't know whether present-day-Eliezer thinks 18-years-ago-Eliezer was wrong about this particular thing, but I would be cautious about taking things he said that long ago as strongly indicative of his present opinions.)

2Pablo8y

Yes, I am aware that this is what Eliezer has said, and I wasn't implying that those early statements reflect Eliezer's current thinking. There is a clear difference between "Eliezer believed this in the past, so he must believe it at present" and "Eliezer made some wrong predictions in the past, so we must treat his current predictions with caution". Eliezer is entitled to ask his readers not to assume that his past beliefs reflect those of his present self, but he is not entitled to ask them not to hold him responsible for having once said stuff that some may think was ill-judged.

2gjm8y

I hadn't realised anyone was arguing for not treating Eliezer's current predictions with caution. I can't imagine why anyone wouldn't treat anyone's predictions with caution in this field.

2Pablo8y

My point is that these early pronouncements are (limited) evidence that we should treat Eliezer's predictions with more caution than we would otherwise.

0gjm8y

OK, I guess. I have to say that the main impression I'm getting from this exchange is that you wanted to say "boo Eliezer"; it seems like if you wanted to make an actual usefully constructive point you'd have been somewhat more explicit in your original comment. ("Eliezer wrote this in 1999: [...]. I know that Eliezer has since repudiated a lot of his opinions and thought processes of that period, but if his opinions were that badly wrong in 1999 then we shouldn't take them too seriously now either." or whatever.) I will vigorously defend anyone's right to say "boo Eliezer" or "yay Eliezer", but don't have much optimism about getting a useful outcome from a conversation that begins that way, and will accordingly drop it now.

0Pablo8y

Thanks for the feedback. I agree that a comment worded in the manner you suggest would have communicated my point more effectively.

4Houshalter8y

Yudkowsky has changed his views a lot over the last 18 years though. A lot of his earlier writing is extremely optimistic about AI and it's timeline.

0bogus8y

Well, a nanowar is just a conflict on a very, very small scale - like many orders of magnitude less serious than your average barfight. Perhaps we had one before 2015 and nobody noticed! Now we just have to wait until 2020 for the seed AI to transcend.

3Peter Wildeford11y

7CarlShulman11y

Yes, and his posts about intelligence explosion on Overcoming Bias, this, this, and unfortunately comments scattered around Less Wrong or various interveiws that would take some work to find and gather in one place. Nick Bostrom's book on superintelligence probably provides the best single treatment now, having synthesized most pre-existing work. It is moving towards publication, but you might ask him if you can read the draft.

2lukeprog11y

Most pre-existing work? I would've said "having synthesized ~5% of pre-existing work related to superintelligence strategy that has been done at or near MIRI and FHI."

2lukeprog11y

Good news! Having now read the near-finished draft, my new guess is that Bostrom's book synthesizes more like 20% of pre-existing work related to superintelligence strategy that has been done at or near MIRI and FHI. A lot has been added to the book since April. It's really killing me that the book won't be published until mid 2014.

0CarlShulman11y

One can delve indefinitely into any subtopic, but with diminishing returns. Do you think that it doesn't address most of the higher-level topic areas, if not all of the issues arising therein?

0lukeprog11y

No, I think it does a pretty good job of that. I'm not arguing that the book should be different than it is. I'm just saying that it definitely doesn't synthesize "most" pre-existing work.

[-]ColonelMustard11y60

I am thinking of writing a discussion thread to propose MIRI make it a priority to create a (video/pamphlet/blog post), tailored to intelligent non-rationalists and with as little jargon as possible (e.g. no terms like Kolmogorov complexity), to explain the dangers of UFAI. Please upvote this comment if you think LW is better with such a post, because I have zero karma.

[-]Rain12y20

I give them about 10 percent probability, and statements I've read from Eliezer and Luke cause me to believe they're below medium in self-assessment nowadays, or their definition of "medium" may be lower than you think. I support them to increase the chance.

That said, low chances of success aren't good for recruiting or motivation, and maximizing the probability requires as much effort and skill as possible to obtain.

9Peter Wildeford12y

1.) What do you base the 10% estimate on? 2.) Eliezer is very much against the idea of supporting MIRI based on a "low probability of really high impact" argument. What do you think?

[-]Larks11y130

Eliezer is very much against the idea of supporting MIRI based on a "low probability of really high impact" argument.

I hate to put words in his mouth, but I think

he means 0.0001% chance, not 10% chance. 10% is well within the range of probabilities humans can reason about (to the extent that humans can reason about any probabilities).
Eliezer thinks the case for MIRI does not depend on very small chances, and furthermore, is sceptical that these arguments are used in practice by Xrisk organisations, etc. He wouldn't necessary turn away someone's money who said "I'm donating because of a 10^-10 chance." (though equally he might for PR/paternalistic reasons)

0Peter Wildeford11y

Where does this 10% probability come from?

3Rain11y

Anchoring from my butt-number?

1lukeprog11y

I believe the correct term is "ass-pull number." :)

[-]Rain11y110

1.) What do you base the 10% estimate on?

I base it on everything I've read and seen on technology, human nature, historical uses of power, trends in tech capabilities, the effects of intelligence, MIRI's mission, team, and focus, and the greater realm of philanthropic endeavors.

If you mean, 'you pulled that number out of your butt, and therefore I call you on it,' then I'll have to admit defeat due to inability to break it down quantitatively. Sorry.

2.) Eliezer is very much against the idea of supporting MIRI based on a "low probability of really high impact" argument. What do you think?

I think that's taken out of context. The way I understand it, he means superintelligence will have a really high impact regardless (near 100% probability), and is therefore a 'lever point' which can have a higher probability of being impacted by anyone paying attention to it, and since MIRI is one of very few groups paying attention, they have a medium probability of being such an impactor.

2Peter Wildeford11y

Yeah. On one hand, I think there is something to be said about needing to make these fast and loose estimates and that there's some basis for them. But on the other hand, I think one needs to recognize just how fast and loose they are. I think our error bars on MIRI's chance of success is really high. ~ Let me put that in premise-conclusion form: P1: Superintelligence will, with probability greater than 99.999%, dramatically impact the future. P2: One can change how superintelligence will unfold by working on superintelligence. C3: Therefore from P1 and P2, working on superintelligence will dramatically impact the future. P4: MIRI is one of the only groups working on superintelligence. C5: Therefore from C3 and P4, MIRI will dramatically impact the future. Do you think that's right? If so, I think P2 could be false, but I'll accept it for the sake of argument. The real problem is, I think, C5 is a fallacy. It either assumes that any work in the domain will affect how superintelligence unfolds in a controlled way (which seems false) or that MIRI's work will have impact (which seems unproven).

9Nornagest11y

P1 is almost certainly an overestimate: independent of everything else, there's almost certainly a greater than 0.001% chance that a civilization-ending event will occur before anyone gets around to building a superintelligence. The potential importance of AI research by way of this chain of logic wouldn't be lowered too much if you used 80 or 90%, though.

0Rain11y

I'm not sure which fallacy you're invoking, but saying (to paraphrase), 'superintelligence is likely difficult to aim' and 'MIRI's work may not have an impact' are certainly possible, and already contribute to my estimates.

0Peter Wildeford11y

I think a fair amount of people argue that because a cause is important, anyone working on that cause must be doing important work.

3Rain11y

The method is even more important (practice vs. perfect practice, philanthropy vs. givewell). I believe in the mission, not MIRI per se. If Eliezer decided that magic was the best way to achieve FAI and started searching for the right wand and hand gestures rather than math and decision theory, I would look elsewhere.

2gjm11y

Could you clarify your definition of success? Your PredictionBook link says you reckon 10% probability of humanity still being around in ~100 years, but that's not the same thing as MIRI succeeding. Superhuman AI might turn out to be beyond human capability, so we could survive without MIRI achieving anything. Superhuman UFAI might be feasible, and MIRI might successfully stop it happening (in which case I'd say they succeeded), but FAI might be just too hard or too weak and we might then get wiped out by something else. (I agree that that seems low probability.)

2Rain11y

From MIRI's mission statement: "the creation of smarter-than-human intelligence has a positive impact." I see smarter-than-human intelligence as required to overcome the combined threat of existential risks in the long run.

6Pablo11y

The full sentence reads: "MIRI exists to ensure that the creation of smarter-than-human intelligence has a positive impact." (emphasis added) Clearly, if smarter-than-human intelligence ends up having a positive impact independently (or in spite of) MIRI's efforts, that would count as a success only in a Pickwickian sort of sense. To succeed in the sense obviously intended by the authors of the mission statement, MIRI would have to be at least partially causally implicated in the process leading to the creation of FAI. So the question remains: on what grounds do you believe that, if smarter-than-human intelligence ends up having a positive impact, this will be necessarily at least partly due to MIRI's efforts? I find that view implausible, and instead agree with Carl Shulman that "the impact of MIRI in particular has to be far smaller subset of the expected impact of the cause as a whole," for the reasons he mentions.

4Rain11y

I subscribe to the view that AGI is bad by default, and don't see anyone else working on the friendliness problem.

2Omid11y

So we're probably doomed then?

3CarlShulman11y

Surveys at the FHI's Winter Intelligence conference, and sent out to AI experts showed median estimates of existential risk from AI as less likely than good outcomes from AI. I would say we are probably not doomed by that channel, as would, e.g. Paul Christiano, although I would say (as do the surveys, including of neutral AI experts) the risk is significant.

2JonahS11y

I agree with this, but I would be interested in knowing your own reasons for saying that we're probably not doomed via AI risk. I spelled out some reasons for my own belief here.

0Yosarian211y

Personally, I don't think that there is that high of a chance that we're doomed via AI risk because I think that the odds of "AI goes foom" are significantly lower then the MIRI people do. I think it's somewhat more likely that subhuman-level GAI's and vaugly human level AI's increase the rate of AI research somewhat, but not to the dramatic levels the EY thinks. With a slower paced rate of AI development, we're probably not dealing with a singleton AI, we have time to work to perfect friendly AI research while getting assistance from roughly human level AI's, while slowly developing the slightly smarter AI's, and while improving friendly-AI software with every step. I don't think the "AI goes foom" scenario is impossible, but I don't put it at a high likelihood, maybe 10%-20%. I just don't think that it's all that likely that a human level GAI (say, one with the equivilent of IQ-90 to IQ-100) is able to rapidly turn itself into a IQ-300 GAI; if we can't do it that quickly, then a AI of similar intelligence shouldn't be able to do much better. And in the slower AI takeoff scenario I think is more likely, there is still some AI risk, but research in friendly AI we do now at this early stage is likely to be mostly made obsolete by research we do at that stage, and we're more likely to adapt and tweak solutions rather then having to have a totally flawless solution ready the day before someone develops the first moderately useful-level GAI. So i would probably put the total odds of MIRI directly having a big impact to be maybe around 5%. Which, considering the stakes involved, is still very significant and worth the cost, and there is also a significant change that their work has other useful spinoffs even if it turns out to not be necessary for the reason intended.

0Rain11y

I'm pessimistic and depressed.

[-]Shmi11y-10

How does MIRI Know it Has a Medium Probability of Success?

it doesn't know of course. The uncertainty in any probability estimate is bound to be huge, which is reflected in the variety of estimates and opinions event within MIRI. I hope to see some day a thoughtfully constructed reference class with a large enough number of members to be worth drawing conclusions from.

[+]Locaha11y-140

Moderation Log

146Comments