Eliezer_Yudkowsky comments on Why We Can't Take Expected Value Estimates Literally (Even When They're Unbiased) - Less Wrong

75 18 August 2011 11:34PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Sort By: Best

You are viewing a single comment's thread.

Comment author: 18 August 2011 11:56:58PM 51 points [-]

Quick comment one:

This jumped out instantly when I looked at the charts: Your prior and evidence can't possibly both be correct at the same time. Everywhere the prior has non-negligible density has negligible likelihood. Everywhere that has substantial likelihood has negligible prior density. If you try multiplying the two together to get a compromise probability estimate instead of saying "I notice that I am confused", I would hold this up as a pretty strong example of the real sin that I think this post should be arguing against, namely that of trying to use math too blindly without sanity-checking its meaning.

Quick comment two:

I'm a major fan of Down-To-Earthness as a virtue of rationality, and I have told other SIAI people over and over that I really think they should stop using "small probability of large impact" arguments. I've told cryonics people the same. If you can't argue for a medium probability of a large impact, you shouldn't bother.

Part of my reason for saying this is, indeed, that trying to multiply a large utility interval by a small probability is an argument-stopper, an attempt to shut down further debate, and someone is justified in having a strong prior, when they see an attempt to shut down further debate, that further argument if explored would result in further negative shifts from the perspective of the side trying to shut down the debate.

With that said, any overall scheme of planetary philanthropic planning that doesn't spend ten million dollars annually on Friendly AI is just stupid. It doesn't just fail the Categorical Imperative test of "What if everyone did that?", it fails the Predictable Retrospective Stupidity test of, "Assuming civilization survives, how incredibly stupid will our descendants predictably think we were to do that?"

Of course, I believe this because I think the creation of smarter-than-human intelligence has a (very) large probability of an (extremely) large impact, and that most of the probability mass there is concentrated into AI, and I don't think there's nothing that can be done about that, either.

I would summarize my quick reply by saying,

"I agree that it's a drastic warning sign when your decision process is spending most of its effort trying to achieve unprecedented outcomes of unquantifiable small probability, and that what I consider to be down-to-earth common sense is a great virtue of a rationalist. That said, down-to-earth common-sense says that AI is a screaming emergency at this point in our civilization's development, and I don't consider myself to be multiplying small probabilities by large utility intervals at any point in my strategy."

Comment author: 19 August 2011 03:05:30AM *  24 points [-]

I don't consider myself to be multiplying small probabilities by large utility intervals at any point in my strategy

What about people who do think SIAI's probability of success is small? Perhaps they have different intuitions about how hard FAI is, or don't have enough knowledge to make an object-level judgement so they just apply the absurdity heuristic. Being one of those people, I think it's still an important question whether it's rational to support SIAI given a small estimate of probability of success, even if SIAI itself doesn't want to push this line of inquiry too hard for fear of signaling that their own estimate of probability of success is low.

Comment author: 19 August 2011 12:25:37PM *  7 points [-]

I think the creation of smarter-than-human intelligence has a (very) large probability of an (extremely) large impact, and that most of the probability mass there is concentrated into AI

That's the probability statement in his post. He didn't mention the probability of SIAI's success, and hasn't previously when I've emailed him or asked in public forums, nor has he at any point in time that I've heard. Shortly after I asked, he posted When (Not) To Use Probabilities.

Comment author: 19 August 2011 04:37:19PM 7 points [-]

Yes, I had read that, and perhaps even more apropos (from Shut up and do the impossible!):

You might even be justified in refusing to use probabilities at this point. In all honesty, I really don't know how to estimate the probability of solving an impossible problem that I have gone forth with intent to solve; in a case where I've previously solved some impossible problems, but the particular impossible problem is more difficult than anything I've yet solved, but I plan to work on it longer, etcetera.

People ask me how likely it is that humankind will survive, or how likely it is that anyone can build a Friendly AI, or how likely it is that I can build one. I really don't know how to answer. I'm not being evasive; I don't know how to put a probability estimate on my, or someone else, successfully shutting up and doing the impossible. Is it probability zero because it's impossible? Obviously not. But how likely is it that this problem, like previous ones, will give up its unyielding blankness when I understand it better? It's not truly impossible, I can see that much. But humanly impossible? Impossible to me in particular? I don't know how to guess. I can't even translate my intuitive feeling into a number, because the only intuitive feeling I have is that the "chance" depends heavily on my choices and unknown unknowns: a wildly unstable probability estimate.

But it's not clear whether Eliezer means that he can't even translate his intuitive feeling into a word like "small" or "medium". I thought the comment I was replying to was saying that SIAI had a "medium" chance of success, given:

If you can't argue for a medium probability of a large impact, you shouldn't bother.

and

I don't consider myself to be multiplying small probabilities by large utility intervals at any point in my strategy

But perhaps I misinterpreted? In any case, there's still the question of what is rational for those of us who do think SIAI's chance of success is "small".

Comment author: 19 August 2011 06:02:00PM 2 points [-]

I thought he was taking the "don't bother" approach by not giving a probability estimate or arguing about probabilities.

In any case, there's still the question of what is rational for those of us who do think SIAI's chance of success is "small".

I propose that the rational act is to investigate approaches to greater than human intelligence which would succeed.

Comment author: 21 August 2011 04:44:28AM 2 points [-]

I propose that the rational act is to investigate approaches to greater than human intelligence which would succeed.

This. I'm flabbergasted this isn't pursued further.

Comment author: 19 August 2011 06:21:50PM *  2 points [-]

Sufficiently-Friendly AI can be hard for SIAI-now but easy or medium for non-SIAI-now (someone else now, someone else future, SIAI future). I personally believe this, since SIAI-now is fucked up (and SIAI-future very well will be too). (I won't substantiate that claim here.) Eliezer didn't talk about SIAI specifically. (He probably thinks SIAI will be at least as likely to succeed as anyone else because he thinks he's super awesome, but it can't be assumed he'd assert that with confidence, I think.)

Comment author: 19 August 2011 06:38:35PM 20 points [-]

SIAI-now is fucked up (and SIAI-future very well will be too). (I won't substantiate that claim here.)

Will you substantiate it elsewhere?

Comment author: 19 August 2011 10:25:06PM *  8 points [-]

Second that interest in hearing it substantiated elsewhere.

Comment author: 28 December 2011 08:35:59PM 4 points [-]

Your comments are a cruel reminder that I'm in a world where some of the very best people I know are taken from me.

Comment author: 28 December 2011 08:38:28PM *  2 points [-]

SingInst seems a lot better since I wrote that comment; you and Luke are doing some cool stuff. Around August everything was in a state of disarray and it was unclear if you'd manage to pull through.

Comment author: 19 August 2011 10:38:57PM 15 points [-]

Leaving aside Aumann questions: If people like that think that the Future of Humanity Institute, work on human rationality, or Giving What We Can has a large probability of catalyzing the creation of an effective institution, they should quite plausibly be looking there instead. "I should be doing something I think is at least medium-probably remedying the sheerly stupid situation humanity has gotten itself into with respect to the intelligence explosion" seems like a valuable summary heuristic.

If you can't think of anything medium-probable, using that as an excuse to do nothing is unacceptable. Figure out which of the people trying to address the problem seem most competent and gamble on something interesting happening if you give them more money. Money is the unit of caring and I can't begin to tell you how much things change when you add more money to them. Imagine what the global financial sector would look like if it was funded to the tune of \$600,000/year. You would probably think it wasn't worth scaling up Earth's financial sector.

Comment author: 20 August 2011 05:57:23AM *  8 points [-]

If you can't think of anything medium-probable, using that as an excuse to do nothing is unacceptable.

That's my gut feeling as well, but can we give a theoretical basis for that conclusion, which might also potentially be used to convince people who can't think of anything medium-probable to "do something"?

My current thoughts are

1. I assign some non-zero credence to having an unbounded utility function.
2. Bostrom and Toby's moral parliament idea seems to be the best that we have about how to handle moral uncertainty.
3. If Pascal's wager argument works, and to the extent that I have a faction representing unbounded utility in my moral parliament, I ought to spend a fraction of my resources on Pascal's wager type "opportunities"
4. If Pascal's wager argument works, I should pick the best wager to bet on, which intuitively could well be "push for a positive Singularity"
5. But it's not clear that Pascal's wager argument works or what could be the justification for thinking that "push for a positive Singularity" is the best wager. We also don't have any theory to handle this kind of philosophical uncertainty.
6. Given all this, I still have to choose between "do nothing", "push for positive Singularity", or "investigate Pascal's wager". Is there any way, in this decision problem, to improve upon going with my gut?

Anyway, I understand that you probably have reasons not to engage too deeply with this line of thought, so I'm mostly explaining where I'm currently at, as well as hoping that someone else can offer some ideas.

Comment author: 20 August 2011 11:18:45AM 2 points [-]

Imagine what the global financial sector would look like if it was funded to the tune of \$600,000/year. You would probably think it wasn't worth scaling up Earth's financial sector.

And one might even be right about that.

A better analogy might be if regulation of the global financial sector were funded at 600k/yr.

Comment author: 19 August 2011 10:53:43PM 2 points [-]

Money is the unit of caring and it really is impossible to overstate how much things change when you add money to them.

Can you give an example relevant to the context at hand to illustrate what you have in mind? I don't necessarily disagree, but I presently think that there's a tenable argument that money is seldom the key limiting factor for philanthropic efforts in the developed world.

Comment author: 19 August 2011 11:50:27PM 1 point [-]

BTW, note that I deleted the "impossible to overstate" line on grounds of its being false. It's actually quite possible to overstate the impact of adding money. E.g., "Adding one dollar to this charity will CHANGE THE LAWS OF PHYSICS."

Comment author: 20 August 2011 06:58:40AM 0 points [-]

What sort of key limiting factors do you have in mind that are untouched by money? Every limiting factor I can think of, whether it's lack of infrastructure or corruption or lack of political will in the West, is something that you could spend money on doing something about.

Comment author: 20 August 2011 06:51:02PM *  1 point [-]

If nothing else, historical examples show that huge amounts of money lobbed at a cause can go to waste or do more harm than good (e.g. the Iraq war as a means to improve relations with the middle East).

Eliezer and I were both speaking in vague terms; presumably somebody intelligent, knowledgeable, sophisticated, motivated, energetic & socially/politically astute can levy money toward some positive expected change in a given direction. There remains the question about the conversion factor between money and other goods and how quickly it changes at the margin; it could be negligible in a given instance.

The main limiting factor that I had in mind was human capital: an absence of people who are sufficiently intelligent, knowledgeable, sophisticated, motivated, energetic & socially/politically astute.

I would add that a group of such people would have substantially better than average odds of attracting sufficient funding from some philanthropist; further diminishing the value of donations (on account of fungibility).

Comment author: 19 August 2011 12:50:11AM 12 points [-]

Regarding the graphs, I assumed that they were showing artificial examples so that we could viscerally understand at a glance what the adjustment does, not that this is what the prior and evidence should look like in a real case.

Comment author: 19 August 2011 04:42:59AM 2 points [-]

10 million dollars buys quite a few programmers. SIAI is presently nowhere near that amount of money, and doesn't look likely to be any time soon. When does it make sense to start talking to volunteer programmers? Presumably, when the risk of opening up the project is less than the risk of failing to get it done before someone else does it wrong.

Comment author: 19 August 2011 02:19:51PM 2 points [-]

When is "soon" for these purposes. It seems to me that with continuing support, SIAI will be able to hire as many of the right programmers as we can find and effectively integrate into a research effort. We certainly would hire any such programmers now.

Comment author: 19 August 2011 05:03:29PM 4 points [-]

It seems to me that with continuing support, SIAI will be able to hire as many of the right programmers as we can find and effectively integrate into a research effort.

My main source of uncertainty as to SIAI's value comes from the fact that as far as I can tell nobody has a viable Friendly AI research program.

Comment author: 19 August 2011 06:03:55PM *  14 points [-]

Comment author: 21 August 2011 04:22:15PM 7 points [-]

I have a weird rather mixed reputation among the greater LW community, so if that affects you negatively please pretend that someone with a more solid reputation but without super high karma is asking this question, like Steven Kaas.

Unless you would be much less involved in this potential program than the comment indicates, this seems like an inappropriate request. If people view you negatively due to your posting history, they should absolutely take that information into account in assessing how likely they would be to provide financial support to such a program (assuming that the negative view is based on relevant considerations such as your apparent communication or reasoning skills as demonstrated in your comments).

Comment author: 21 August 2011 05:47:06PM 6 points [-]

Unless you would be much less involved in this potential program than the comment indicates, this seems like an inappropriate request.

I was more interested in Less Wrong's interest in new FAI-focused organizations generally than in anything particularly tied to me.

Comment author: 21 August 2011 10:11:13PM 6 points [-]

Fair enough, but in light of your phrasing in both the original comment ("If I [did the following things]") and your comment immediately following it (quoted below; emphasis added), it certainly appeared to me that you seemed to be describing a significant role for yourself, even though your proposal was general overall.

(Some people, including me, would really like it if a competent and FAI-focused uber-rationalist non-profit existed. I know people who will soon have enough momentum to make this happen. I am significantly more familiar with the specifics of FAI (and of hardcore SingInst-style rationality) than many of those people and almost anyone else in the world, so it'd be necessary that I put a lot of hours into working with those who are higher status than me and better at getting things done but less familiar with technical Friendliness. But I have many other things I could be doing. Hence the question.)

Comment author: 20 August 2011 05:57:06AM 4 points [-]

Sorry, could you say again what exactly you want to do? I mean, what's the output here that the money is paying for; a Friendly AI, a theory that can be used to construct a Friendly AI, or an analysis that purports to say whether or not Friendly AI is "feasible", or what?

Comment author: 21 August 2011 08:51:58AM *  2 points [-]

Money would pay for marginal output, e.g. in the form of increased collaboration, I think, since the best Friendliness-cognizant x-rationalists would likely already be working on similar things.

I was trying to quickly gauge vague interest in a vague notion. I think that my original comment was at roughly the most accurate and honest level of vagueness (i.e. "aimed largely [i.e. primarily] at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty [e.g. how many Von Neumanns, Turings, and/or Aristotles would it take?] of Friendly AI for various (logical) probabilities of Friendliness [e.g. is the algorithm meta-reflective enough to fall into (one of) some imagined Friendliness attractor basin(s)?]"). Value of information regarding difficulty of Friendly-ish AI is high, but research into that question is naturally tied to Friendly AI theory itself. I'm thinking... Goedel machine stability more than ambient decision theory, history of computation more than any kind of validity semantics. To some extent it depends on who plans to actually work on what stuff from the open problems lists. There are many interesting technical threads that people might start pulling on soon, and it's unclear to me to what extent they actually will pull on them or to what extent pulling on them will give us a better sense of the problem.

[Stuff it would take too many paragraphs to explain why it's worth pointing out specifically:] Theory of justification seems to be roughly as developed as theory of computation was before the advent of Leibniz; Leibniz saw a plethora of connections between philosophy, symbolic logic, and engineering and thus developed some correctly thematically centered proto-theory. I'm trying to make a Leibniz, and hopefully SingInst can make a Turing. (Two other roughly analogous historical conceptual advances are natural selection and temperature.)

Comment author: 21 August 2011 10:06:50AM 4 points [-]

Well, my probability that you could or would do anything useful, given money, just dropped straight off a cliff. But perhaps you're just having trouble communicating. That is to say: What the hell are you talking about.

If you're going to ask for money on LW, plain English response, please: What's the output here that the money is paying for; (1) a Friendly AI, (2) a theory that can be used to construct a Friendly AI, or (3) an analysis that purports to say whether or not Friendly AI is "feasible"? Please pick one of the pre-written options; I now doubt your ability to write your response ab initio.

Comment author: 21 August 2011 10:16:06AM *  3 points [-]

That was amusingly written, but probably too harsh. You want people to like you, even if it's only so they say nice things about you.

Comment author: 21 August 2011 10:47:34AM *  4 points [-]

Dude, it's right there: "feasibility and difficulty", in this sentence which I am now repeating for the second time:

aimed largely [i.e. primarily] at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty [e.g. how many Von Neumanns, Turings, and/or Aristotles would it take?] of Friendly AI for various (logical) probabilities of Friendliness [e.g. is the algorithm meta-reflective enough to fall into (one of) some imagined Friendliness attractor basin(s)?]").

(Bold added for emphasis, annotations in [brackets] were in the original.)

The next sentence:

Value of information regarding difficulty of Friendly-ish AI is high, but research into that question is naturally tied to Friendly AI theory itself.

Or if you really need it spelled out for you again and again, the output would primarily be (3) but secondarily (2) as you need some of (2) to do (3).

Because you clearly need things pointed out multiple times, I'll remind you that I put my response in the original comment that you originally responded to, without the later clarifications that I'd put in for apparently no one's benefit:

If I had a viable preliminary Friendly AI research program, aimed largely at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty of Friendly AI for various values of "Friendly" [...]

(Those italics were in the original comment!)

If you're going to ask for money on LW

I wasn't asking for money on Less Wrong! As I said, "I was trying to quickly gauge vague interest in a vague notion." What the hell are you talking about.

I've doubted your ability to read for a long time, but this is pretty bad. The sad thing is you're probably not doing this intentionally.

Comment author: 21 August 2011 11:15:35AM *  18 points [-]

I think the problem here is that your posting style, to be frank, often obscures your point.

In most cases, posts that consist of a to-the-point answer followed by longer explanations use the initial statement to make a concise case. For instance, in this post, my first sentence sums up what I think about the situation and the rest explains that thought in more detail so as to convey a more nuanced impression.

By contrast, when Eliezer asked "What's the output here that the money is paying for," your first sentence was "Money would pay for marginal output, e.g. in the form of increased collaboration, I think, since the best Friendliness-cognizant x-rationalists would likely already be working on similar things." This does not really answer his question, and while you clarify this with your later points, the overall message is garbled.

The fact that your true answer is buried in the middle of a paragraph does not really help things much. Though I can see what you are trying to say, I can't in good and honest conscience describe it as clear. Had you answered, on the other hand, "Money would pay for the technical analysis necessary to determine as well as possible the feasibility and difficulty of FAI..." as your first sentence, I think your post would have been more clear and more likely to be understood.

Comment author: 21 August 2011 11:02:50AM 4 points [-]

To save some time and clarify, this was option 3: an analysis that purports to say whether or not Friendly AI is "feasible".

Comment author: 19 August 2011 06:31:36PM *  2 points [-]

(Some people, including me, would really like it if a competent and FAI-focused uber-rationalist non-profit existed. I know people who will soon have enough momentum to make this happen. I am significantly more familiar with the specifics of FAI (and of hardcore SingInst-style rationality) than many of those people and almost anyone else in the world, so it'd be necessary that I put a lot of hours into working with those who are higher status than me and better at getting things done but less familiar with technical Friendliness. But I have many other things I could be doing. Hence the question.)

Comment author: 19 August 2011 07:40:13PM 3 points [-]

Does "FAI-focused" mean what I called code first? What are your thoughts on that post and its followup? What is this new non-profit planning to do differently from SIAI and why? What are the other things that you could be doing?

Comment author: 19 August 2011 08:01:29PM *  4 points [-]

Incomplete response:

Does "FAI-focused" mean what I called code first?

Jah. Well, at least determining whether or not "code first" is even reasonable, yeah, which is a difficult question in itself and only partially tied in with making direct progress on FAI.

What are your thoughts on that post and its followup?

You seem to have missed Oracle AI? (Eliezer's dismissal of it isn't particularly meaningful.) I agree with your concerns. This is why the main focus would at least initially be determining whether or not "code first" is a plausible approach (difficulty-wise and safety-wise). The value of information on that question is incredibly high and as you've pointed out it has not been sufficiently researched.

What is this new non-profit planning to do differently from SIAI and why?

Basically everything. SingInst is focused on funding a large research program and gaining the prestige necessary to influence (academic) culture and academic and political policy. They're not currently doing any research on Friendly AI, and their political situation is such that I don't expect them to be able to do so effectively for a while, if ever. I will not clarify this. (Actually their research associates are working on FAI-related things, but SingInst doesn't pay them to do that.)

What are the other things that you could be doing?

Learning, mostly. Working with an unnamed group of x-risk-cognizant people that LW hasn't heard of, in a way unrelated to their setting up a non-profit.

Comment author: 22 August 2011 05:16:07PM 6 points [-]

They're not currently doing any research on Friendly AI, and their political situation is such that I don't expect them to be able to do so effectively for a while, if ever.

My understanding is that SIAI recently tried to set up a new in-house research team to do preliminary research into FAI (i.e., not try to build an FAI yet, but just do whatever research that might be eventually helpful to that project). This effort didn't get off the ground, but my understanding again is that it was because the researchers they tried to recruit had various reasons for not joining SIAI at this time. I was one of those they tried to recruit, and while I don't know what the others' reasons were, mine were mostly personal and not related to politics.

You must also know all this, since you were involved in this effort. So I'm confused why you say SIAI won't be doing effective research on FAI due to its "political situation". Did the others not join SIAI because they thought SIAI was in a bad political situation? (This seems unlikely to me.) Or are you referring to the overall lack of qualified, recruitable researchers as a "political situation"? If you are, why do you think this new organization would be able to do better?

(Or did you perhaps not learn the full story, and thought SIAI stopped this effort for political reasons?)

Comment author: 22 August 2011 07:48:39PM 1 point [-]

The answer to your question isn't among your list of possible answers. The recent effort to start an in-house research team was a good attempt and didn't fail for political reasons. I am speaking of other things. However I want to take a few weeks off from discussion of such topics; I seem to have given off the entirely wrong impression and would prefer to start such discussion anew in a better context, e.g. one that better emphasizes cooperation and tentativeness rather than reactionary competition. My apologies.

Comment author: 22 August 2011 08:16:12AM 3 points [-]

Working with an unnamed group of x-risk-cognizant people that LW hasn't heard of, in a way unrelated to their setting up a non-profit.

Could you tell us about them?

Comment author: 19 August 2011 08:28:26PM *  1 point [-]

Well, looking at the post on SIAI finances from a few months back, SIAI's annual revenue is growing at a rate of roughly 100k/year, and thus would take nearly a century to reach 10 million / year. Of course, all sorts of things could change these numbers. Eliezer has stated that he believes that UFAI will happen somewhat sooner than a century.

Since SIAI does seem to have at least some unused budget for programmers now, I emailed a friend who might be a good fit for a research associate to suggest that he apply.

Comment author: 22 August 2011 05:29:58PM 1 point [-]

Even if past values are linear, exponential estimates are probably more clarifying.

Comment author: 20 August 2011 11:10:41AM *  -2 points [-]

It seems to me that with continuing support, SIAI will be able to hire as many of the right programmers as we can find and effectively integrate into a research effort.

ADBOC. SIAI should never be content with any funding level. It should get the resources to hire, bribe or divert people who may otherwise make breakthroughs on UAI, and set them to doing anything else.

Comment author: 27 August 2011 04:04:36PM 0 points [-]

That intuition seems to me to follow from the probably false assumption that if behavior X would, under some circumstances, be utility maximizing, it is also likely to be utility maximizing to fund a non-profit to engage in behavior X. SIAI isn't a "do what seems to us to maximize expected utility" organization because such vague goals don't make for a good organizational culture. Organizing and funding research into FAI and research inputs into FAI, plus doing normal non-profit fund-raising and outreach, that's a feasible non-profit directive.

Comment author: 27 August 2011 11:02:07PM 0 points [-]

It also follows from the assumption that the claims in any comment submitted on August 20, 2011 are true. Yet I do not believe this.

I had, to the best of my ability, considered the specific situation when giving my advice.

Any advice can be dismissed by suggesting it came from a too generalized assumption.

If you thought someone was about to foom an unfriendly AI, you would do something about it, and without waiting to properly update your 501(c) forms.

Comment author: 19 August 2011 12:18:27AM *  2 points [-]

Upvoted.

This jumped out instantly when I looked at the charts: Your prior and evidence can't possibly both be correct at the same time. Everywhere the prior has non-negligible density has negligible likelihood. Everywhere that has substantial likelihood has negligible prior density. If you try multiplying the two together to get a compromise probability estimate instead of saying "I notice that I am confused", I would hold this up as a pretty strong example of the real sin that I think this post should be arguing against, namely that of trying to use math too blindly without sanity-checking its meaning.

(I deleted my response to this following othonormal's comments; see this one for my revised thought here.)

Of course, I believe this because I think the creation of smarter-than-human intelligence has a (very) large probability of an (extremely) large impact, and that most of the probability mass there is concentrated into AI, and I don't think there's nothing that can be done about that, either.

Why do you think that there's something that can be done about it?

Comment author: 19 August 2011 12:48:13AM 17 points [-]

I disagree. It can be rational to shift subjective probabilities by many orders of magnitude in response to very little new information.

What your example looks like is a nearly uniform prior over a very large space- nothing's wrong when we quickly update to believe that yesterday's lottery numbers are 04-15-21-31-36.

But the point where you need to halt, melt, and catch fire is if your prior assigns the vast majority of the probability mass to a small compact region, and then the evidence comes along and lands outside that region. That's the equivalent of starting out 99.99% confident that you know tomorrow's lottery numbers will begin with 01-02-03, and being proven wrong.

Comment author: 19 August 2011 12:57:29AM 5 points [-]

Yes, you're right, I wasn't thinking clearly, thanks for catching me. I think there's something to what I was trying to say, but I need to think about it through more carefully. I find the explanation that you give in your other comment convincing (that the point of the graphs is to clearly illustrate the principle).

Comment author: 26 June 2013 12:32:25AM 0 points [-]

I have told other SIAI people over and over that I really think they should stop using "small probability of large impact" arguments.

I confirm this.