= 27d567bc2d48d9f40e9ccc20ea370b15)
Unknowns comments on The Importance of Self-Doubt - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (726)
I see. So it's not that any one of these statements is a forbidden premise, but that their combination leads to a forbidden conclusion. Would you agree with the previous sentence?
BTW, nobody please vote down the parent below -2, that will make it invisible. Also it doesn't particularly deserve downvoting IMO.
I wouldn't put it in terms of forbidden premises or forbidden conclusions.
But if each of these statements has a 90% of being true, and if they are assumed to be independent (which admittedly won't be exactly true), then the probability that all three are true would be only about 70%, which is not an extremely high degree of confidence; more like saying, "This is my opinion but I could easily be wrong."
Personally I don't think 1) or 3), taken in a strict way, could reasonably be said to have more than a 20% chance of being true. I do think a probability of 90% is a fairly reasonable assignment for 2), because most people are not going to bother about Friendliness. Accounting for the fact that these are not totally independent, I don't consider a probability assignment of more than 5% for the conjunction to be reasonable. However, since there are other points of view, I could accept that someone might assign the conjunction a 70% chance in accordance with the previous paragraph, without being crazy. But if you assign a probability much more than that I would have to withdraw this.
If the statements are weakened as Carl Shulman suggests, then even the conjunction could reasonably be given a much higher probability.
Also, as long as it is admitted that the probability is not high, you could still say that the possibility needs to be taken seriously because you are talking about the possible (if yet improbable) destruction of the world.
I certainly do not assign a probability as high as 70% to the conjunction of all three of those statements.
And in case it wasn't clear, the problem I was trying to point out was simply with having forbidden conclusions - not forbidden by observation per se, but forbidden by forbidden psychology - and using that to make deductions about empirical premises that ought simply to be evaluated by themselves.
I s'pose I might be crazy, but you all are putting your craziness right up front. You can't extract milk from a stone!
That's good to know. I hope multifoliaterose reads this comment, as he seemed to think that you would assign a very high probability to the conjunction (and it's true that you've sometimes given that impression by your way of talking.)
Also, I didn't think he was necessarily setting up forbidden conclusions, since he did add some qualifications allowing that in some circumstances it could be justified to hold such opinions.
Just curious (and not being 100% serious here): Would you have any concerns about the following argument (and I am not saying I accept it)?
That doesn't seem scary to me at all. I still know that there is at least one of me that I can consider 'real'. I will continue to act as if I am one of the instances that I consider me/important. I've lost no existence whatsoever.
You can see Eliezer's position on the Simulation Argument here.
To be quite clear about which of Unknowns' points I object, my main objection is to the point:
where 'I' is replaced by "Eliezer." I assign a probability of less than 10^(-9) to you succeeding in playing a critical role on the Friendly AI project that you're working on. (Maybe even much less than that - I would have to spend some time calibrating my estimate to make a judgment on precisely how low a probability I assign to the proposition.)
My impression is that you've greatly underestimated the difficulty of building a Friendly AI.
I wish the laws of argument permitted me to declare that you had blown yourself up at this point, and that I could take my toys and go home. Alas, arguments are not won on a points system.
Out of weary curiosity, what is it that you think you know about Friendly AI that I don't?
And has it occurred to you that if I have different non-crazy beliefs about Friendly AI then my final conclusions might not be so crazy either, no matter what patterns they match in your craziness recognition systems?
On the other hand, assuming he knows what it means to assign something a 10^-9 probability, it sounds like he's offering you a bet at 1000000000:1 odds in your favour. It's a good deal, you should take it.
Indeed. I do not know how many people are actively involved in FAI research, but i would guess that it is only in the the dozens to hundreds. Given the small pool of competition, it seems likely that at some point Eliezer will, or already has, made a unique contribution to the field. Get Multi to put some money on it, offer him 1 cent if you do not make a useful contribution in the next 50 years, and if you do, he can pay you 10 million dollars.
I agree it's kind of ironic that multi has such an overconfident probability assignment right after criticizing you for being overconfident. I was quite disappointed with his response here.
Why does my probability estimate look overconfident?
One could offer many crude back-of-envelope probability calculations. Here's one: let's say there's
That seems conservative to me and implies at least a 1 in 10^4 chance. Obviously there's lots of room for quibbling here, but it's hard for me to see how such quibbling could account for five orders of magnitude. And even if post-quibbling you think you have a better model that does imply 1 in 10^9, you only need to put little probability mass on my model or models like it for them to dominate the calculation. (E.g., a 9 in 10 chance of a 1 in 10^9 chance plus a 1 in 10 chance of a 1 in 10^4 chance is close to a 1 in 10^5 chance.)
I don't find these remarks compelling. I feel similar remarks could be used to justify nearly anything. Of course, I owe you an explanation. One will follow later on.
Unless you've actually calculated the probability mathematically, a probability of one in a billion for a natural language claim that a significant number of people accept as likely true is always overconfident. Even Eliezer said that he couldn't assign a probability as low as one in a billion for the claim "God exists" (although Michael Vassar criticized him for this, showing himself to be even more overconfident than Eliezer.)
I'm afraid I have to take severe exception to this statement.
You give the human species far too much credit if you think that our mere ability to dream up a hypothesis automatically raises its probability above some uniform lower bound.
I am aware of your disagreement, for example as expressed by the absurd claims here. Yes, my basic idea is, unlike you, to give some credit to the human species. I think there's a limit on how much you can disagree with other human beings-- unless you're claiming to be something superhuman.
Did you see the link to this comment thread? I would like to see your response to the discussion there.
The product of two probabilities above your threshold-for-overconfidence can be below your threshold-for-overconfidence. Have you at least thought this through before?
For instance, the claim "there is a God" is not that much less spectacular than the claim "there is a God, and he's going to make the next 1000 times you flip a coin turn up heads." If one-in-a-billion is a lower bound for the probability that God exists, then one-in-a-billion-squared is a generous lower bound for the probability that the next 1000 times you flip a coin will turn up heads. (One-in-a-billion-squared is about 2-to-the-sixty). You're OK with that?
Yes. As long as you think of some not-too-complicated scenario where the one would lead to the other, that's perfectly reasonable. For example, God might exist and decide to prove it to you by effecting that prediction. I certainly agree this has a probability of at least one in a billion squared. In fact, suppose you actually get heads the next 60 times you flip a coin, even though you are choosing different coins, it is on different days, and so on. By that point you will be quite convinced that the heads are not independent, and that there is quite a good chance that you will get 1000 heads in a row.
It would be different of course if you picked a random series of heads and tails: in that case you still might say that there is at least that probability that someone else will do it (because God might make that happen), but you surely cannot say that it had that probability before you picked the random series.
This is related to what I said in the torture discussion, namely that explicitly describing a scenario automatically makes it far more probable to actually happen than it was before you described it. So it isn't a problem if the probability of 1000 heads in a row is more likely than 1 in 2-to-1000. Any series you can mention would be more likely than that, once you have mentioned it.
Also, note that there isn't a problem if the 1000 heads in a row is lower than one in a billion, because when I made the general claim, I said "a claim that significant number of people accept as likely true," and no one expects to get the 1000 heads.
My estimate does come some effort at calibration, although there's certainly more that I could do. Maybe I should have qualified my statement by saying "this estimate may be a gross overestimate or a gross underestimate."
In any case, I was not being disingenuous or flippant. I have carefully considered the question of how likely it is that Eliezer will be able to play a crucial role in a FAI project if he continues to exhibit a strategy qualitatively similar to his current one and my main objection to SIAI's strategy is that I think it extremely unlikely that Eliezer will be able to have an impact if he proceeds as he has up until this point.
I will be detailing why I don't think that Eliezer's present strategy toward working toward an FAI is a fruitful one in a later top level post.
It sounds, then, like you're averaging probabilities geometrically rather than arithmetically. This is bad!
I don't understand this remark.
What probability do you assign to your succeeding in playing a critical role on the Friendly AI project that you're working on? I can engage with a specific number. I don't know if your object is that my estimate is off by a single of order of magnitude or by many orders of magnitude.
I should clarify that my comment applies equally to AGI.
I think that I know the scientific community better than you, and have confidence that if creating an AGI was as easy as you seem to think it is (how easy I don't know because you didn't give a number) then there would be people in the scientific community who would be working on AGI.
Yes, this possibility has certainly occurred to me. I just don't know what your different non-crazy beliefs might be.
Why do you think that AGI research is so uncommon within academia if it's so easy to create an AGI?
This question sounds disingenuous to me. There is a large gap between "10^-9 chance of Eliezer accomplishing it" and "so easy for the average machine learning PhD." Whatever else you think about him, he's proved himself to be at least one or two standard deviations above the average PhD in ability to get things done, and some dimension of rationality/intelligence/smartness.
My remark was genuine. Two points:
I think that the chance that any group of the size of SIAI will develop AGI over the next 50 years is quite small.
Eliezer has not proved himself to be at the same level of the average machine learning PhD at getting things done. As far as I know he has no experience with narrow AI research. I see familiarity with narrow AI as a prerequisite to AGI research.
He actually stated that himself several times.
Yes, ok, this does not mean his intellectual power isn't on par, but his ability to function in an academic environment.
Well...
Most things can be studied through the use of textbooks. Some familiarity with AI is certainly helpful, but it seems that most AI-related knowledge is not on the track to FAI (and most current AGI stuff is nonsense or even madness).
The reason that I see familiarity with narrow AI as a prerequisite to AGI research is to get a sense of the difficulties present in designing machines to complete certain mundane tasks. My thinking is the same as that of Scott Aaronson in his The Singularity Is Far posting: "there are vastly easier prerequisite questions that we already don’t know how to answer."
Um, and there aren't?
Give some examples. There may be a few people in the scientific community working on AGI, but my understanding is that basically everybody is doing narrow AI.
What is currently called the AGI field will probably bear no fruit, perhaps except for the end-game when it borrows then-sufficiently powerful tools from more productive areas of research (and destroys the world). "Narrow AI" develops the tools that could eventually allow the construction of random-preference AGI.
The folks here, for a start.
Why are people boggling at the 1-in-a-billion figure? You think it's not plausible that there are three independent 1-in-a-thousand events that would have to go right for EY to "play a critical role in Friendly AI success"? Not plausible that there are 9 1-in-10 events that would have to go right? Don't I keep hearing "shut up and multiply" around here?
Edit: Explain to me what's going on. I say that it seems to me that events A, B are likely to occur with probability P(A), P(B). You are allowed to object that I must have made a mistake, because P(A) times P(B) seems too small to you? (That is leaving aside the idea that 10-to-the-minus-nine counts as one of these too-small-to-be-believed numbers, which is seriously making me physiologically angry, ha-ha.)
The 1-in-a-billion follows not from it being plausible that there are three such events, but from it being virtually certain. Models without such events will end up dominating the final probability. I can easily imagine that if I magically happened upon a very reliable understanding of some factors relevant to future FAI development, the 1 in a billion figure would be the right thing to believe. But I can easily imagine it going the other way, and absent such understanding, I have to use estimates much less extreme than that.
I'm having trouble parsing your comment. Could you clarify?
A billion is not so big a number. Its reciprocal is not so small a number.
Edit: Specifically, what's "it" in "it being virtually certain." And in the second sentence -- models of what, final probability of what?
Edit 2: -1 now that I understand. +1 on the child, namaste. (+1 on the child, but I just disagree about how big one billion is. So what do we do?)
"it being virtually certain that there are three independent 1 in 1000 events required, or nine independent 1 in 10 events required, or something along those lines"
Models of the world that we use to determine how likely it is that Eliezer will play a critical role through a FAI team. Final probability of that happening.
A billion is big compared to the relative probabilities we're rationally entitled to have between models where a series of very improbable successes is required, and models where only a modest series of modestly improbable successes is required.
Yes, this is of course what I had in mind.
Replied to this comment and the other (seeming contradictory) one here.