The pattern I see is tribal. You feel a strong commitment to certain points of view, like that old-style A.I. was all wrong, that there is nothing mysterious about consciousness, and that Jewish religion has no plausibility. When people disagree about these topics you lower your opinion about those people, not about the topics. But for people who have agreed with you about key topics and impressed you with their technical ability, you are more willing to take their disagreement seriously as more undermining your opinion on the topic, and less undermining your opinion of them.

blinks

Robin, how should I choose who to trust, if not by seeing their competence displayed in handling other issues? Yes, the algorithm is incestuous, but what's your alternative that you think works better in real life? E.g., your disagreement case studies with Cutler?

Tribal affiliation forces should logically operate to bind me far more strongly to people who think AGI is possible, than to people who think consciousness is non-mysterious. The former are my friends; I go with them to conferences; some of them pay my rent. But I lost respect for Pinker more than Dennett, because of what I thought their respective opinions implied about their general competence.

Robin, you seem to be implying EY's preferred views are as casually come by as a sports affiliation.

We see patterns of belief that on the surface might be explained as rationality or as tribalism. So we need to go beyond the surface, to analyze in more careful detail what each theory implies, so that we can distinguish them in the data. This is holy quest! Or, at least one of my quests ...

Robin, I may have to call fundamental attribution error on this. You disagree with Cutler, and say:

I don't know what David thinks of me, but I accept that he is clearly objectively more expert than I on this topic, given his prestigious position and many more years of focus on the topic. But given the strong usual tendency to give medicine the benefit of the doubt, my impression that David gives medicine this benefit of doubt on other topics, and his inability to point to any concrete supporting evidence, I'm willing to attribute David's more positive assessment here to such wishful thinking, rather than to his superior intuition on this matter. How rational am I?

(I like that last touch, but I'm not giving you any modesty credit for humbly asking the question, only for actual shifts in opinion.)

Anyway, when it comes to your own disagreements, you attribute them to strictly situational factors - like a discount you apply for David's (dispositional) tendency to "give medicine the benefit of the doubt".

Then you look at my disagreements, and attribute them to persistent dispositional tendencies, like tribalism.

Perhaps you can see inside your own head to your oh-so-demanding specific reasons, but not see inside mine?

I really can't back your reading, looking over the cases. Pearl, Thrun, and Aumann are all three of them Bayesians and eminent, masters in their separate ways of my chosen art. In the cases of Pearl and Aumann, I had read their work and been impressed. In Thrun's case he had graciously agreed to present at the Singularity Summit. Pearl I humbly petitioned, and his verdict I accepted in both parts; Thrun I questioned, and came to terms with; Aumann I dismissed. Where is the tribal disposition? I do not see it, but I do see a situational difference between disagreeing with a math book, disagreeing with a math anecdote, and disagreeing with a religious profession.

Eliezer, I do not mean to set my disagreements up as a model of rationality, nor to express much confidence in any particular reading of your disagreements. I mainly mean to call attention to the fact that on the surface both of our disagreements might be read as rationality or as tribalism - we need to dig deeper to discern the difference. Only if you insist that for one of us one of these theories is just not a plausible explanation would I persist in arguing for that plausibility.

Whoa, said I to myself, Steven Pinker is a mysterian?

Well, he had already said as much in How the Mind Works. And also in this conversation with Robert Wright.

There's a big difference between two-way disagreements and one-way disagreements. In two-way disagreements, people interact but are unable to come to agreement. Eliezer's interaction with Nick Bostrom might be an example of this; it's not clear what the time scale was but it sounds like they may have left off, each respecting the other but unable to come to agreement on their question. However it sounds like in the aftermath their positions became very close.

One-way disagreements are where you read or hear that someone said something that seems wrong to you, and you have to decide what to do. Theoretically, and modulo some assumptions that not everyone accepts, Aumann's theorem says that two-way disagreements are impossible for rational, honest truth-seekers. However one-way disagreements are not affected by this result. Clearly you would not want to just accept everything you read or hear that someone said, so one-way disagreements are perfectly valid on theoretical grounds. The practical question is when to disagree.

Eliezer describes a number of cases where experts made claims near to their fields of expertise, but he disagreed with them. Although there are a couple of instances where his initial disagreement was at least partially wrong, in each case he is able to go back and trump the other person by improving their result in a manner they did not foresee. Eliezer FTW.

I don't think most people could apply Eliezer's strategy for one-way disagreements successfully. I don't see it as a particularly useful model. Most people, most of the time will probably do better to believe experts when they make claims in their areas of expertise.

Two-way disagreements are more interesting to me, because they so often violate Aumann's theorem. Robin and Eliezer often seem to dance around certain matters where they are not perhaps 100% in agreement, but I seldom see the issue fully joined. I have to admit that I often veer off when I find myself approaching open disagreement with someone I respect, falling back into cautious bepuzzlement. I am afraid that this may be an evasion of the reality of mutual disagreement.

Hal, I started this conversation with the dispute between A.I. old-timers and singularitarians(?) over plausible future rates of A.I. progress. These sides have had enough back and forth contact for the dispute to approach a disagreement. Also, don't forget we can't forsee to disagree at any point in a conversation.

Eliezer, why doesn't the difficulty of creating this AGI count as a reason to think it won't happen soon?

You've said it's extremely, incredibly difficult. Don't the chances of it happening soon go down the harder it is?

In the video link that komponisto gave, the relevant section of the video starts at 49:00 or so. He doesn't argue there though that consciousness -- he uses the term sentience -- will necessarily remain a mystery but only that this might turn out to be the case. He makes the analogy of trying to think about time before the big bang and that there is some kind of conceptual category-type error going on there that gives it its subjective feeling of being mysterious and unknowable, and states that this may be the case with sentience/consciousness, free-will, etc.

Then if you open the envelope and find that amount of money, roll a die and switch the envelope at that probability.

The probability of the die coming up == f(amount of dollars)? f being the probability function.

It seems to me that Eliezer's disagreements have this common factor: when they relate to one of his already considered opinions, he does not modify his opinion when confronted with disagreement. The case of Nick Bostrom is a slight exception to this, but only very slightly: Eliezer continued to maintain that an Oracle AI would have to be explicitly programmed as Friendly. (And this might very well not be true.)

The disagreements where he modified his opinion had to do with mathematical formulas, for example, he he did not have a previously considered opinion at all; for example he had never considered Thrun's precise example.

This confirms my general position that Eliezer is overconfident: if he has considered a question and come to a conclusion, finding that others, even others whom he respected, disagree with it, this does not modify his opinion regarding the question, as Robin said, but his opinion of the other person.

if he has considered a question and come to a conclusion, finding that others, even others whom he respected, disagree with it, this does not modify his opinion regarding the question, as Robin said, but his opinion of the other person

When there is a disagreement between your opinion and that of another person, we need to carefully seek out evidence to tip the balance.

Problems arise when we start treating the fact that we hold opinions as evidence in favor of those opinions. It is extremely important to maintain the correct hierarchy of reason: no conclusion reached by an argument that takes certain premises for granted can be used in an argument in which those premises are the conclusion.

If you always presume that your every whimsy is correct, people will become tired of arguing with you, and your errors will never be rectified.

I gave insufficient support for my last statement, so I should correct this by illustrating the point in each case:

The math book: Eliezer hadn't seen that particular false claim in the past, so he took it into account sufficiently to try to prove its falsity, rather than just assuming it.

Judea Pearl: a similar case.

Daniel Dennett: Dennett contradicted a previous opinion, so Eliezer was surprised and wanted to know the reasons. But Eliezer did not indicate any shift whatsoever in his own opinion. He does not indicate, for example, that he now thinks that there is a 40% chance that there will not be human level AI in the next hundred years, or anything like this. As far as we can tell, he is still convinced that there is a high chance that it is definitely coming soon. He would like to know Dennett's reasons, but only so as to see where Dennett went wrong.

Pinker: again, this contradicts a previous opinion of Eliezer, so he does not shift his opinion. The particular way in which he changes his opinion about Pinker, relative to how he changed his opinion about Dennett, is not really relevant. In both cases, Eliezer's opinions about the matter at hand do not change.

Aumann: again, Eliezer refuses to change his opinion in the slightest, not even assigning a 0.001% chance that Aumann is right and he is wrong.

Many-Worlds: Eliezer has a determined opinion on this matter, and so it does not matter to him how expert some physicist might be in physics who disagrees with him.

Nick Bostrom: Here Eliezer came close to modifying his opinion, but to the degree that he continued to claim that it was necessary to directly program Friendliness, again, he did not do so. It does seem, though, that if anything can modify Eliezer's already formed opinions, it would be the influence of such a personal acquaintance.

Thrun: a mathematical formula that Eliezer hadn't previously considered, so he was willing to change his mind.

Roger Schank: Eliezer simply dismisses him as disagreeing with one of his fixed opinions.

Conclusion: once Eliezer has made up his mind, his mind is made up, and cannot be changed or modified to any degree whatsoever by the influence of someone else's opinion, no matter how expert the person may be.

Unknown: Well, maybe yeah, but so what? It's just practically impossible the completely re-evaluate every belief you hold whenever someone says something that asserts the belief to be wrong. That's nothing at all to do with "overconfidence", but it's everything to do with sanity. The time to re-evaluate your beliefs is when someone gives a possibly plausible argument about the belief itself, not just an assertion that it is wrong. Like e.g. whenever someone argues anything, and the argument is based on the assumption of a personal god, I dismiss it out of hand without thinking twice - sometimes I do not even take the time to hear them out! Why should I, when I know it's gonna be a waste of time? Overconfidence? No, sanity!

Frank: I do not suggest "completely re-evalating" a belief when it is contradicted. And it is true that if you have a considered opinion on some matter, even though you know that many people disagree with it, you are unlikely to completely change your mind when you hear that some particular person disagrees with it.

However, if you are surprised to hear that some particular person disagrees with it, then you should update your opinion in such a way that there is a greater probability (than before) that the person holds an unreasonable opinion in the matter. But you should also update your opinion in such a way that there is a greater probability than before that you are wrong.

For example, since Eliezer was surprised to hear of Dennett's opinion, he should assign a greater probability than before to the possibility that human level AI will not be developed with the foreseeable future. Likewise, to take the more extreme case, assuming that he was surprised at Aumann's religion, he should assign a greater probability to the Jewish religion, even if only to a slight degree.

Of course this would be less necessary to the degree that you are unsurprised by his disagreeing opinion, just as if you knew about it completely when you originally formed your opinion, you would not need to update.

So, let's see, Unknown thinks I'm overconfident for not believing in collapse fairies, for believing in reductionism, for not shifting my opinion toward Robert Aumann's belief that an omnipotent deity took sides in ancient tribal quarrels, and... what else?

Foolish mortal, it is a hundred years too early for you to accuse me of arrogance.

Oh, and point me to any single comment on Overcoming Bias where you've changed your opinion. Ever. Googling on "you know, you're right" produced this and this from me. There've been other cases but I can't recall their Google keywords offhand.

Re: "This confirms my general position that Eliezer is overconfident."

Best to have evidence of him being wrong - rather than just disagreements, then. I don't mean to boost his confidence - but Eliezer has views on many subjects, and while a few strike me as being a bit odd, he seems to me to be right a lot more than most people are. This may even be a consequence of his deliberate efforts to act like a rational agent.

Eliezer, you should also update your opinion about whether I have ever changed my opinions.

See (I don't know how to do the links):

http://www.overcomingbias.com/2007/11/no-evolution-fo.html#comment-90291902

Here I partly changed my opinion.

See also:

http://www.overcomingbias.com/2008/01/something-to-pr.html#comment-99342014

This represented a rather dramatic change in my opinion.

Finally, although I admit that I never admitted it, you persuaded me that zombies are impossible (although you did not persuade me to assign this a 100% probability.)

@Unknown: Okay, updated.

Foolish mortal, it is a hundred years too early for you to accuse me of arrogance.

Posted without further comment.

@Unknown: Okay, updated.

Eliezer, if this whole 'Friendly AI' thing doesn't work out, a career writing comedy awaits.

Love this blog to bits.

Foolish mortal, it is a hundred years too early for you to accuse me of arrogance.

Please don't do that.

Eliezer, if this whole 'Friendly AI' thing doesn't work out, a career writing comedy awaits.

Huh? I cannot imagine what is comic about that exchange. Do some find it comically sincere or geeky?

[Unknown wrote:] [...] you should update your opinion [to] a greater probability [...] that the person holds an unreasonable opinion in the matter. But [also to] a greater probability [...] that you are wrong.

In principle, yes. But I see exceptions.

[Unknown wrote:] For example, since Eliezer was surprised to hear of Dennett's opinion, he should assign a greater probability than before to the possibility that human level AI will not be developed with the foreseeable future. Likewise, to take the more extreme case, assuming that he was surprised at Aumann's religion, he should assign a greater probability to the Jewish religion, even if only to a slight degree.

Well, admittedly, the Dennett quote depresses me a bit. If I were in Eliezers shoes, I'd probably also choose to defend my stance - you can't dedicate your life to something with just half a heart!

About Auman's religion: That's one of the cases where I refuse to adapt my assigned probability one iota. His belief about religion is the result of his prior alone. So is mine, but it is my considered opinion that my prior is better! =)

Also, if I may digress a bit, I am sceptical about Robin's Hypothesis that humans in general update to little from other people's beliefs. My first intuition about this was that the opposite was the case (because of premature convergence and resistance to paradigm shifts). After having second thoughts, I believe the amount is probably just about right. Why? 1) Taking other people's beliefs as evidence is an evolved trait, and so is probably the approximate amount. 2) Evolution is smarter than I (and Robin, I presume).

"You can teach people Bayesian math, but even if they're genuinely very good with the math, applying it to real life and real beliefs is a whole different story."

The problem is that people do things, or believe things, or say they believe things for reasons other than instrumental truth.

Many people have done that throughout history. Homosexuals married and had children. Atheists attended church. While they may have had private disagreements that their closest relatives knew about, they were advised to keep quiet and not embarrass their family.

In fact, religion in America is something of an oddity. Here, you're actually expected to believe what you espouse. That is not the case throughout most of the world. People treat their religion more like an ethnicity or family tradition. They espouse things for social reasons.

In Europe, people call themselves Anglican or Lutheran or Catholic often without much thought to what that really means. When I, at the age of 12, told my European mother that I didn't believe in God, she responded, "What do you mean? You're Catholic!" As if that meant something other than the system of beliefs in question.

It's similar to the legend about the man in Northern Ireland who said he was an atheist. "But are you a Catholic Atheist or a Protestant Atheist?" was the reply.

In the Far East, the situation is even more complicated. Traditions like Buddhism and Confucianism are not mutually exclusive. And in Hinduism there are many gods (or many versions of one God, depending on who you ask). Each family "worships" a different God by invoking that God during various ceremonies. But if you attend the wedding of another family, you supplicate to that God, and you don't argue about it. You don't care. Your association as a Shiva-worshiping Hindu or a Vishnu-worshiping Hindu is ambiguous at best.

Likewise, I think that many intelligent and scholarly people who identify as Catholic (Ken Miller) or Jewish (Aumann) do so in just this (social) sense.

Frank, our belief tendencies did evolve, but selection pressures often prefer inaccurate beliefs.

"In IEEE Spectrum's sad little attempt at Singularity coverage, one bright spot is Paul Wallich's "Who's Who In The Singularity",..."

Brightness here being a relative quality... I am labeled green, meaning "true believer, thinks it will happen within 30 years." Yet I am quoted (correctly) as saying "I would... assign less than a 50% probability to superintelligence being developed by 2033." (I also don't endorse "once the singularity comes near, we will all be kicking ourselves for not having brought it about sooner", even though they attribute this to me as my "central argument".)

Reg Oracle AI, I'm not sure how much of a disagreement there exists between Eliezer and me. My position has not been that it is definitely the case that Oracle AI is the way to go. Rather, my position is something like "this seems to have at least something going for it; I have not yet been convinced by the arguments I've heard against it; it deserves some further consideration". (The basic rationale is this: While I agree that a utility function that is maximized by providing maximally correct and informative answers to our questions is clearly unworkable (since this could lead the SI to transform all Earth into more computational hardware so as to better calculate the answer), it might turn out to be substantially easier to specify the needed constraints to avoid such catastrophic side-effects of an Oracle AI than it is to solve the Friendliness problem in its general form--I'm not at all sure it is easier, but I haven't yet been persuaded it is not.)

Reg disagreement between Robin and Eliezer on singularity: They've discussed this many times, both here and on other mailinglists. But the discussion always seems to end prematurely. I think this would make for a great disagreement case study--topic is important, both are disagreement savvy, both know and respect one another, both have some subject matter expertise... I would like them to try once to get to the bottom of the issue, and continue discussion until they either cease disagreeing or at least agree exactly on what they disagree about, and why, and on how each person justifies the persistent disagreement.

Nick, I like where you're coming from on this topic. I'd like for you to have more urgent desperation though. Also, I'm curious where you stand on mass breeding/cloning our best existential risk minimizers. If you support it, I encourage you to start writing/blogging about it anonymously. We have such similar worldviews, that I'd be interested in knowing if you diverge from me on this topic, and why.

Nick:

Reg disagreement between Robin and Eliezer on singularity: They've discussed this many times, both here and on other mailinglists. But the discussion always seems to end prematurely. I think this would make for a great disagreement case study--topic is important, both are disagreement savvy, both know and respect one another, both have some subject matter expertise... I would like them to try once to get to the bottom of the issue, and continue discussion until they either cease disagreeing or at least agree exactly on what they disagree about, and why, and on how each person justifies the persistent disagreement.

I think that's the general idea, but I want to do as much foundational blogging in advance before that point that I can refer back to - otherwise it's duplicated effort, or hurried half-arguments.

Eliezer, may I ask, I'm having trouble figuring this out, what is your algorithm which is superior to Thrun's?

""" Moreover, once you realized how you were using your prior knowledge, you could see a derandomized version of the algorithm which was superior, even though it didn't make the exact guarantee Thrun had made. """

Nevermind. I actually started to think about it. You can just make a cutoff at the 50% cdf point in your probability distribution over infinity.

I lost track completely when I read about this Aumann.

Discovering that Aumann was Jewish, did not cause me to revisit the issues even momentarily.

What? What are these issues and why is Robert Aumann in the article? What has he said? What are you talking about? It seemed out of context. I mean, did not understand what you meant by

Then if you open the envelope and find that amount of money, roll a die and switch the envelope at that probability.

either, but that I can lookup. I don´t feel I should need to investigate Aumann just because he was mentioned.