By Aumann's agreement theorem (and some related/inferred ideas), two rationalists can't agree to disagree.
They can if they are instrumental rationalists. Then they may be signalling, lying or manipulating.
The theorem doesn't bother with such distracting side-shows as dishonest communicatio. The paper starts with the much more straightforward statement: If two people have the same priors, and their posteriors for a given event A are common knowledge, then these posteriors must be equal. Obviously, if one of them is successfully deceiving the other, the second hypothesis is false.
In Aumann's agreement theorem the two rationalists have common knowledge of the subject and share common priors. These two conditions are not true in the common cases you are discussing. Two rationalists can agree that it is not worth while to take the time to make all pertinent knowledge common and come to common priors. This can only happen with unimportant topics because otherwise it would be worth spending more time on it, this is still not agreeing to disagree.
If both rationalists have a high probability of very different answers(strongly disagree) it can indicate, drastically different knowledge or priors, and in real life where no one is perfectly rational it often indicates a lack of rationality in one or both parties. If it is the last case it is probably worth discussing the unimportant topic just to uncover and correct the irrational thought processes in both parties. So if two rationalists that share a large common knowledge base and strongly disagree there is a higher probability the disagreement arises from irrationality on one or both parts and it is therefore a good idea to discuss the topic further to isolate the irrationality and correct it.
On the other hand if the two rationalist have very different knowledge bases then it is likely their disagreement arises from their different knowledge bases and/or priors. Sharing the two knowledge bases could take a great deal of time and may not be worth the effort for an unimportant problem. If the two rationalists decide to walk away from the with out sharing knowledge they should both devalue their knowledge proportionally to how much they judge the other to be rational(both in logic and ability to curate their data and formulate priors and taking into account that it is harder to judge the others irrationality dues to the large difference in knowledge bases).
They have common knowledge of their disagreement, not of the subject! They need not share "all pertinent knowledge"!
I still need to reread the paper in more detail, but an argument for common priors meaning identical or shared knowledge(both people have the same knowledge base not information transfer from one to the other). There was also a counter argument but it seem like a weak argument to me at my last reading.
From lesswrong wiki on priors:
This requires that, even in advance of seeing the evidence, you have beliefs about what the evidence means - how likely you are to see the evidence, if various hypotheses are true - and how likely those hypotheses were, in advance of seeing the evidence.
I do not see how you can share all of this and not be sharing a common knowledge base(or at least an isomorphic knowledge base). The specific example of different priors in the wiki also seems to fits this view.
So if two rationalists that share a large common knowledge base and strongly disagree there is a higher probability the disagreement arises from irrationality on one or both parts and it is therefore a good idea to discuss the topic further to isolate the irrationality and correct it.
I followed along with the premises there cheerfully enough but put approximately the opposite conclusion after the 'therefore'. I want to speak to people who have a lot of knowledge that I don't have and are rational, not people who know the same stuff that I do but can't think straight.
It's far more useful to learn new stuff from people than to try to fix broken ones.
Given that you are not perfectly rational along with everyone else, if you and another rationalist are in a discussion and he/she realize that you are irrational on a particular narrow topic what action would you advocate if not help you become more rational by at least pointing out your irrationality?
This idea is important to me because it seems like the best way to increase the rational population. Which makes my life better and th life of my descendants better.
edit: Also the op was original taking about an unimportant topic that was only with taking about for a small amount of time. So a large data transfer is not possible in this case and why it is not possible to resolve the disagreement. For topics/problems worth spending time on it is of course very valable to share knowledge with other rationalists.
Pardon me, I just finished writing an extended reply here then lost it when I bumped the mouse. Since I was rather pleased with the comment it would be rather frustrating to reconstruct it. (What sort of silly web browser doesn't keep text areas populated in the back cache? Lame.)
I understand your pain. Chrome I think is one example, at least for lesswrong.com it does not have this problem with all websites. I now have ctl-a ctl-c in muscle memory to prevent such pain.
I now have ctl-a ctl-c in muscle memory to prevent such pain.
Ditto. I started doing that a long time ago, no matter what the browser.
In Aumann's agreement theorem the two rationalists have common knowledge of the subject and share common priors. These two conditions are not true in the common cases you are discussing. Two rationalists can agree that it is not worth while to take the time to make all pertinent knowledge common and come to common priors. This can only happen with unimportant topics because otherwise it would be worth spending more time on it, this is still not agreeing to disagree.
Only their posteriors need to be "common knowledge" - which is pretty weak - and seems worlds away from making "all pertinent knowledge common". See page 1 here for details.
For a nice expression of the idea, see here:
Imagine I think there are 200 balls in the urn, but Robin Hanson thinks there are 300 balls in the urn. Once Robin tells me his estimate, and I tell him mine, we should converge upon a common opinion. In essence his opinion serves as a "sufficient statistic" for all of his evidence.
http://www.marginalrevolution.com/marginalrevolution/2005/10/robert_aumann_n.html
In particular the bit: "In essence his opinion serves as a 'sufficient statistic' for all of his evidence".
This is usually considered to mean that sharing the information required to produce convergence is usually a rather small effort - since relatively few bits of opinion need to be passed back and forth - rather than lots of facts and evidence.
I was not using the term "common knowledge" the same way Aumann paper was. I was baseing my use of the term on what I found in the lesswrong wiki. I used common knowledge and common priors as essentially the same object in my post. Having the same priors seems to require "all pertinent knowledge" be known by both parties or known in common(this is how I used the term in my post) or a large coincidence where two, pertinent, partial non-overlapping(at least), lead to the same priors.
Imagine I think there are 200 balls in the urn, but Robin Hanson thinks there are 300 balls in the urn. Once Robin tells me his estimate, and I tell him mine, we should converge upon a common opinion. In essence his opinion serves as a "sufficient statistic" for all of his evidence.
May be I do not understand priors correctly. It the provided example it seems like Robin Hanson and and the author have different priors. These two cases seem to parellel what is consider two separate priors in the less wrong wiki:
Suppose you had a barrel containing some number of red and white balls. If you start with the belief that each ball was independently assigned red color (vs. white color) at some fixed probability between 0 and 1, and you start out ignorant of this fixed probability (the parameter could anywhere between 0 and 1), then each red ball you see makes it more likely that the next ball will be red. (By Laplace's Rule of Succession.)
On the other hand, if you start out with the prior belief that the barrel contains exactly 10 red balls and 10 white balls, then each red ball you see makes it less likely that the next ball will be red (because there are fewer red balls remaining).
"Common knowledge" is a highly misleading piece of technical terminology - in the context of Aumann's paper.
A two-person Aumann agreement exchange example (of degrees C warming next century) looks like:
A: I think it's 1.0 degrees...
B: Well, I think it's 2.0 degrees...
A: Well, in that case, I think it's 1.2 degrees...
B: Well, in that case, I think it's 1.99 degrees...
A: Well...
The information exchanged is not remotely like all the pertinent knowledge - and so making the exchange is often relatively quick and easy.
That is not the definition at the top of the paper you just linked for me: http://www.ma.huji.ac.il/~raumann/pdf/Agreeing%20to%20Disagree.pdf
"Two people, 1 and 2, are said to have common knowledge of an event E if both know it, 1 knows that 2 knows it, 2 knows that 1 knows is, 1 knows that 2 knows that 1 knows it, and so on."
Or I may have missed you point entirely. You have introduced the concept of "Aumann agreement exchange" but not what misconception you are trying to clear up with it.
Examples often help. I don't know if you have a misconception - but a common misconception is that A and B need to share their evidence pertaining to the temperature rise before they can reach agreement.
What Aumann says - counterintuitively - is that no, they just need to share their estimates of the temperature rise with each other repeatedly - so each can update on the other's updates - and that is all. As the Cowen quote from earlier says:
"In essence his opinion serves as a 'sufficient statistic' for all of his evidence".
(nods) Examples do indeed help.
Suppose agent A has access to observations X1..X10, on the basis of which A concludes a 1-degree temperature rise.
Suppose agent B has access to observations X1..X9, on the basis of which A concludes a 2-degree temperature rise, and A and B are both perfect rationalists whose relevant priors are otherwise completely shared and whose posterior probabilities are perfectly calibrated to the evidence they have access to.
It follows that if B had access to X10, B would update and conclude a 2-degree rise. But neither A nor B know that.
In this example, A and B aren't justified in having the conversation you describe, because A's estimate already takes into account all of B's evidence, so any updating A does based on the fact of B's estimate in fact double-counts all of that evidence.
But until A can identify what it is that B knows and doesn't know, A has no way of confirming that. If they just share their estimates, they haven't done a fraction of the work necessary to get the best conclusion from the available data... in fact, if that's all they're going to do, A was better off not talking to B at all.
Of course, one might say "But we're supposed to assume common priors, so we can't say that A has high confidence in X10 while B doesn't." But in that case, I'm not sure what caused A and B to arrive at different estimates in the first place.
I don't think Aumann's agreement theorem is about getting "the best conclusion from the available data". It is about agreement. The idea is not that an exchange produces a the most accurate outcome from all the evidence held by both parties - but rather that their disagreements do not persist for very long.
This post questions the costs of reaching such an agreement. Conventional wisdom is as follows:
But two key questions went unaddressed: first, can the agents reach agreement after a conversation of reasonable length? Second, can the computations needed for that conversation be performed efficiently? This paper answers both questions in the affirmative, thereby strengthening Aumann's original conclusion.
Huh. In that case, I guess I'm wondering why we care.
That is, if we're just talking about a mechanism whereby two agents can reach agreement efficiently and we're OK with them agreeing on conclusions the evidence doesn't actually support, isn't it more efficient to, for example, agree to flip a coin and agree on A's estimate if heads, and B's estimate if tails?
I can't speak for all those interested - but I think one common theme is that we see much persistent disagreement in the world when agents share their estimates - Aumann says it is unlikely to be epistemically rational and honest (although it often purports to be) - so what is going on?
Your proposed coin-flip is certainly faster than Aumann agreement - but does not offer such good quality results. In an Aumann agreement, agents take account of each others' confidence levels.
Here is a concrete example. Two referees independently read the same paper, giving it a mark in the range [0,1], where 0 is a definite reject and 1 is a definite accept. They then meet to decide on a joint verdict.
Alice rates the paper at 0.9. Bob rates the paper at 0.1. Assuming their perfect rationality:
Assume that they both have the same prior over the merits of the papers they receive to review: their true worths are uniformly distributed over [0,1]. They have read the same paper and have honestly attempted to judge it according to the same criteria. They may have other, differing information available to them, but the Aumann agreement process does not involve sharing such information.
I've been trying to analyse this in terms of Aumann's original paper and Scott Aaronson's more detailed treatment but I am not getting very far. In Aaronson's framework, if we require a 90% chance of Alice and Bob agreeing to within 0.2, then this can be achieved (Theorem 5) with at most 1/(0.1*0.2^2) = 250 messages in which one referee tells the other their current estimate of the paper. It is as yet unclear to me what calculations Alice and Bob must perform to update their estimates, or what values they might converge on.
In practice, disagreements like this are resolved by sharing not posteriors, but evidence. In this example, Bob might know something that Alice does not, viz. that the authors already published almost the same work in another venue a year ago and that the present paper contains almost nothing new. Or, on the other hand, Bob might simply have missed the point of the paper due to lacking some background knowledge that Alice has.
They may have other, differing information available to them, but the Aumann agreement process does not involve sharing such information.
What they do indirectly share is something like their confidence levels - and how much their confidence is shaken by the confidence of their partner in a different result.
Yes, Aumann agreement is not very realistic - but the point is that the partners can be expected to relatively quickly reach agreement, without very much effort - if they are honest, truth-seekers with some energy for educating others - and know the other is the same way.
So, the prevalance of persistent disagreements suggests that the world is not filled with honest, truth-seekers. Not very surprising, perhaps.
Yes, Aumann agreement is not very realistic - but the point is that the partners can be expected to relatively quickly reach agreement, without very much effort - if they are honest, truth-seekers with some energy for educating others - and know the other is the same way.
250 rounds of "I update my estimate to..." strikes me as rather a lot of effort, but that's not the important point here. My question is, assume that Alice and Bob are indeed honest truth-seekers with some energy for educating others, and have common knowledge that this is so. What then will the Aumann agreement process actually look like for this example, where the only thing that is directly communicated is each party's latest expectation of the value of the paper? Will it converge to the true value of the paper in both of the following scenarios:
No, I don't think it is true that both parties necessarily wind up with more accurate estimates after updating and agreeing, or even an estimate closer to what they would have obtained by sharing all their data.
No, I don't think it is true that both parties necessarily wind up with more accurate estimates after updating and agreeing, or even an estimate closer to what they would have obtained by sharing all their data.
That greatly diminishes the value of the theorem, and implies that it fails to justify blaming dishonesty and irrationality for the prevalance of persistent disagreements.
I'm not sure. Aumann's paper seems to only bill itself as leading to agreement - with relatively little discussion of the properties of what is eventually agreed upon. Anyway, I think you may be expecting too much from it; and I don't think it fails in the way that you say.
Why should ideal Bayesian rationalists alter their estimates to something that is not more likely to be true according to the available evidence? The theorem states that they reach agreement because it is the most likely way to be correct.
The parties do update according to their available evidence. However, neither has access to all the evidence. Also, evidence can be misleading - and subsets of the evidence are more likely to mislead.
Parties can become less accurate after updating, I think.
For example, consider A in this example.
For another example, say A privately sees 5 heads, and A's identical twin, B privately sees 7 tails - and then they Auman agree on the issue of whether the coin is fair. A will come out with more confidence in thinking that the coin is biased. If the coin is actually fair, A will have become more wrong.
If A and B had shared all their evidence - instead of going through an Auman agreement exchange - A would have realised that the coin was probably fair - thereby becoming less wrong.
Sometimes following the best available answer will lead you to an answer that is incorrect, but from your own perspective it is always the way to maximize your chance of being right.
To recap, what I originally said here was:
I don't think it is true that both parties necessarily wind up with more accurate estimates after updating and agreeing, or even an estimate closer to what they would have obtained by sharing all their data.
The scenario in the grandparent provides an example of an individual's estimate becoming worse after Aumann agreeing - and also an example of their estimate getting further away from what they would have believed if both parties had shared all their evidence.
I am unable to see where we have any disagreement. If you think we disagree, perhaps this will help you to pinpoint where.
Perhaps I was reading an implication into your comment that you didn't intend, but I took it that you were saying that Aumann's Agreement Theorem leads to agreement between the parties, but not necessarily as a result of their each attempting to revise their estimates to what is most likely given the data they have..
That wasn't intended. Earlier, I cited this.
Imagine I think there are 200 balls in the urn, but Robin Hanson thinks there are 300 balls in the urn. Once Robin tells me his estimate, and I tell him mine, we should converge upon a common opinion. In essence his opinion serves as a "sufficient statistic" for all of his evidence.
http://www.marginalrevolution.com/marginalrevolution/2005/10/robert_aumann_n.html
My comments were intended to suggest that the results of going through an Aumann agreement exchange could quite be different from what you would get if the parties shared all their relevant evidence.
The main similarity is that the parties end up agreeing with each other in both cases.
By Aumann's agreement theorem (and some related/inferred ideas), two rationalists can't agree to disagree.
Isn't this claim obviously false? Doesn't agreement map directly to the halting problem, which is undecidable?
I question the utility/rationality of spend time optimizing small probability updates from topics "not sufficiently important to spend significant time on discussing them." It seems to me that: unimportant topic->small probability update(low utility update) means you are better of discussing important topics that are worth spending time on and optimizing those cases.
And, how much probability can be shifted in such a short time (given sufficiently complex topic) — would that amount be tiny?
The amount of probability that can be shifted can be large. What must be tiny is the complexity of the change in probability. It is easy to provoke a large update with a simple contradiction from a trusted expert but what you cannot do is gain a significant amount of information about exactly which of the detailed premises you most needed to update on.
Also, would it be more right to shift probability distribution towards the other person's beliefs or towards uncertainty (unitary distribution of probabilities, AFAIU
Interesting thought.
First, uncertainty isn't always a uniform distribution, it depends on your basic information (simple things like scale-invariance will make it not uniform, for example). But if the value is in some finite range, the expected value for the uncertain distribution is probably in the middle.
So the thought becomes "can someone espousing a more extreme view make your view more moderate?"
If the other person was perfect, the answer would clearly be "no." Evidence is evidence. If they have evidence, and they tell you the evidence, now you have the evidence too.
If we model people as making mistakes sometimes, and drawing their answer from the uncertain distribution when they make a mistake, it seems like it could happen. However, it would require you to change what you think the chance of making a mistake is by some amount large enough to counteract the weight other person's view.
It's a little tough to find real-world examples of this, because it can only happen in the right class of problems: if you get it wrong you have to get it totally wrong. A novel math calculation might fit the bill, but even then there are reasons to not be totally wrong when you're wrong.
-