If the two of you are wrapping things up by preparing to agree to disagree, you have to bite the bullet and say that the other is being irrational, or is lying, or is not truth seeking. There is no respectful way to agree to disagree. You must either be extremely rude, or reach agreement.
Hal, is this still your position, a year later? If so, I'd like to argue against it. Robin Hanson wrote in http://hanson.gmu.edu/disagree.pdf (page 9):
Since Bayesians with a common prior cannot agree to disagree, to what can we attribute persistent human disagreement? We can generalize the concept of a Bayesian to that of a Bayesian wannabe, who makes computational errors while attempting to be Bayesian. Agreements to disagree can then arise from pure differences in priors, or from pure differences in computation, but it is not clear how rational these disagreements are. Disagreements due to differing information seem more rational, but for Bayesians disagreements cannot arise due to differing information alone.
Robin argues in another paper that differences in priors really are irrational. I presume that he believes that differences in computation are also irrational, although I don't know if he made a detailed case for it somewhere.
Suppose we grant that these differences are irrational. It seems to me that disagreements can still be "reasonable", if we don't know how to resolve these differences, even in principle. Because we are products of evolution, we probably have random differences in priors and computation, and since at this point we don't seem to know how to resolve these differences, many disagreements may be both honest and reasonable. Therefore, there is no need to conclude that the other disagreer must be be irrational (as an individual), or is lying, or is not truth seeking.
Assuming that the above is correct, I think the role of a debate between two Bayesian wannabes should be to pinpoint the exact differences in priors and computation that caused the disagreement, not to reach immediate agreement. Once those differences are identified, we can try to find or invent new tools for resolving them, perhaps tools specific to the difference at hand.
My Bayesian wannabe paper is an argument against disagreement based on computation differences. You can "resolve" a disagreement by moving your opinion in the direction of the other opinion. If failing to do this reduces your average accuracy, I feel I can call that failure "irrational".
Two ideal Bayesians cannot have common knowledge of disagreement; this is a theorem. If two rationalist-wannabes have common knowledge of a disagreement between them, what could be going wrong?
The obvious interpretation of these theorems is that if you know that a cognitive machine is a rational processor of evidence, its beliefs become evidence themselves.
If you design an AI and the AI says "This fair coin came up heads with 80% probability", then you know that the AI has accumulated evidence with an likelihood ratio of 4:1 favoring heads - because the AI only emits that statement under those circumstances.
It's not a matter of charity; it's just that this is how you think the other cognitive machine works.
And if you tell an ideal rationalist, "I think this fair coin came up heads with 80% probability", and they reply, "I now think this fair coin came up heads with 25% probability", and your sources of evidence are independent of each other, then you should accept this verdict, reasoning that (before you spoke) the other mind must have encountered evidence with a likelihood of 1:12 favoring tails.
But this assumes that the other mind also thinks that you're processing evidence correctly, so that, by the time it says "I now think this fair coin came up heads, p=.25", it has already taken into account the full impact of all the evidence you know about, before adding more evidence of its own.
If, on the other hand, the other mind doesn't trust your rationality, then it won't accept your evidence at face value, and the estimate that it gives won't integrate the full impact of the evidence you observed.
So does this mean that when two rationalists trust each other's rationality less than completely, then they can agree to disagree?
It's not that simple. Rationalists should not trust themselves entirely, either.
So when the other mind accepts your evidence at less than face value, this doesn't say "You are less than a perfect rationalist", it says, "I trust you less than you trust yourself; I think that you are discounting your own evidence too little."
Maybe your raw arguments seemed to you to have a strength of 40:1, but you discounted for your own irrationality to a strength of 4:1, but the other mind thinks you still overestimate yourself and so it assumes that the actual force of the argument was 2:1.
And if you believe that the other mind is discounting you in this way, and is unjustified in doing so, then when it says "I now think this fair coin came up heads with 25% probability", you might bet on the coin at odds of 57% in favor of heads - adding up your further-discounted evidence of 2:1 to the implied evidence of 1:6 that the other mind must have seen to give final odds of 2:6 - if you even fully trust the other mind's further evidence of 1:6.
I think we have to be very careful to avoid interpreting this situation in terms of anything like a reciprocal trade, like two sides making equal concessions in order to reach agreement on a business deal.
Shifting beliefs is not a concession that you make for the sake of others, expecting something in return; it is an advantage you take for your own benefit, to improve your own map of the world. I am, generally speaking, a Millie-style altruist; but when it comes to belief shifts I espouse a pure and principled selfishness: don't believe you're doing it for anyone's sake but your own.
Still, I once read that there's a principle among con artists that the main thing is to get the mark to believe that you trust them, so that they'll feel obligated to trust you in turn.
And - even if it's for completely different theoretical reasons - if you want to persuade a rationalist to shift belief to match yours, you either need to persuade them that you have all of the same evidence they do and have already taken it into account, or that you already fully trust their opinions as evidence, or that you know better than they do how much they themselves can be trusted.
It's that last one that's the really sticky point, for obvious reasons of asymmetry of introspective access and asymmetry of motives for overconfidence - how do you resolve that conflict? (And if you started arguing about it, then the question wouldn't be which of these were more important as a factor, but rather, which of these factors the Other had under- or over-discounted in forming their estimate of a given person's rationality...)
If I had to name a single reason why two wannabe rationalists wouldn't actually be able to agree in practice, it would be that, once you trace the argument to the meta-level where theoretically everything can be and must be resolved, the argument trails off into psychoanalysis and noise.
And if you look at what goes on in practice between two arguing rationalists, it would probably mostly be trading object-level arguments; and the most meta it would get is trying to convince the other person that you've already taken their object-level arguments into account.
Still, this does leave us with three clear reasons that someone might point to, to justify a persistent disagreement - even though the frame of mind of justification and having clear reasons to point to in front of others, is itself antithetical to the spirit of resolving disagreements - but even so:
Since we don't want to go around encouraging disagreement, one might do well to ponder how all three of these arguments are used by creationists to justify their persistent disagreements with scientists.
That's one reason I say clearly - if it isn't obvious even to outside onlookers, maybe you shouldn't be confident of resolving the disagreement there. Failure at any of these levels implies failure at the meta-levels above it, but the higher-order failures might not be clear.