I really like this. It feels very important and I feel like I'm going to have to read through it a could more times to absorb and process.
This stuff just feels so fundamental. I'd love to see more discussion like this.
To see why it is wrong in general, consider an extreme case: a universal law, which you mostly already believe to be true.
I feel that this example might create another misconception, that certainty usually begets greater certainty. So here's an opposing example, where high confidence in a belief coexists with high probability that it's going to get less and less certain. You are at a bus stop, and you believe that you'll get to your destination on time, as busses here are usually reliable, though they only arrive once an hour and can deviate from schedule by several minutes. If you see a bus, you'll probably arrive in time, but every minute that you see no bus, it's evidence of something having gone wrong, so that the bus won't arrive at all in the next hour. You expect a highly certain belief (that you'll arrive on time) to decrease (a little bit), which is balanced by an unlikely alternative (for each given minute of waiting) of it going in the direction of greater certainty (if the bus does arrive within that minute).
Promoted to curated. One of LessWrong's main goals is to advance the art of rationality, and spotting patterns in the ways we process and misprocess evidence is a central piece of that. I also appreciated the Bayesian grounding, the epistemic statuses and the recapping and links to older work. I'm pretty sure most have made these errors before, and I expect that fitting them into a pattern will make them easier to recognize in the future.
I think all of this is fairly important—if you’re like me, you’ve likely made some mistakes along these lines.
Can you give some examples of specific mistakes you made or you observed others make? Have you observed me making any such mistakes? I ask because there's a lot of rather abstract arguments in this post and I'm not sure if it's worth the effort to think carefully about them to try to figure out if they apply to me. It seems like giving more concrete examples would help others who might have this reaction to your post too.
Thinking up actual historical examples is hard for me. The following is mostly true, partly made up.
On "If you can't provide me with a reason ...", I think the correct position is: when someone says X (and apparently means it, is someone whose opinions you expect to have some correlation with reality, etc.) you update towards X, and if they then can't give good reasons why X you then update towards not-X. Your overall update could end up being in either direction; if the person in question is particularly wise but not great at articulating reasons, or if X is the sort of thing whose supporting evidence you expect to be hard to articulate, the overall update is probably towards X.
That seems about right.
A concern I didn't mention in the post -- it isn't obvious how to respond to game-theoretic concerns. Carefully estimating the size of the update you should make when someone fails to provide good reason can be difficult, since you have to model other agents, and you might make exploitable errors.
An extreme way of addressing this is to ignore all evidence short of mathematical proof if you have any non-negligible suspicion about manipulation, similar to the mistake I describe myself making in the post. This seems too extreme, but it isn't clear what the right thing to do overall is. The fully-Bayesian approach to estimating the amount of evidence should act similarly to a good game-theoretic solution, I think, but there might be reason to use a simpler strategy with less chance of exploitable patterns.
This is a great post in a humanistic sense, that I only just read today via curatorial cleverness giving me appreciation for author and curators both :-)
I will now proceed to ignore the many redeeming qualities and geek out on math quibbling <3
Something jumped out at me related to the presumed equivalence between predicate logic and bayesian probability and subjective observations.
Consider how this is presented:
Then also this:
I instantly and intuitively "get what this is pointing to" within the essay as a way of phenomenologically directing my thoughts towards certain experiences of looking at things (or not), and trying experiments (or not), or even bringing things up in a discussion (or not)...
...and yet also some part of me feels that this is a sort of "an abuse of notation" that might have massive cascading consequences?
Like... it is nearly axiomatic that:
But it does NOT seem obviously "nearly axiomatic" that:
Also, I feel like "the predictable failure of anticipated observations to logically sum to unity" is close to the core of many confusions?
Until it is formalized, I find that I just do not know for sure how that "obs" operator is supposed to work mechanically and syntactically and in terms of whether adding that operator to the rest of the bayesian tools would maintain any particular properties like soundness or completeness!
Like maybe it is possible to "obs(E & not-E)" by two different methods, during a single observational session, and maybe that's just... in the range of formally allowed (sometimes empirically faulty?) "observations"?
Is this intended as syntactic sugar somehow for... sequential updating? (Surely the main point of Bayesian notation is to turn "pre-OBServational prior probabilities" into "post-OBServational posterior probabilities"?)
Or maybe you are trying to evoke SQL's "three valued logic" where NULL sort of "contaminates" the output of boolean logical operators applied to it? Like in SQL you say "(True AND NULL) is NULL" while "(True OR NULL) is True" and "(False OR NULL) is NULL".
Or is this intended to be evocative of something like the Pearlian DO() operator?
((
Maybe the OBS() operator is literally syntactic sugar for DO() in a special case?
Maybe you could build a larger belief network that has a boolean "observed_the_thing" as a "belief" node with a causal arrow to a different node that represents "observation_of_the_thing" and then P(observation_of_the_thing=SQLNULL|observed_the_thing=False) > 99% as an epistemic belief in a larger table of conditional probabilities that spell out exactly how "observing CAUSES observations"?
Then maybe OBS() is literally just DO() in the restricted case of DO(observed_the_thing=True)?
))
My hope, given that you authored this in 2019 and I'm commenting in 2023, is that you already noticed something proximate to these possibilities, and can just point me to some other essay or textbook chapter or something <3
Have you explored much of this in other ways since writing it?
It feels like it could be an evocative call to "invent new notation to formalize something not-yet-formalized" <3
Until it is formalized, I find that I just do not know for sure how that "obs" operator is supposed to work mechanically and syntactically and in terms of whether adding that operator to the rest of the bayesian tools would maintain any particular properties like soundness or completeness!
I entirely agree here. Since December, I have been writing a megapost about interpreting "obs" as "box" in modal logic, so that it's like a concept of necessity or proof, but mostly in a critical way where I question every assumption that anyone has made in the literature. Hopefully that post will se the light of day some time this year.
Currently I think of Obs(X) as the proposition that you have observed X. The idea that you should update on Obs(X) when you observe X is related to the idea that Obs(X) -> Obs(Obs(X)); ie, the idea that you always observe that you osberve X, if you observe X at all. I think this proposition is faulty, so we can't necessarily update on Obs(X), as most Bayesians recommend, even though we would be better off doing so.[1]
So I don't particularly think the "do" operator is involved here.
Like maybe it is possible to "obs(E & not-E)" by two different methods, during a single observational session, and maybe that's just... in the range of formally allowed (sometimes empirically faulty?) "observations"?
An example which appears in the philosophical literature: you observe that it's 6pm, and later you observe that it's 7pm, which contradicts the 6pm observation.
(The contradiction seems pretty dumb, and should be easy to take care of, but the important question is how exactly we take care of it.)
My hope, given that you authored this in 2019 and I'm commenting in 2023, is that you already noticed something proximate to these possibilities, and can just point me to some other essay or textbook chapter or something <3
I would point you to Novelty, Information, and Surprise with the caveat that I don't entirely buy the approach therein, since it firmly assumes Obs(X) -> Obs(Obs(X)). However, it still yields an interesting generalization of information theory, by rejecting the critical 'partition assumption', an assumption about how Obs() can work (due to Aumann, afaict) which I briefly argued against in my recent post on common knowledge. I think re-reading Aumann's classic paper 'agreeing to disagree' and thinking carefully about what's going on is a good place to start. Also, Probabilistic Reasoning in Intelligent Systems by Judea Pearl has a careful, thoughtful defense of conditioning on Obs(X) instead of X somewhere in the early chapters.
I'll construct a counterexample.
Define the "evidence relation" R to mean that when I'm in w1, I think I might be in w2. Define Obs(X) to mean that I think I'm within the set of worlds X. (That is to say, X is equal to, or a superset of, the worlds I think I might be in.) The "information set at a world w" is the set of worlds I think I might be in at w. To "know" a proposition X is to Obs(X); that is, to rule out worlds where X is not the case.
Obs(X) -> Obs(Obs(X)) implies that R is transitive: if I think I might be in a world w2, then I must think I might be in any world w2 would think it might be in. Otherwise, I think I might be in a world w2 where I thought I might be in some world w3, which I don't currently think I might be in. In other words, the information set at w2 contains something which the information set at w1 does not contain. Setting X to be something that's false in w3 but true in w1 and w2, we see that Obs(X) but not Obs(Obs(X)), contradicting our initial assumption.
Indeed, Obs(X) -> Obs(Obs(X)) is equivalent to transitivity of R.
But transitivity of R is implausible for an agent embedded in the physical world. For example, if I observe a coffee cup on a table, I can only measure its location to within some measurement error. For every possible location of the coffee cup, the information set includes a small region around that precise location. Transitivity says that if we can always slide the coffee cup by 1mm to make an indistinguishable observation, then we must be able to slide it by Xmm for any X. So, if we accept both transitivity and realistic imprecision of measurement, we are forced to conclude that the coffee cup could be anywhere.
I would almost nominate this post for this quote alone:
The right thing to do is closer to this: figure out how convincing you expect evidence to look given the extent of selection bias. Then, update on the difference between what you see and what's expected. If a clever arguer makes a case which is much better than what you would have expected they could make, you can update up. If it is worse that you'd expect, even if the evidence would otherwise look favorable, you update down.
I've used this heuristic several times over the last year, and it was better than whatever I would have done otherwise.
Zooming out, I personally found points 1, 4, 5, and 6 to be insightful one year on.
Posts like this are deceptively hard to write, so I really appreciate how well done this is.
Providing reasons feels fractal, or ship of theseus like to me. The metaphor that comes to mind is something like
Imagine two martial artists sparring, you are listening to a commentator describe the match over a radio. Two commentators would describe the match differently. In principle, a fight between two novices and a fight between two masters might sound very similar if the commentary captures a low enough resolution of events. When trying to communicate, we're something like the commentator looking directly at the mashing together of felt senses and using various mental moves to carve up the high dimensional space differently. Groups of people will fall into commentator norms to improve bandwidth, but these choices carry (usually unacknowledged) trade offs. Reification at one particular abstraction level forces a lot of structure on things that is a result of the choice of level as much as a result of the territory.
This is one of the reasons for Chapman's 'if a problem seems hard, the representation is probably wrong.' Different initial basis choices tend to push the complexity around to different parts of the model. And this process isn't even always perverse. Often the whole point is that you really can shove the uncertainty somewhere where it doesn't matter for your current purposes.
If you, while awake, decide to doubt whether you're awake (no matter how compelling the evidence that you're awake seems to be), then you're not really improving your overall correctness.
It builds a habit that makes you also doubt while dreaming.
I like that this responds to a conflict between two of Eliezer's posts that are far apart in time. That seems like a strong indicator that it's actually building on something.
Either "just say the truth", or "just say whatever you feel you're expected to say" are both likely better strategies.
I find this believable but not obvious. For example, if the pressure on you is you'll be executed for saying the truth, saying nothing is probably better that saying the truth. If the pressure on you is remembering being bullied on tumblr, and you're being asked if you disagree with the common wisdom at a LW meetup, saying nothing is better than saying what you feel expected to say.
I find it pretty plausible that those are rare circumstances where the triggering uncertainty state doesn't arise, but then there are some bounds on when the advice applies that haven't been discussed at all.
a little cherry-picking is OK
I think the claim being made here is that in most cases, it isn't practical to review all existing evidence, and if you attempt to draw out a representative sub-sample of existing evidence, it will necessarily line up with your opinion.
In cases where you can have an extended discussion you can mention contradicting evidence and at least mention that it is not persuasive, and possibly why. But in short conversations there might only be time for one substantial reference. I think that's distinct from what I would call "cherry-picking." (it does seem like it would create some weird dynamics where your estimate of the explainer's bias rises as you depart from uncertainty, but I think that's extrapolating too far for a review)
I think the comment of examples is helpful here.
I wonder about the impact of including something like this, especially with social examples, in a curated text that is at least partly intended for reading outside the community.
It seems to me that the dream example doesn't actually violate the principle "Yes require the possibility of no", but just a very tricky case.
If i understand correctly, what you're saying is basically that the observation itself is evidence. now, it's only evidence if it happens only in one case, when you're awake. so wouldn't saying "observation as evidence requires the possibility of no observation" be correct and consistent with the principle?
Yeah. But I fear that a more common reading of "yes requires the possibility of no" takes it to mean "yes requires the possibility of an explicit no", when in fact it's just "yes requires the possibility of not-yes". I would rather explicitly highlight this by adding "yes requires the possibility of no, or at least, silence", rather than just lumping this under "tricky cases" of yes-requires-the-possibility-of-no.
Fair enough, thanks for the clarification . "not-yes" actually makes things clearer to me than "silence", but unfortunately it doesn't sound elegant. anyway I'm happy my intuition of the principle was right, and it was more a matter of labeling.
Lots more of the basics. Practical heuristics explained with simple theory.
"If you can't provide me with a reason, I have to assume you're wrong."
One: I make a mathematical claim, while talking to a (smart) mathematician. They say "That doesn't * hold. **"
Two: I explain the proof/the conditions for the proof. The mathematician says, "Right, it holds under those conditions."
The only problem is, when I can't generate a proof. Then "One" can happen, but not "Two".
Unspoken:
*necessarily
**in all cases.
I no longer think it makes sense to clam up when you can't figure out how you originally came around to the view which you now hold
Either you can say "I came to this conclusion at some point, and I trust myself", or you should abandon the belief.
You don't need to know how or why your brain happened to contain the belief; you just need to know your own justification for believing it now. If you can't sufficiently justify your belief to yourself (even through things like "My-memory-of-myself-from-a-few-minutes-ago thinks it's likely" or "First-order intuition thinks it's likely"), you should abandon it (unless you're bad at this, which is probably not the case for most people who might try it).
From my perspective, I just had an original thought. If there's any writing about something related, or if someone else has something to add or subtract, I would probably very much like to read it.
it would imply that you should ignore mathematical proofs if the person who came up with the proof only searched for positive proofs and wouldn't have spend time trying to prove the opposite. (This ties in with the very first section -- failing to find a proof is like remaining silent.)
I think this strategy becomes coherent if you update on the claim "fact X is true, here's its proof" being made? After all, there's lower probability that person publishes such claim if they fail to find the proof.
(Generalization: it doesn't matter much on what arguments you update, it matters more what you end up believing.)
The strategy "ignore the arguments" still goes wrong if they've published an incorrect mathematical proof, with a flaw you could have spotted. So it's still clearly wrong in general, even with this adjustment.
This is great. A point which helped me understand number 6: If you ask someone "why do you believe X", since you're presumably going to update your probability of X upwards if they give a reason, you should update downwards if they don't give a reason. But you probably already updated upwards as soon as they said "I believe X", and there is no theorem which says this update has to be smaller than the latter update. So you can still end up with a higher or equal probability of X compared to where you were at the beginning of the conversation.
I view the issue of intellectual modesty much like the issue of anthropics. The only people who matter are those whose decisions are subjunctively linked to yours (it only starts getting complicated when you start asking whether you should be intellectually modest about your reasoning about intellectual modesty)
One issue with the clever arguer is that the persuasiveness of their arguments might have very little to do with how persuasive they should be, so attempting to work off expectations might fail.
I view the issue of intellectual modesty much like the issue of anthropics. The only people who matter are those whose decisions are subjunctively linked to yours (it only starts getting complicated when you start asking whether you should be intellectually modest about your reasoning about intellectual modesty)
I agree fairly strongly, but this seems far from the final word on the subject, to me.
One issue with the clever arguer is that the persuasiveness of their arguments might have very little to do with how persuasive they should be, so attempting to work off expectations might fail.
Ah. I take you to be saying that the quality of the clever arguer's argument can be high variance, since there is a good deal of chance in the quality of evidence cherry-picking is able to find. A good point. But, is it 'too high'? Do we want to do something (beyond the strategy I sketched in the post) to reduce variance?
I agree fairly strongly, but this seems far from the final word on the subject, to me.
Hmm, actually I think you're right and that it may be more complex than this.
Ah. I take you to be saying that the quality of the clever arguer's argument can be high variance, since there is a good deal of chance in the quality of evidence cherry-picking is able to find. A good point.
Exactly. There may only be a weak correlation between evidence and truth. And maybe you can do something with it or maybe it's better to focus on stronger signals instead.
Epistemic Status: I've really spent some time wrestling with this one. I am highly confident in most of what I say. However, this differs from section to section. I'll put more specific epistemic statuses at the end of each section.
Some of this post is generated from mistakes I've seen people make (or, heard people complain about) in applying conservation-of-expected-evidence or related ideas. Other parts of this post are based on mistakes I made myself. I think that I used a wrong version of conservation-of-expected-evidence for some time, and propagated some wrong conclusions fairly deeply; so, this post is partly an attempt to work out the right conclusions for myself, and partly a warning to those who might make the same mistakes.
All of the mistakes I'll argue against have some good insight behind them. They may be something which is usually true, or something which points in the direction of a real phenomenon while making an error. I may come off as nitpicking.
1. "You can't predict that you'll update in a particular direction."
Starting with an easy one.
It can be tempting to simplify conservation of expected evidence to say you can't predict the direction which your beliefs will change. This is often approximately true, and it's exactly true in symmetric cases where your starting belief is 50-50 and the evidence is equally likely to point in either direction.
To see why it is wrong in general, consider an extreme case: a universal law, which you mostly already believe to be true. At any time, you could see a counterexample, which would make you jump to complete disbelief. That's a small probability of a very large update downwards. Conservation of expected evidence implies that you must move your belief upwards when you don't see such a counterexample. But, you consider that case to be quite likely. So, considering only which direction your beliefs will change, you can be fairly confident that your belief in the universal law will increase -- in fact, as confident as you are in the universal law itself.
The critical point here is direction vs magnitude. Conservation of expected evidence takes magnitude as well as direction into account. The small but very probable increase is balanced by the large but very improbable decrease.
The fact that we're talking about universal laws and counterexamples may fool you into thinking about logical uncertainty. You can think about logical uncertainty if you want, but this phenomenon is present in the fully classical Bayesian setting; there's no funny business with non-Bayesian updates here.
Epistemic status: confidence at the level of mathematical reasoning.
2. "Yes requires the possibility of no."
Scott's recent post, yes requires the possibility of no, is fine. I'm referring to a possible mistake which one could make in applying the principle illustrated there.
Sometimes, look around, and ask myself whether I'm in a dream. When this happens, I generally conclude very confidently that I'm awake.
I am not similarly capable of determining that I'm dreaming. My dreaming self doesn't have the self-awareness to question whether he is dreaming in this way.
(Actually, very occasionally, I do. I either end up forcing myself awake, or I become lucid in the dream. Let's ignore that possibility for the purpose of the thought experiment.)
I am not claiming that my dreaming self is never deluded into thinking he is awake. On the contrary, I have those repeatedly-waking-up-only-to-find-I'm-still-dreaming dreams occasionally. In those cases, I vividly believe myself to be awake. So, it's definitely possible for me to vividly believe I'm awake and be mistaken.
What I'm saying is that, when I'm asleep, I am not able to perform the actually good test, where I look around and really consciously consider whether or not I might be dreaming. Nonetheless, when I can perform that check, it seems quite reliable. If I want to know if I'm awake, I can just check.
A "yes-requires-the-possibility-of-no" mindset might conclude that my "actually good test" is no good at all, because it can't say "no". I believe the exact opposite: my test seems really quite effective, because I only successfully complete it while awake.
Sometimes, your thought processes really are quite suspect; yet, there's a sanity check you can run which tells you the truth. If you're deluding yourself, the general category of "things which you think are simple sanity checks you can run" is not trustworthy. If you're deluding yourself, you're not even going to think about the real sanity checks. But, that does not in itself detract from the effectiveness of the sanity check.
The general moral in terms of conservation of expected evidence is: "'Yes' only requires the possibility of silence.". In many cases, you can meaningfully say yes without being able to meaningfully say no. For example, the axioms of set theory could prove their own inconsistency. They could not prove themselves consistent (without also proving themselves inconsistent). This does not detract from the effectiveness of a proof of inconsistency! Again, although the example involves logic, there's nothing funny going on with logical uncertainty; the phenomenon under discussion is understandable in fully Bayesian terms.
Symbolically: as is always the case, you don't really want to update on the raw proposition, but rather, the fact that you observed the proposition, to account for selection bias. Conservation of expected evidence can be written P(H)=P(H|E)P(E)+P(H|¬E)P(¬E), but if we re-write it to explicitly show the "observation of evidence", it becomes P(H)=P(H|obs(E))P(obs(E))+P(H|¬obs(E))P(¬obs(E)). It does not become P(H)=P(H|obs(E))P(obs(E))+P(H|obs(¬E))P(obs(¬E)). In English: evidence is balanced between making the observation and not making the observation, not between the observation and the observation of the negation.
Epistemic status: confidence at the level of mathematical reasoning for the core claim of this section. However, some applications of the idea (such as to dreams, my central example) depend on trickier philosophical issues discussed in the next section. I'm only moderately confident I have the right view there.
3. "But then what do you say to the Republican?"
I suspect that many readers are less than fully on board with the claims I made in the previous section. Perhaps you think I'm grossly overconfident about being awake. Perhaps you think I'm neglecting the outside view, or ignoring something to do with timeless decision theory.
A lot of my thinking in this post was generated by grappling with some points made in Inadequate Equilibria. To quote the relevant paragraph of against modest epistemology:
Let's put that in (pseudo-)conservation-of-expected-evidence terms: we know that just applying one's best reasoning will often leave one overconfident in one's idiosyncratic beliefs. Doesn't that mean "apply your best reasoning" is a bad test, which fails to conserve expected evidence? So, should we not adjust downward in general?
In the essay, Eliezer strongly advises allowing yourself to have an inside view even when there's an outside view which says inside views broadly similar to yours tend to be mistaken. But doesn't that go against what he said in Ethical Injunctions?
Ethical Injunctions argues that there are situations where you should not trust your reasoning, and fall back on a general rule. You do this because, in the vast majority of cases of that kind, your oh-so-clever reasoning is mistaken and the general rule saves you from the error.
In Against Modest Epistemology, Eliezer criticizes arguments which rely on putting arguments in very general categories and taking the outside view:
He instead advises that we should update on all the information we have, use our best arguments, reason about situations in full detail:
In Ethical Injunctions, he seems to warn against that very thing:
Now, maybe Eliezer has simply changed views on this over the years. Even so, that leaves us with the problem of how to reconcile these arguments.
I'd say the following: modest epistemology points out a simple improvement over the default strategy: "In any group of people who disagree, they can do better by moving their beliefs toward each other." "Lots of crazy people think they've discovered secrets of the universe, and the number of sane people who truly discover such secrets is quite small; so, we can improve the average by never believing we've discovered secrets of the universe." If we take a timeless decision theory perspective (or similar), this is in fact an improvement; however, it is far from the optimal policy, and has a form which blocks further progress.
Ethical Injunctions talks about rules with greater specificity, and less progress-blocking nature. Essentially, a proper ethical injunction is actually the best policy you can come up with, whereas the modesty argument stops short of that.
Doesn't the "actually best policy you can come up with" risk overly-clever policies which depend on broken parts of your cognition? Yes, but your meta-level arguments about which kinds of argument work should be independent sources of evidence from your object-level confusion. To give a toy example: let's say you really, really want 8+8 to be 12 due to some motivated cognition. You can still decide to check by applying basic arithmetic. You might not do this, because you know it isn't to the advantage of the motivated cognition. However, if you do check, it is actually quite difficult for the motivated cognition to warp basic arithmetic.
There's also the fact that choosing a modesty policy doesn't really help the republican. I think that's the critical kink in the conservation-of-expected-evidence version of modest epistemology. If you, while awake, decide to doubt whether you're awake (no matter how compelling the evidence that you're awake seems to be), then you're not really improving your overall correctness.
So, all told, it seems like conservation of expected evidence has to be applied to the details of your reasoning. If your put your reasoning in a more generic category, it may appear that a much more modest conclusion is required by conservation of expected evidence. We can justify this in classical probability theory, though in this section it is even more tempting to consider exotic decision-theoretic and non-omnescience considerations than it was previously.
Epistemic status: the conclusion is mathematically true in classical Bayesian epistemology. I am subjectively >80% confident that the conclusion should hold in >90% of realistic cases, but it is unclear how to make this into a real empirical claim. I'm unsure enough of how ethical injunctions should work that I could see my views shifting significantly. I'll mention pre-rationality as one confusion I have which seems vaguely related.
4. "I can't credibly claim anything if there are incentives on my words."
Another rule which one might derive from Scott's Yes Requires the Possibility of No is: you can't really say anything if pressure is being put on you to say a particular thing.
Now, I agree that this is somewhat true, particularly in simple cases where pressure is being put on you to say one particular thing. However, I've suffered from learned helplessness around this. I sort of shut down when I can identify any incentives at all which could make my claims suspect, and hesitate to claim anything. This isn't a very useful strategy. Either "just say the truth", or "just say whatever you feel you're expected to say" are both likely better strategies.
One idea is to "call out" the pressure you feel. "I'm having trouble saying anything because I'm worried what you will think of me." This isn't always a good idea, but it can often work fairly well. Someone who is caving to incentives isn't very likely to say something like that, so it provides some evidence that you're being genuine. It can also open the door to other ways you and the person you're talking to can solve the incentive problem.
You can also "call out" something even if you're unable or unwilling to explain. You just say something like "there's some thing going on"... or "I'm somehow frustrated with this situation"... or whatever you can manage to say.
This "call out" idea also works (to some extent) on motivated cognition. Maybe you're worried about the social pressure on your beliefs because it might influence the accuracy of those beliefs. Rather than stressing about this and going into a spiral of self-analysis, you can just state to yourself that that's a thing which might be going on, and move forward. Making it explicit might open up helpful lines of thinking later.
Another thing I want to point out is that most people are willing to place at least a little faith in your honesty (and not irrationally so). Just because you have a story in mind where they should assume you're lying doesn't mean that's the only possibility they are -- or should be -- considering. One problematic incentive doesn't fully determine the situation. (This one also applies internally: identifying one relevant bias or whatever doesn't mean you should block off that part of yourself.)
Epistemic status: low confidence. I imagine I would have said something very different if I were more an expert in this particular thing.
5. "Your true reason screens off any other evidence your argument might include."
In The Bottom Line, Eliezer describes a clever arguer who first writes the conclusion which they want to argue for at the bottom of a sheet of paper, and then comes up with as many arguments as they can to put above that. In the thought experiment, the clever arguer's conclusion is actually determined by who can pay the clever arguer more. Eliezer says:
Now, Eliezer is trying to make a point about how you form your own beliefs -- that the quality of the process which determines which claims you make is what matters, and the quality of any rationalizations you give doesn't change that.
However, reading that, I came away with the mistaken idea that someone listening to a clever arguer should ignore all the clever arguments. Or, generalizing further, what you should do when listening to any argument is try to figure out what process wrote the bottom line, ignoring any other evidence provided.
This isn't the worst possible algorithm. You really should heavily discount evidence provided by clever arguers, because it has been heavily cherry-picked. And almost everyone does a great deal of clever arguing. Even a hardboiled rationalist will tend to present evidence for the point they're trying to make rather than against (perhaps because that's a fairly good strategy for explaining things -- sampling evidence at random isn't a very efficient way of conversing!).
However, ignoring arguments and attending only to the original causes of belief has some absurd consequences. Chief among them is: it would imply that you should ignore mathematical proofs if the person who came up with the proof only searched for positive proofs and wouldn't have spend time trying to prove the opposite. (This ties in with the very first section -- failing to find a proof is like remaining silent.)
This is bonkers. Proof is proof. And again, this isn't some special non-Bayesian phenomenon due to logical uncertainty. A Bayesian can and should recognize decisive evidence, whether or not it came from a clever arguer.
Yet, I really held this position for a while. I treated mathematical proofs as an exceptional case, rather than as a phenomenon continuous with weaker forms of evidence. If a clever arguer presented anything short of a mathematical proof, I would remind myself of how convincing cherry-picked evidence can seem. And I'd notice how almost everyone mostly cherry-picked when explaining their views.
This strategy was throwing out data when it has been contaminated by selection bias, rather than making a model of the selection bias so that I could update on the data appropriately. It might be a good practice in scientific publications, but if you take it as a universal, you could find reasons to throw out just about everything (especially if you start worrying about anthropic selection effects).
The right thing to do is closer to this: figure out how convincing you expect evidence to look given the extent of selection bias. Then, update on the difference between what you see and what's expected. If a clever arguer makes a case which is much better than what you would have expected they could make, you can update up. If it is worse that you'd expect, even if the evidence would otherwise look favorable, you update down.
My view also made me uncomfortable presenting a case for my own beliefs, because I would think of myself as a clever-arguer any time I did something other than recount the actual historical causes of my belief (or honestly reconsider my belief on the spot). Grognor made a similar point in Unwilling or Unable to Explain:
While I think there's something to this mindset, I no longer think it makes sense to clam up when you can't figure out how you originally came around to the view which you now hold. If you think there are other good reasons, you can give them without violating good faith.
Actually, I really wish I could draw a sharper line here. I'm essentially claiming that a little cherry-picking is OK if you're just trying to convince someone of the view which you see as the truth, so long as you're not intentionally hiding anything. This is an uncomfortable conclusion.
Epistemic status: confident that the views I claim are mistaken are mistaken. Less confident about best-practice claims.
6. "If you can't provide me with a reason, I have to assume you're wrong."
If you take the conclusion of the previous section too far, you might reason as follows: if someone is trying to claim X, surely they're trying to give you some evidence toward X. If they claim X and then you challenge them for evidence, they'll try to tell you any evidence they have. So, if they come up with nothing, you have to update down, since you would have updated upwards otherwise. Right?
I think most people make this mistake due to simple conversation norms: when navigating a conversation, people have to figure out what everyone else is willing to assume, in order to make sensible statements with minimal friction. So, we look for obvious signs of whether a statement was accepted by everyone vs rejected. If someone was asked to provide a reason for a statement they made and failed to do so, that's a fairly good signal that the statement hasn't been accepted into the common background assumptions for the conversation. The fact that other people are likely to use this heuristic as well makes the signal even stronger. So, assertions which can't be backed up with reasons are likely to be rejected.
This is almost the opposite mistake from the previous section; the previous one was justifications don't matter, whereas this idea is only justifications matter.
I think something good happens when everyone in a conversation recognizes that people can believe things for good reason without being able to articulate those reasons. (This includes yourself!)
You can't just give everyone a pass to make unjustified claims and assert that they have strong inarticulable reasons. Or rather, you can give everyone a pass to do that, but you don't have to take them seriously when they do it. However, in environments of high intellectual trust, you can take it seriously. Indeed, applying the usual heuristic will likely cause you to update in the wrong direction.
Epistemic status: moderately confident.
Conclusion
I think all of this is fairly important -- if you're like me, you've likely made some mistakes along these lines. I also think there are many issues related to conservation of expected evidence which I still don't fully understand, such as explanation vs rationalization, ethical injunctions and pre-rationality. Tsuyoku Naritai!