Your Strength as a Rationalist
(The following happened to me in an IRC chatroom, long enough ago that I was still hanging around in IRC chatrooms. Time has fuzzed the memory and my report may be imprecise.)
So there I was, in an IRC chatroom, when someone reports that a friend of his needs medical advice. His friend says that he's been having sudden chest pains, so he called an ambulance, and the ambulance showed up, but the paramedics told him it was nothing, and left, and now the chest pains are getting worse. What should his friend do?
I was confused by this story. I remembered reading about homeless people in New York who would call ambulances just to be taken someplace warm, and how the paramedics always had to take them to the emergency room, even on the 27th iteration. Because if they didn't, the ambulance company could be sued for lots and lots of money. Likewise, emergency rooms are legally obligated to treat anyone, regardless of ability to pay. (And the hospital absorbs the costs, which are enormous, so hospitals are closing their emergency rooms... It makes you wonder what's the point of having economists if we're just going to ignore them.) So I didn't quite understand how the described events could have happened. Anyone reporting sudden chest pains should have been hauled off by an ambulance instantly.
And this is where I fell down as a rationalist. I remembered several occasions where my doctor would completely fail to panic at the report of symptoms that seemed, to me, very alarming. And the Medical Establishment was always right. Every single time. I had chest pains myself, at one point, and the doctor patiently explained to me that I was describing chest muscle pain, not a heart attack. So I said into the IRC channel, "Well, if the paramedics told your friend it was nothing, it must really be nothing - they'd have hauled him off if there was the tiniest chance of serious trouble."
Thus I managed to explain the story within my existing model, though the fit still felt a little forced...
Later on, the fellow comes back into the IRC chatroom and says his friend made the whole thing up. Evidently this was not one of his more reliable friends.
I should have realized, perhaps, that an unknown acquaintance of an acquaintance in an IRC channel might be less reliable than a published journal article. Alas, belief is easier than disbelief; we believe instinctively, but disbelief requires a conscious effort.
So instead, by dint of mighty straining, I forced my model of reality to explain an anomaly that never actually happened. And I knew how embarrassing this was. I knew that the usefulness of a model is not what it can explain, but what it can't. A hypothesis that forbids nothing, permits everything, and thereby fails to constrain anticipation.
Your strength as a rationalist is your ability to be more confused by fiction than by reality. If you are equally good at explaining any outcome, you have zero knowledge.
We are all weak, from time to time; the sad part is that I could have been stronger. I had all the information I needed to arrive at the correct answer, I even noticed the problem, and then I ignored it. My feeling of confusion was a Clue, and I threw my Clue away.
I should have paid more attention to that sensation of still feels a little forced. It's one of the most important feelings a truthseeker can have, a part of your strength as a rationalist. It is a design flaw in human cognition that this sensation manifests as a quiet strain in the back of your mind, instead of a wailing alarm siren and a glowing neon sign reading "EITHER YOUR MODEL IS FALSE OR THIS STORY IS WRONG."
Loading…
Comments (72)
It's strange that it sounds like a rationalist is saying that he should have listened to his instincts. A true rationalist should be able to examine all the evidence without having to rely on feelings to make a judgment, or would be able to truly understand the source of his feelings, in which case it's more than just a feeling. The unfortunate thing is that people are more likely to remember the cases when they didn't listen to their feelings which ended up being correct in the end, than all the times when they were wrong.
The "quiet strain in the back of your mind" is what drives some people to always expect the worst to happen, and every so often they are right which reinforces their confidence in their intuitions more than their confidence diminishes each time they are wrong.
In some cases, it might be possible for someone to have a rational response to a stimulus only to think that it is intuition because they don't quite understand or aren't able to fully rationalize the source of the feeling. From my own experiences, it seems that some people don't make a hard enough effort to search for the source... they either don't seem to think that there is a rational source, or don't care to take the effort.... as long as they are able to ascertain what their feelings suggest they do, they really don't seem to care whether or not the source is rational or irrational.
A true rationalist would be able to determine the source and rationality of the feeling. The interesting question is if he fails to rationally explain the feeling, should he ignore the feeling, chalking it up to his weakness as a perfect rationalist.
Since we are all human and cannot be perfectly rational, shouldn't a rationalist decide that a seemingly irrational feeling is just that, irrational. Is it not more rational to believe that a seemingly irrational feeling is the result of our own imperfection as a human?
Anon, see Why Truth?:
When people think of "emotion" and "rationality" as opposed, I suspect that they are really thinking of System 1 and System 2 - fast perceptual judgments versus slow deliberative judgments. Deliberative judgments aren't always true, and perceptual judgments aren't always false; so it is very important to distinguish that dichotomy from "rationality". Both systems can serve the goal of truth, or defeat it, according to how they are used.
"I should have paid more attention to that sensation of still feels a little forced."
The force that you would have had to counter was the impetus to be polite. In order to boldly follow your models, you would have had to tell the person on the other end of the chat that you didn't believe his friend. You could have less boldly held your tongue, but that wouldn't have satisfied your drive to understand what was going on. Perhaps a compromise action would have been to point out the unlikelihood, (which you did: "they'd have hauled him off if there was the tiniest chance of serious trouble"), and ask for a report on the eventual outcome.
Given the constraints of politeness, I don't know how you can do better. If you were talking to people who knew you better, and understood your viewpoint on rationality, you might expect to be forgiven for giving your bald assessment of the unlikeliness of the report.
In it's strongest form, not believing system 1 amounts to not believing perceptions, hence not believing in empiricism. This is possibly the oldest of philosophical mistakes, made by Plato, possibly Siddhartha, and probably others even earlier.
Sounds like good old cognitive dissonance. Your mental model was not matching the information being presented.
That feeling of cognitive dissonance is a piece of information to be considered in arriving at your decision. If something doesn't feel right, usually either te model or the facts are wrong or incomplete.
T
"And this is where I fell down as a rationalist. I remembered several occasions where my doctor would completely fail to panic at the report of symptoms that seemed, to me, very alarming. And the Medical Establishment was always right. Every single time. I had chest pains myself, at one point, and the doctor patiently explained to me that I was describing chest muscle pain, not a heart attack. So I said into the IRC channel, "Well, if the paramedics told your friend it was nothing, it must really be nothing - they'd have hauled him off if there was the tiniest chance of serious trouble.""
My own "hold on a second" detector is pinging mildly at that particular bit. Specifically, isn't there a touch of an observer selection effect there? If the docs had been wrong and you ended up dying as a result, you wouldn't have been around to make that deduction, so you're (Well, anyone is) effectively biased to retroactively observe outcomes in which if the doctor did say you're not in a life threatening situation, you're genuinely not?
Or am I way off here?
A valid point, Psy-Kosh, but I've seen this happen to a friend too. She was walking along the streets one night when a strange blur appeared across her vision, with bright floating objects. Then she was struck by a massive headache. I had her write down what the blur looked like, and she put down strange half-circles missing their left sides.
That point was when I really started to get worried, because it looked like lateral neglect - something that I'd heard a lot about, in my studies of neurology, as a symptom of lateralized brain damage from strokes.
The funny thing was, nobody in the medical profession seemed to think this was a problem. The medical advice line from her health insurance said it was a "yellow light" for which she should see a doctor in the next day or two. Yellow light?! With a stroke, you have to get the right medication within the first three hours to prevent permanent brain damage! So we went to the emergency room - reluctantly, because California has enormously overloaded emergency rooms - and the nurse who signed us in certainly didn't seem to think those symptoms were very alarming.
The thing is, of course, that non-doctors are legally prohibited from making diagnoses. So neither the nurse on the advice line, or the nurse who signed us into the emergency room, were allowed to say: "It's a migraine headache, you idiots."
You see, I'd heard the phrase "migraine headache", but I'd had no idea of what the symptoms of a "migraine headache" were. My studies in neurology told me about strokes and lateral brain damage, because those are very important to the study of functional neuroanatomy. So I knew about these super dangerous and rare killer events that seemed sort of like the symptoms we were encountering, but I didn't know about the common events that a doctor sees every day.
When you see symptoms, you think of lethal zebras, because those are what you read about in the newspapers. The doctor thinks of much less exciting horses. This is why the Medical Establishment has always been right, in my experience, every single time I'm alarmed and they're not.
But in answer to your question about selection effects, Psy-Kosh, I think I'd have noticed if my friend had actually had a stroke. In fact, it would have been much more likely to have been reported and repeated than the reverse case.
Okie, and yeah, I imagine you would have noticed.
Also, of course, docs that habitually misdiagnose would presumably be sued or worse to oblivion by friends and family of the deceased. I was just unsure about the actual strength of that one thing I mentioned.
I think one would be the closest to truth by replying: "I don't quite believe that your story is true, but if it is, you should... etc" because there is no way for you to surely know whether he was bluffing or not. You have to admit both cases are possible even if one of them is highly improbable.
Doesn't any model contain the possibility, however slight, of seeing the unexpected? Sure this didn't fit with your model perfectly — and as I read the story and placed myself in your supposed mental state while trying to understand the situation, I felt a great deal of similar surprise — but jumping to the conclusion that someone was just totally fabricating is something that deserves to be weighed against other explanations for this deviation from your model.
Your model states that pretty much under all circumstances an ambulance is going to pick up a patient. This is true to my knowledge as well, but what happens if the friend didn't report to you that once the ambulance he called it off and refused to be transported. Or perhaps at the same time his chest pains were being judged as not-so-severe the ambulance got another call in that a massive car pileup required their immediate presence.
Your strength as a rationalist must not be the rejection of things unlikely in your model but instead the act of providing appropriate levels of concern. Perhaps the best response is something along the lines of "Sounds like a pretty strange occurrence. Are you sure your friend told you everything?" Now we're starting to judge our level of confidence in the new information being valid.
Which is honestly a pretty difficult model to shake as well. So much of every bit of information you build your world with comes from other people that I think it pretty decent to trust with some amount of abandon.
See antiprediction.
That's certainly sensible, and in But There's Still a Chance Eleiezer makes examples where this seems strong. In the above example, it depends a whole lot on how much belief you have in people (or, rather, lines of IRC chat).
I think then that your strength as a rationalist comes in balancing that uncertainty against some your prior trust in people. At which point, instead of predicting the negative, I'd seek more information.
The level of "trust" you have in a person should be inversely proportional to the sensationalism of the claim that he's making.
If a person tells you he was abducted by a UFO, you demand evidence.
If a person tells you that on the way to work he slipped and fell down, and you have no concrete reason to doubt the story in particular or the person in general, you take that at face value. It is a reasonable assumption that a perfect stranger in all likelihood will NOT be delusional or a compulsive liar.
DP
That makes sense if you're only evaluating complete strangers. In other words, your uncertainty about the population-inferred trustworthiness of a person is pretty high and so instead the mere (Occam Factor style) complexity of their statement is the overruling component of your decision.
In the stated case, this isn't a totally random stranger. I feel quite justified in having a less-than uninformative prior about trusting IRC ghosts. In this case, my rationally acquired prejudice overrules in inference about the truth of even somewhat ordinary tales.
The author did not mention anything about an exceptionally high percentage of liars in IRC relative to the general population (which would be quite relevant to his statement) therefore there's no reason to believe that such had been HIS experience in the past.
Given that, there is no reason for HIM to presume that the percentage of compulsive liars in IRC would different from the general population. YOUR experiences may, of course, be drastically different, but they are not the subject of discussion here.
DP
I don't see that you did anything at all irrational. You're talking to a complete stranger on the internet. He doesn't know you, and cannot have any possible interest in deceiving you. He tells you a fairly detailed story and asks for you advice. For him to make the whole thing up just for kicks is an example of highly irrational and fairly unlikely behavior.
Conversely, a person's panicking over chest pains and calling the ambulance is a comparatively frequent occurrence. Your having read somewhere something about ambulance policies does not amount to having concrete, irrefutable knowledge that an ambulance crew cannot make an on-site determination that there's no need to take a person to the hospital. To a person without extensive medical knowledge there is nothing particularly unlikely about the story you were told.
Therefore, the situation is this -- you are told by a complete stranger that has no reason to lie to you a perfectly believable story. You have no concrete reason ("read something somewhere" does not qualify) to doubt either the story or the man's sanity. Thus there is nothing illogical about taking the story at face value. You did the perfectly rational thing.
Since there was no irrationality in your initial behavior, the conclusions that you arrive at further in your post are unfounded.
DP
You're talking to a complete stranger on the internet. He doesn't know you, and cannot have any possible interest in deceiving you.
There's plenty of evidence that some people (a smallish minority, I think) will deceive strangers for the fun of it.
Which, as I said later on in the same paragraph, is irrational and unlikely behavior. Therefore, when lacking any factual evidence, the reasonable presumption is that that's not the case.
DP
I think many of us have actually encountered liars on the Internet. I'm not sure what you mean when you say "lacking any factual evidence".
I presume that you have encountered liars in the real world as well. Do you, on that basis, habitually assume that a random stranger engaging in casual conversation with you is a liar?
My point is that pathological liars are a small minority. So if you're dealing with a person that you know absolutely nothing about, and who does not have any conceivable reason to lie to you, there is nothing unreasonable in assuming that he's telling you the truth, unless you have factual evidence (i.e. you have accurate, verifiable knowledge of ambulance policies) that contradicts what he's saying.
DP
I think at this point the questions have become (a) "how many bits of evidence does it take to raise 'someone is lying' to prominence as a hypothesis?" and (b) "how many bits of evidence can I assign to 'someone is lying' after evaluating the probability of this story based on what I know?"
I believe your argument is that a > b (specifically, that a is large and b is small), where the post asserts that a < b. I'm not going to say that's unreasonable, given that all we know is what Eliezer Yudkowsky wrote, but often actual experience has much more detail than any feasible summary - I'm willing to grant him the benefit of the doubt, given that his tiny note of discord got the right answer in this instance.
My argument is what I stated, nothing more. Namely that there is nothing unreasonable about assuming that a perfect stranger that you're having a casual conversation with is not trying to deceive you. I already laid out my reasoning for it. I'm not sure what more I can add.
DP
Wait, why not?
I read somewhere that if spin about and click my heels 3 times I will be transported to the land of Oz. Does that qualify as a concrete reason to believe that such a land does indeed exist?
DP
That indeed serves as evidence for that fact, though we have much stronger evidence to the contrary.
N.B. You do not need to sign your comments; your username appears above every one.
And not just because clicking the heels three times is more canonically (and more often) said to be way to return to Kansas from Oz. and not to Oz.
So the fact that something was written somewhere is sufficient to meet your criteria for considering it evidence? I take it you have actually tried clicking your heels to check whether or not you would be teleported to Oz then?
Also, does my signing my comments offend you?
DP
It hurts aesthetically by disrupting uniformity of standard style.
Fair enough. It's a habit of mine that I'm not married to. If members of this board take issue with it, I can stop.
Yes. It's really sucky evidence.
This doesn't remotely follow and is far weaker evidence than other available sources. For a start, everyone knows that you get to Oz with tornadoes and concussions.
It makes you look like an outsider who isn't able to follow simple social conventions and may have a tendency towards obstinacy. (Since you asked...)
"This doesn't remotely follow and is far weaker evidence than other available sources. For a start, everyone knows that you get to Oz with tornadoes and concussions."
Let's not get bogged down in the specific procedure of getting to Oz. My point was that if you truly adapt merely seeing something written somewhere as your standard for evidence, you commit yourself to analyzing and weighing the merits of EVERYTHING you read about EVERYWHERE. Do you mean to tell that when you read a fairy tale you truly consider whether or not what's written there is true? That you don't just dismiss it offhand without giving it a second thought?
"It makes you look like an outsider who isn't able to follow simple social conventions and may have a tendency towards obstinacy. (Since you asked...)"
Like I said above to Vladimir, it's not a big deal, but you're reading quite a bit into a simple habit.
The fact that something is really written is true; whether it implies that the written statements themselves are true is a separate theoretical question. Yes, ideally you'd want to take into account everything you observe in order to form an accurate idea of future expected events (observable or not). Of course, it's not quite possible, but not for the want of motivation.
Well I didn't think I needed to clarify that I'm not questioning whether or not something that's written is really written. Of course, I'm questioning the truthfulness of the actual statement.
Or not so much it's truthfulness, but rather whether or not it can be considered evidence. Though I realize that you take issue with arguing over word definitions, to me the word evidence has certain meaning that goes beyond every random written sentence, whisper or rumor that you encounter.
The fact that something is written, or not written, is evidence about the way world is, and hence to some extent evidence about any hypothesis about the world. Whether it's strong evidence about a given hypothesis is a different question, and whether the statement written/not written is correct is yet another question.
(See also the links from this page.)
Around these parts, a claim that B is evidence for A is a taken to be equivalent to claiming that B is more probable if A is true than if not-A is true. Something can be negligible evidence without being strictly zero evidence, as in your example of a fairy story.
No, you can acknowledge that something is evidence while also believing that it's arbitrarily weak. Let's not confuse the practical question of how strong evidence has to be before it becomes worth the effort to use it ("standard of evidence") with the epistemic question of what things are evidence at all. Something being written down, even in a fairy tale, is evidence for its truth; it's just many orders of magnitude short of the evidential strength necessary for us to consider it likely.
Vladimir, Cyan, and jimrandomh, since you essentially said the same thing, consider this reply to be addressed to all three of you.
Answer me honestly, when reading a fairy tale, do you really stop to consider what's written there, qualify its worth as evidence, and compare it to everything else you know that might contradict it, before making the decision that the probability of the fairy tale being true is extremely low? Do you really not just dismiss it offhand as not true without a second thought?
When I pick up a work of fiction, I do not spend time assessing its veracity. If I read a book of equally fantastic claims which purports to be true, I do spend a little time. You might want to peruse bounded rationality for an overview.
No, but only because that would be cognitively burdensome. We're boundedly rational.
Immediate observation is only that something is written. That it's also true is a theoretical hypothesis about that immediate observation. That what you are reading is a fairy tale is evidence against the things written there being true, so the theory that what's written in a fairy tale is true is weak. On the other hand, the fact that you observe the words of a given fairy tale is strong evidence that the person (author) whose name is printed on the cover really existed.
An alternative explanation? You put your energy into solving a practical problem with a large downside (minimizing the loss function in nerdese). Yes, to be perfectly rational you should have said: "the guy is probably lying, but if he is not then...".
I wouldn't call it a flaw; blaring alarms can be a nuisance. Ideally you could adjust the sensitivity settings . . . hence the popularity of alcohol.
Thank you, Eliezer. Now I know how to dissolve Newcomb type problems. (http://lesswrong.com/lw/nc/newcombs_problem_and_regret_of_rationality/)
I simply recite, "I just do not believe what you have told me about this intergalactic superintelligence Omega".
And of course, since I do not believe, the hypothetical questions asked by Newcomb problem enthusiasts become beneath my notice; my forming a belief about how to act rationally in this contrary-to-fact hypothetical situation cannot pay the rent.
This sort of brings to my mind Pirsig's discussions about problem solving in ZATAOMM. You get that feeling of confusion when you are looking at a new problem, but that feeling is actually a really natural, important part of the process. I think the strangest thing to me is that this feeling tends to occur in a kind of painful way -- there is some stress associated with the confusion. But as you say, and as Pirsig says, that stress is really a positive indication of the maturation of an understanding.
I'm not sure that listening to ones intuitions is enough to cause accurate model changes. Perhaps it is not rational to hold a single model in your head, as your information is incomplete. Instead one can consciously examine the situation from multiple perspectives, in this way the nicer (simpler, more consistent, whatever your metric is) model response can be applied. Alternatively you could legitimately assume that all the models you hold have merit and produce a response that balances their outcomes e.g. if your model of the medical profession is wrong and they die from your advice it is much worse than the unnecessary calling of the ambulance (letting the medical profession address the balance of resources). This would lead a rational person to simultaneously believe many contradictory perspectives and act as if they were all potentially true. Does anyone know of any theory in this area? the modelling of models (and efficiently resolving multiple models) would be very useful in AI.