I felt a similar pang of dissent when I read Feeling Rational for the first time, and I couldn't quite put my finger on it. After reading your comment and thinking on it, I think I'll formulate it thus: "A distinction should be made between seeking comfort and seeking comfort in false beliefs. The former is acceptable and the latter is unacceptable."
For me, some of Eliezer's words at the 'How Should Rationalists Approach Death' panel at Skepticon 4 were brought to mind:
I guess the most important message I might have to give on the topic of how to deal with death is: It's alright to be angry; it's alright not to be comforted. For me, the prototype of this experience will always be the funeral of my little brother, who died at the age of 19. I went to his funeral; I was the only open atheist there, I believe. And, for me, there was no confusion in that experience. Pure anger. Pure wishing-it-didn't-happen. No need to seek comfort for it. And that may not have been a pleasant experience, but I think that it was, in a fundamental sense, more healthy, for being less conflicted, than what I saw on the faces of my relatives, and my parents, as they tried to attribute it to God.
People often seek comfort in false beliefs when faced with the death of themselves or others, and the implication is that this is very harmful when you take cryonics seriously like I do. Since this was the context of the panel, I can see how publicly encouraging people to avoid those false beliefs is instrumentally rational. But those words didn't completely mesh with me, because it felt like he was encouraging people to suffer more than they needed to if they didn't have false beliefs. It didn't feel like the difference between seeking comfort and believing false but comforting things had been adequately distinguished.
It's entirely possible that I just harmfully misinterpreted his words, but I should say that I had some difficulty when my mother died because I felt like it was Bad for me not to realize and feel, and to not want to realize and feel, on a gut level, how much value was destroyed when she died. That not feeling it, and not wanting to feel it, was simply inaccurate, and that I was making a mistake that I should feel bad about because my emotions were inaccurate; wrong; Bad. But there's a difference between the belief and the affect. There's a sense in which System 1 doesn't even know that my mother is dead. I haven't let it, because it does no good. The conventional wisdom is that this is suppressing Feelings That Need To Be Felt, but I don't feel generally stressed, like I'm holding something in that I need to get out. I'm never going to be okay with it in the sense that it will be congruent with my values. And viscerally feeling it just hurts. System 2 can have accurate verbal beliefs about death and cryonics without System 1's unintended side effect of thrashing around trying to tell me that something is very wrong with one of my parent-child relationship bonds. System 2 can have accurate beliefs without System 1 tracking their emotional implications. Feeling death on a gut level is only instrumentally valuable if you have false beliefs about death and longevity is a term in your utility function. Otherwise, it's just torture.
But it's also pretty clear to me that Eliezer doesn't want people to suffer, so I want to be charitable and consider how he might come to the sorts of conclusions and say the sorts of things that he has. My hypothesis is that, for Eliezer, cognitive consonance is its own comfort. The idea that cognitive dissonance is viscerally upsetting for him fits my model well. For him, perceiving his beliefs and emotions as congruent is as comforting as having true beliefs and regulating my emotions out of existence is for me. Later on, in speech and writing, the comfort of cognitive consonance and the instrumental value of viscerally feeling the evil of death when your beliefs about death are false are conflated into one larger and less precise point.
And sometimes we must learn to remove certain facts from our mind, so they are only called upon when necessary, else they will severely depress us.
Yes, your summary line at the top feels like exactly the distinction I was trying to make. Rereading my comment, I think I was attacking the straw man position, that after admitting that EY had said rationality and feelings were not orthogonal, I was arguing as though he'd said they were the opposite of that - directly related, parallel, etc.
You could explain Eliezer's non-standard approach in terms of his being fundamentally different to many in that regard; or you could explain it as depending more heavily on his situation: he was a highly intelligent at...
This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.
Welcome to the Rationality reading group. This week we discuss the Preface by primary author Eliezer Yudkowsky, Introduction by editor & co-author Rob Bensinger, and the first sequence: Predictably Wrong. This sequence introduces the methods of rationality, including its two major applications: the search for truth and the art of winning. The desire to seek truth is motivated, and a few obstacles to seeking truth--systematic errors, or biases--are discussed in detail.
This post summarizes each article of the sequence, linking to the original LessWrong posting where available, and offers a few relevant notes, thoughts, and ideas for further investigation. My own thoughts and questions for discussion are in the comments.
Reading: Preface, Biases: An Introduction, and Sequence A: Predictably Wrong (pi-xxxv and p1-42)
Introduction
Preface. Introduction to the ebook compilation by Eliezer Yudkowsky. Retrospectively identifies mistakes of the text as originally presented. Some have been corrected in the ebook, others stand as-is. Most notably the book focuses too much on belief, and too little on practical actions, especially with respect to our everyday lives. Establishes that the goal of the project is to teach rationality, those ways of thinking which are common among practicing scientists and the foundation of the Enlightenment, yet not systematically organized or taught in schools (yet).
Biases: An Introduction. Editor & co-author Rob Bensinger motivates the subject of rationality by explaining the dangers of systematic errors caused by *cognitive biases*, which the arts of rationality are intended to de-bias. Rationality is not about Spock-like stoicism -- it is about simply "doing the best you can with what you've got." The System 1 / System 2 dual process dichotomy is explained: if our errors are systematic and predictable, then we can instil behaviors and habits to correct them. A number of exemplar biases are presented. However a warning: it is difficult to recognize biases in your own thinking even after learning of them, and knowing about a bias may grant unjustified overconfidence that you yourself do not fall pray to such mistakes in your thinking. To develop as a rationalist actual experience is required, not just learned expertise / knowledge. Ends with an introduction of the editor and an overview of the organization of the book.
A. Predictably Wrong
1. What do I mean by "rationality"? Rationality is a systematic means of forming true beliefs and making winning decisions. Probability theory is the set of laws underlying rational belief, "epistemic rationality": it describes how to process evidence and observations to revise ("update") one's beliefs. Decision theory is the set of laws underlying rational action, "instrumental rationality", independent of what one's goals and available options are. (p7-11)
2. Feeling rational. Becoming more rational can diminish feelings or intensify them. If one cares about the state of the world, it is expected that he or she should have an emotional response to the acquisition of truth. "That which can be destroyed by the truth should be," but also "that which the truth nourishes should thrive." The commonly perceived dichotomy between emotions and "rationality" [sic] is more often about fast perceptual judgements (System 1, emotional) vs slow deliberative judgements (System 2, "rational" [sic]). But both systems can serve the goal of truth, or defeat it, depending on how they are used. (p12-14)
3. Why truth? and... Why seek the truth? Curiosity: to satisfy an emotional need to know. Pragmatism: to accomplish some specific real-world goal. Morality: to be virtuous, or fulfill a duty to truth. Curiosity motivates a search for the most intriguing truths, pragmatism the most useful, and morality the most important. But be wary of the moral justification: "To make rationality into a moral duty is to give it all the dreadful degrees of freedom of an arbitrary tribal custom. People arrive at the wrong answer, and then indignantly protest that they acted with propriety, rather than learning from their mistake." (p15-18)
4. ...what's a bias, again? A bias is an obstacle to truth, specifically those obstacles which are produced by our own thinking processes. We describe biases as failure modes which systematically prevent typical human beings from determining truth or selecting actions that would have best achieved their goals. Biases are distinguished from mistakes which originate from false beliefs or brain injury. Do better seek truth and achieve our goals we must identify our biases and do what we can to correct for or eliminate them. (p19-22)
5. Availability. The availability heuristic is judging the frequency or probability of an event by the ease with which examples of the event come to mind. If you think you've heard about murders twice as much as suicides then you might suppose that murder is twice as common as suicide, when in fact the opposite is true. Use of the availability heuristic gives rise to the absurdity bias: events that have never happened are not recalled, and hence deemed to have no probability of occurring. In general, memory is not always a good guide to probabilities in the past, let alone to the future. (p23-25)
6. Burdensome details. The conjunction fallacy is when humans rate the probability of two events has higher than the probability of either event alone: adding detail can make a scenario sound more plausible, even though the event as described necessarily becomes less probable. Possible fixes include training yourself to notice the addition of details and discount appropriately, thinking about other reasons why the central idea could be true other than the added detail, or training oneself to hold a preference for simpler explanations -- to feel every added detail as a burden. (p26-29)
7. Planning fallacy. The planning fallacy is the mistaken belief that human beings are capable of making accurate plans. The source of the error is that we tend to imagine how things will turn out if everything goes according to plan, and do not appropriately account for possible troubles or difficulties along the way. The typically adequate solution is to compare the new project to broadly similar previous projects undertaken in the past, and ask how long those took to complete. (p30-33)
8. Illusion of transparency: why no one understands you. The illusion of transparency is our bias to assume that others will understand the intent behind our attempts to communicate. The source of the error is that we do not sufficiently consider alternative frames of mind or personal histories, which might lead the recipient to alternative interpretations. Be not too quick to blame those who misunderstand your perfectly clear sentences, spoken or written. Chances are, your words are more ambiguous than you think. (p34-36)
9. Expecting short inferential distances. Human beings are generally capable of processing only one piece of new information at at time. Worse, someone who says something with no obvious support is a liar or an idiot, and if you say something blatantly obvious and the other person doesn't see it, they're the idiot. This is our bias towards explanations of short inferential distance. A clear argument has to lay out an inferential pathway, starting from what the audience already knows or accepts. If at any point you make a statement without obvious justification in arguments you've previously supported, the audience just thinks you're crazy. (p37-39)
10. The lens that sees its own flaws. We humans have the ability to introspect our own thinking processes, a seemingly unique skill among life on Earth. As consequence, a human brain is able to understand its own flaws--its systematic errors, its biases--and apply second-order corrections to them. (p40-42)
It is at this point that I would generally like to present an opposing viewpoint. However I must say that this first introductory sequence is not very controversial! Educational, yes, but not controversial. If anyone can provide a link or citation to one or more decent non-strawman arguments which oppose any of the ideas of this introduction and first sequence, please do so in the comments. I certainly encourage awarding karma to anyone that can do a reasonable job steel-manning an opposing viewpoint.
This has been a collection of notes on the assigned sequence for this week. The most important part of the reading group though is discussion, which is in the comments section. I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!
The next reading will cover Sequence B: Fake Beliefs (p43-77). The discussion will go live on Wednesday, 6 May 2015 at or around 6pm PDT, right here on the discussion forum of LessWrong.