I get an uncomfortable feeling, Eliezer, that this work is to ultimately lead to a mechanism to attract:
people of libertarian bent
people interested in practically unbounded longevity of consistent, continual consciousness
and also lead to a mechanism to tar people disinclined to those two goals; tar them with the label "sentimentally irrational".
Rationality to me is simply a tool. I would have absolutely no confidence in it without the ongoing experiences of applying it iteratively, successfully to specific goals.
...And of course, no matter
The success of science was and is because it is useful, and similarly for rationalism. But one of the critiques of rationalism and of the overcoming-bias program is that it is sometimes counterproductive. The unbiased tend to be unhappy and/or insane. If someone's goals are to be happy and successful in life, he does best not to be fully rational. Irrationality is the most useful policy if these are your goals.
Your argument suggests that this is true only because this is setting the goalposts too low. For someone who merely seeks happiness, yes, irrational...
I rarely post, only read in hopes of learning. Today, I comment: I appreciate the beauty of this post.
Thank you, Eliezer.
I am often confused by your writing, because I don't see where you have "skin in the game". Where are you exercising your tools of rationality?
If I'd went ahead and said that within the post, it would've transformed a piece on rationality into overt propaganda, destroying its internal aesthetics. Read my website.
Perhaps I am one of the "sentimentally irrational," but I would pick the 400 certain lives saved if it were a one-time choice, and the 500 @ 90% if it were an iterated choice I had to make over, and over again. In the long run, probabilities would take hold, and many more people would be saved. But for a single instance of an event never to be repeated? I'd save the 400 for certain.
Your 80% and 90% figures don't really add up either. You don't describe how many people in total will die, regardless of you decision. If the max death number poss...
Caledonian, I think you're misreading him. He's not saying: the cause is the one thing you never think rationally about. He's saying: the cause is good (rationally good) and to protect/preserve it you have to pull yourself into conformance with the real world, because that's where the action is. To achieve that you have to hold up what you (perhaps mistakenly) think of as "reason" against the real world, and be prepared to re-evaluate if it doesn't work. What your re-evaluation seeks is better techniques of reason - not to throw reason away.
"Rationality by its nature cannot be only a means towards an end."
Rationality is conformance to reality. You can conform to reality for a cause. (You're saying, you can't mold reality to your cause - I agree, but that's not what he was meaning.) He was meaning that people have thought themselves rational when applying formal, skillful, pedigreed academic techniques that DON'T WORK, such as Jesuit style casuistry. So you have to hold the technique up against reality. You won't do that if you put the technique first by saying "I serve reason&q...
Julian, Caledonian is a well-known troll on OB. We've decided against censorship for now, but you might not want to waste too much time. I generally don't respond to Caledonian unless I see someone else agreeing with him.
I totally agree with "Anon", and others who made similar points in the Circular Altruism post. Context matters! Is it a one-time choice, or an iterated choice? Is there an upper limit to the number of deaths, or no limit? Are the 500 the number of people on the sinking ship/last people on planet earth, or possible victims from a much larger pool? You can only do the math and make a rational decision when you have ALL the numbers from the relevant context.
The first steps of rationality lie not in separating problems from their context, but in determining what context is relevant.
I don't have anything desperately important to me, and you say I'm not allowed to just pick something. Given this, what am I supposed to do, to become more rational? Am I just doomed? I really desperately want to believe true things and not false things, but you say that's not good enough.
Good question, Nominull. Unfortunately I lack the ability to answer your question from personal experience. Mine just fell into my lap.
But is believing true things what you most desperately want, in all the world?
Caledonian, I gather Eliezer put "rationality" in quotes because people may believe they are committed to rationality when in fact they are not. If they have a goal which is contingent on rationality that will help them from straying from the path.
What he said immediately after the part you mention was: "The Art must have a purpose other than itself, or it collapses into infinite recursion"
He wasn't talking about pseudo-rationality. When he talks about "The Art", he's talking about rationality.
And he's wrong: truth points to itself.
Anon: do you suggest that others follow your policy as well? Then when many people have individual made isolated choices like that, far fewer lives will have been saved. And in the whole history of the world, choices like that must have been made many times. Why does it matter whether it is you who are repeating the choice or other people?
The question about whether the 500 are that last people in the world is adding other utilities into the issue, such as preserving the human race, and so on. In that case you have a different comparison; naturally, you may have to consider other factors besides the utility of the lives. But as long as you consider only the lives, Eliezer is right.
Caledonian: "I think rationality has to be the starting point."
Can you expand on this? A rationalistic moral relativist might say that actions require goals, ultimate goals are arbitrary, and so rationality cannot be the starting point there. In the real world, by the time one is able to entertain ideas like 'choosing to be more rational', you're already going to have goals, preferences, ideas about how you should live your life. So it could be countered that 'rationality' never has to supply everything; its purpose will largely be to critique ex...
Anon, Wendy:
Certainly finding out all of the facts that you can is good. But rationality has to work no matter how many facts you have. If the only thing you know is that you have two options:
i.e.
Any takes for #2? I seem to remember Ben Jones saying he would choose #1 in a case similar to the second case.
Formerly, I think I would have chosen #2 in the first case and #1 in the second. But Eliezer has converted me. Now I choose #2 in both cases. But would he do that himself? Consider:
"Perhaps I am one of the 'sentimentally irrat...
"It takes visceral panic, channeled through cold calculation, to cut away all the distractions." - this just made it to my quotes file.
If i understand Eliezer's point correctly in terms of the map/territory analogy, what he says is that having somewhere to go and actually needing to put your map to use will motivate you to make that map as accurate as possible, if you care about your destination more than you 'believe in' the current iteration of your map and/or the techniques used to derrive it.
A rationalistic moral relativist might say that actions require goals, ultimate goals are arbitrary, and so rationality cannot be the starting point there.
Lots of things act without having any sort of goals. Does fire have a goal of reducing high-energy compounds into oxidized components and free energy? No, but it does it anyway.
You can limit 'action' to intentional events only, I suppose.
However, how does declaring that goals are arbitrary rule out assertions about necessary starting points?
So it could be countered that 'rationality' never has to su...
Formerly, I think I would have chosen #2 in the first case and #1 in the second. But Eliezer has converted me. Now I choose #2 in both cases. But would he do that himself?
Isn't that implicitly what he does for a living? Eliezer could become a firefighter or emergency medical technician, or work for clean drinking water in rural Africa, with a near-certainty of preventing several deaths in the next year. Meanwhile, there is a very small chance of someone creating an non-Friendly AI in the next year. We can argue about the probabilities (of the problem arising, of successfully presenting it), but Eliezer has already chosen the existential threat.
"So I hereby retract my argument against voting, Pascal's Mugging, and Pascal's Wager. In the particular Mugging we discussed, there may have been anthropic reasons to make it proportionally improbable. But without such reasons, it should be accepted."
I'm certainly glad you think so, Unknown, because I was just contacted by the Dark Lords of the Matrix. It turns out that we are living in a simulation. I have no idea what the physics of the world outside are like, but they're claiming that unless you personally send $100 to SIAI right now, they're...
(same anon from above who asked about the context of the 400/500 problem being an issue)
In response to GreedyAlgorithm who said:
Certainly finding out all of the facts that you can is good. But rationality has to work no matter how many facts you have. If the only thing you know is that you have two options:
Z. M. Davis, given the existence of that many people, and given that threat, the probability that I personally would be the one threatened in that must be multiplied by one over the number of people, since it could have been anyone else. So the expected disutility from your mugging is one dust speck multiplied by the probability that the Matrix scenario is actually true. This probability is very low, and even if it were unity, the disutility of one dust speck isn't going to get me to pay $100.
So again, I said "without such reasons, it should be accept...
It's probably just that I'm stupid, but I don't understand the anthropic solution to Pascal's Mugging. Why does it matter that other people could have been asked? What if it were stipulated that the mugger threatens everyone?
Maybe I should actually study Kolmogorov complexity before trying to grapple with such matters.
Viz. the dilemma posed in Circular Altruism, what should we do? When forced to "Shup up and multiply", we have forgone our intuitions and picked the choice based upon our mathematics. However, we are not just overcoming our own intuitions, but also the intuitions of everyone who does not simply "Shut up and multiply". We are held accountable not by those who knows the math, but by those who have intuitions like ourselves.
If we save everyone, we are heroes. If we do not, we are held accountable not for the math, but for the very intuitio...
So, is your point that we need a cause against which to evaluate the success of our mathematics? That perhaps this sort of feedback that, persumably, you encounter on a daily basis, is something that does not come through rationality itself, but through the very real feedback of what you have chosen to protect?
I guess my previous post was a reflection that I am just a budding rationalist, and also that my skills have not been sharpened against the proper stone.
So, is your point that we need a cause against which to evaluate the success of our mathematics? That perhaps this sort of feedback that, persumably, you encounter on a daily basis
I'm not going to get feedback on my final success or failure for, oh, probably at least another 10 years.
My point, rather, was that your post illustrated very clearly why rationality comes from having something to protect - you thought of doing something rational, but worried about the other people whose intuitions differed from yours, and what they might think of you. So that worry is a force binding you to the old way of thinking.
But if the thing you were protecting was far more important than what anyone thought of you, that wouldn't slow you down. This isn't about iconoclasm - it's about an inertial drag exerted by all the little fears and worries, an inertial drag of the way that you or others previously did things; the motivating force has to be more powerful than that, or you won't move.
"The point is that given this information, rationality picks choice 2." - Posted by: GreedyAlgorithm
Sorry, no. Given this information, rationality says that there is not enough information to make an appropriate decision, and demands to know the context. If contextual information isn't available, rationality will say that either option 1 or 2 may be right, depending on circumstances.
Rationality never dismisses context as irrelevant just because it isn't known. If unknown factors make the right answer uncertain, then you must accept that it is unc...
For some reason this post reminds me of the Buddhist parable asceticsim now, nymphs later.
I don't think it's all that uncommon to begin cultivating an art for some specific purpose, proceed to cultivate it largely for its own sake, and eventually to abandon the original purpose.
Under Multiple Worlds, aren't you condemned, whatever you do or don't do, to there being a number tending to infinity of worlds where what you want to protect is protected, and a number tending to infinity where it is not ?
Caledonian: Let's distinguish between the aesthetics of rationality and the pragmatics of rationality. Is my model of the world consistent, do my goals make sense - that's pragmatics. Aesthetics is by comparison nebulous and subtle, but perhaps it encompasses both admiration for the lawlike nature of reality and self-admiration for one's own relationship to it. :-)
It seems to me that you are taking issue with the idea that the pragmatics of rationality should be trumped by a higher cause. This essay says nothing about that. It says, first, that it's a psyc...
You are not alone, Z. M. Davis: I disagree with Eliezer over whether Robin's anthropic solution is a satisfactory solution to Pascal's Mugging. (Eliezer repeated his endorsement of Robin's anthropic solution here a few weeks ago.) Since I started reading Eliezer 6 years ago, this is the first time I can recall disagreeing with him on a question of fact. (As I have pointed out many times in the comments here, I disagree with him significantly on terminal values.) If anyone wants to reply to this, I humbly suggest doing so by clicking on my name below.
For those saying they have nothing to protect or still need to find something to protect, remember that you are human and, unless you have no natural family or reproductive ties, you always have the people you love to protect. It may seem counterintuitive if you've bought into Hollywood rationality, but love is a powerful motivational force. If you think that, in theory, being more rational is good, but don't see how you can effect greater rationality in your mind, consider the many benefits of your increased rationality (again, not Hollywood rationality...
Excellent point by Worley. Since I have assumed the role on this blog of pointing out that happiness is not the meaning of life, let me hasten to add that happiness is a very useful barometer. Whether you are happier on average now than you were 10 years ago is for example probably a more reliable barometer of whether your life is on a better track than it was 10 years ago than change in financial net worth over those 10 years (though net worth is an important barometer too). And the one situation in which happiness is least likely to steer you wrong is...
Hollerith, if 'most psychologists are idiots', I wonder how they discovered all the cognitive biases ?
if 'most psychologists are idiots', I wonder how they discovered all the cognitive biases
He said 'most', not 'all'. And just because someone is an idiot doesn't mean everything they do is wrong. Even Freud managed to do some good descriptive work before descending into madness and delusion.
I mentioned psychologists in a particular context, namely, how to apply the skills of rationality to the project of nuturing and supporting your friends, lovers and family. Worley and I think rationality can be applied to that project. But I thought just leaving it at that would mislead some of the readers who have not had a lot of practical experience in life: unlike many of the other projects rationality is typically applied to, this project is different in that you cannot just travel to your nearest bookstore and by browsing the shelves expect to fin...
Caledonian: "I don't believe I've ever heard anyone speak of the aesthetic aspects of rational thought before."
It's funny - the phrase "aesthetics of rationality" appears in the final paragraph of Eliezer's post; apparently it's what the whole thing was about. But I didn't notice it either, until I was seriously casting about for some way to show that Caledonian person why their criticism was off the mark. I think Eliezer's point may be something like this: the aesthetics of rationality are all that could truly make it an end in itself;...
Aesthetics are rarely a topic when rationality is discussed. Mostly because they're only relevant to ancient-Greek-style thought.
On the list of things likely to cause unreasonable attachment, it's pretty far down. Love of familiarity, wanting to appear intelligent to others, wanting to appear intelligent to oneself, unwillingness to face conclusions that one finds unpalatable, general inflexibility... these are all plausible causes of failure. But aesthetics?
I think you're pretty close to the core of this one. You identified that having something to protect gives you strength. And having a worthy cause to work for, for the same reason.
But what is that reason? What is it that gives you strength? What is the underlying cause of us gaining strength from certain causes?
I'm not certain I understand the topic well enough myself, but I think I have something that you might find insightful here.
Moral Idealism. That's where your power comes from. Whether you're fighting to protect a loved one, or you're fighting to pro...
Personally, I find aesthetic purity to be a very strong source of attachment for me. It's certainly caused 'unreasonable attachments', like being stuck on being "right" and ascribing a purity to it (eg. I am right about this and you are wrong, therefore I will absolutely refuse to do this small nitpicky thing and I don't care if I jam up the whole process because it's MORALLY WRONG not to do so. I am the lone voice of dissent!). Oh, school..
I came across the same hack, or coping trick. Just remap the definition of what you're being pure about to "winning" or "rationality".
Pretty sure I'm displaying that I missed the point somehow.
The proper choice between (1) certainly save 400 lives and (2) 90% probability of saving 500 lives with 10% probability of saving no lives, depends on your utility function, which depends on the circumstances. If your utility is proportional to the number of lives saved, then sure, go with (2).
On the other hand, suppose that some cataclysm has occurred, those 500 lives are all that remains of the human race, and extinction of the human race has such an extremely negative utility for you that all other considerations amount to rounding error in the utility...
All right, I'd like to attempt a summary to make sure that I am understanding this post, if anyone see's some mistake in my interpretation, I'd appreciate it if they let me know.
Virtually everyone wants their beliefs to be true, this amounts to practically everyone wants to be epistimically rational. Rationality is a rare trait, so obviously that desire is not enough to make you epistimically rational. But that desire mixed with the rare desire to have all of your beliefs make useful predictions about whatever they talk about, is enough, provided that you...
I think it ought to be made explicit in the first scenario that 100 lives are being lost with certainty, because it's not necessarily implied by the proposition. I know a lot of people inferred it, but the hypothetical situation never stated it was 400/500, so it could just as easily be 400/400, in which case choosing it would certianly be preferable to the second option. I think it's important you make your hypothetical situations clear and unambiguous. Besi...
You will never master rationality well enough to win with any weapon, if you do not appreciate the beauty for its own sake.
I have a low prior for this statement, but I don't have any data. I wonder why Eliezer thinks this is the case.
Here I have a question that is slightly unrelated, but I'm looking for a good cognitive science science fair project and I'm having trouble thinking of one that would be not completely impractical for a high-schooler to do, won't take more than a few months, and would be interesting enough to hold people's attention for at least a few minutes before they head off to the physics and medical research projects. No one ever does decent cognitive science projects and I really want to show them that this branch of science can be just as rigorous and awesome as the other ones. Does anyone have any ideas?
I've been coming back to this post for 7 years or so, and the whole time it's obvious that I don't have something to protect, and haven't found one, and haven't yet found a way to find something to protect. It seems pretty cool though - and accurate that people who really care about things are able to go to great lengths to improve the way they think about the thing and their ability to to solve it.
I can say that once I realized I cared about wanting to care about something, that helped me quite a bit and I started improving my life.
Very interesting. I can't help feeling that "trying to be a better rationalist" is somehow a paradoxical aim.
Roughly speaking I would say that we have preferences, and their is no rational way of picking preferences. If you prefer pizza to icecream, or pleasure to pain, or living to dying, then that is that. Rationality is a mechanism for effectively seeking your preferences, ordering pizza, not putting your had in a fire etc. You can't pick rational preferences (goals), you can pick a rational route towards those goals.
If you adopt "I want to be more rati...
I savor the succulent choleric chaos of declaring that I value mere phlegm above yellow bile. That is almost a contradiction, but not quite; and the resulting blend has a choleric quality as well: a delicious humor.
Lessons from experimenting prove to be more valuable than from Authority? I think that Adam and Eve would beg to differ. I know, mentioning them probably disqualifies me as fertile ground for rationalist seeding, huh? Oh well, can't win them all.
But anyway, thanks for the well done Harry Potter fanfic. Truly, I am going to reread it several times, I'm sure.
In the gestalt of (ahem) Japanese fiction, one finds this oft-repeated motif: Power comes from having something to protect.
I'm not just talking about superheroes that power up when a friend is threatened, the way it works in Western fiction. In the Japanese version it runs deeper than that.
In the X saga it's explicitly stated that each of the good guys draw their power from having someone—one person—who they want to protect. Who? That question is part of X's plot—the "most precious person" isn't always who we think. But if that person is killed, or hurt in the wrong way, the protector loses their power—not so much from magical backlash, as from simple despair. This isn't something that happens once per week per good guy, the way it would work in a Western comic. It's equivalent to being Killed Off For Real—taken off the game board.
The way it works in Western superhero comics is that the good guy gets bitten by a radioactive spider; and then he needs something to do with his powers, to keep him busy, so he decides to fight crime. And then Western superheroes are always whining about how much time their superhero duties take up, and how they'd rather be ordinary mortals so they could go fishing or something.
Similarly, in Western real life, unhappy people are told that they need a "purpose in life", so they should pick out an altruistic cause that goes well with their personality, like picking out nice living-room drapes, and this will brighten up their days by adding some color, like nice living-room drapes. You should be careful not to pick something too expensive, though.
In Western comics, the magic comes first, then the purpose: Acquire amazing powers, decide to protect the innocent. In Japanese fiction, often, it works the other way around.
Of course I'm not saying all this to generalize from fictional evidence. But I want to convey a concept whose deceptively close Western analogue is not what I mean.
I have touched before on the idea that a rationalist must have something they value more than "rationality": The Art must have a purpose other than itself, or it collapses into infinite recursion. But do not mistake me, and think I am advocating that rationalists should pick out a nice altruistic cause, by way of having something to do, because rationality isn't all that important by itself. No. I am asking: Where do rationalists come from? How do we acquire our powers?
It is written in the Twelve Virtues of Rationality:
Historically speaking, the way humanity finally left the trap of authority and began paying attention to, y'know, the actual sky, was that beliefs based on experiment turned out to be much more useful than beliefs based on authority. Curiosity has been around since the dawn of humanity, but the problem is that spinning campfire tales works just as well for satisfying curiosity.
Historically speaking, science won because it displayed greater raw strength in the form of technology, not because science sounded more reasonable. To this very day, magic and scripture still sound more reasonable to untrained ears than science. That is why there is continuous social tension between the belief systems. If science not only worked better than magic, but also sounded more intuitively reasonable, it would have won entirely by now.
Now there are those who say: "How dare you suggest that anything should be valued more than Truth? Must not a rationalist love Truth more than mere usefulness?"
Forget for a moment what would have happened historically to someone like that—that people in pretty much that frame of mind defended the Bible because they loved Truth more than mere accuracy. Propositional morality is a glorious thing, but it has too many degrees of freedom.
No, the real point is that a rationalist's love affair with the Truth is, well, just more complicated as an emotional relationship.
One doesn't become an adept rationalist without caring about the truth, both as a purely moral desideratum and as something that's fun to have. I doubt there are many master composers who hate music.
But part of what I like about rationality is the discipline imposed by requiring beliefs to yield predictions, which ends up taking us much closer to the truth than if we sat in the living room obsessing about Truth all day. I like the complexity of simultaneously having to love True-seeming ideas, and also being ready to drop them out the window at a moment's notice. I even like the glorious aesthetic purity of declaring that I value mere usefulness above aesthetics. That is almost a contradiction, but not quite; and that has an aesthetic quality as well, a delicious humor.
And of course, no matter how much you profess your love of mere usefulness, you should never actually end up deliberately believing a useful false statement.
So don't oversimplify the relationship between loving truth and loving usefulness. It's not one or the other. It's complicated, which is not necessarily a defect in the moral aesthetics of single events.
But morality and aesthetics alone, believing that one ought to be "rational" or that certain ways of thinking are "beautiful", will not lead you to the center of the Way. It wouldn't have gotten humanity out of the authority-hole.
In Circular Altruism, I discussed this dilemma: Which of these options would you prefer:
You may be tempted to grandstand, saying, "How dare you gamble with people's lives?" Even if you, yourself, are one of the 500—but you don't know which one—you may still be tempted to rely on the comforting feeling of certainty, because our own lives are often worth less to us than a good intuition.
But if your precious daughter is one of the 500, and you don't know which one, then, perhaps, you may feel more impelled to shut up and multiply—to notice that you have an 80% chance of saving her in the first case, and a 90% chance of saving her in the second.
And yes, everyone in that crowd is someone's son or daughter. Which, in turn, suggests that we should pick the second option as altruists, as well as concerned parents.
My point is not to suggest that one person's life is more valuable than 499 people. What I am trying to say is that more than your own life has to be at stake, before a person becomes desperate enough to resort to math.
What if you believe that it is "rational" to choose the certainty of option 1? Lots of people think that "rationality" is about choosing only methods that are certain to work, and rejecting all uncertainty. But, hopefully, you care more about your daughter's life than about "rationality".
Will pride in your own virtue as a rationalist save you? Not if you believe that it is virtuous to choose certainty. You will only be able to learn something about rationality if your daughter's life matters more to you than your pride as a rationalist.
You may even learn something about rationality from the experience, if you are already far enough grown in your Art to say, "I must have had the wrong conception of rationality," and not, "Look at how rationality gave me the wrong answer!"
(The essential difficulty in becoming a master rationalist is that you need quite a bit of rationality to bootstrap the learning process.)
Is your belief that you ought to be rational, more important than your life? Because, as I've previously observed, risking your life isn't comparatively all that scary. Being the lone voice of dissent in the crowd and having everyone look at you funny is much scarier than a mere threat to your life, according to the revealed preferences of teenagers who drink at parties and then drive home. It will take something terribly important to make you willing to leave the pack. A threat to your life won't be enough.
Is your will to rationality stronger than your pride? Can it be, if your will to rationality stems from your pride in your self-image as a rationalist? It's helpful—very helpful—to have a self-image which says that you are the sort of person who confronts harsh truth. It's helpful to have too much self-respect to knowingly lie to yourself or refuse to face evidence. But there may come a time when you have to admit that you've been doing rationality all wrong. Then your pride, your self-image as a rationalist, may make that too hard to face.
If you've prided yourself on believing what the Great Teacher says—even when it seems harsh, even when you'd rather not—that may make it all the more bitter a pill to swallow, to admit that the Great Teacher is a fraud, and all your noble self-sacrifice was for naught.
Where do you get the will to keep moving forward?
When I look back at my own personal journey toward rationality—not just humanity's historical journey—well, I grew up believing very strongly that I ought to be rational. This made me an above-average Traditional Rationalist a la Feynman and Heinlein, and nothing more. It did not drive me to go beyond the teachings I had received. I only began to grow further as a rationalist once I had something terribly important that I needed to do. Something more important than my pride as a rationalist, never mind my life.
Only when you become more wedded to success than to any of your beloved techniques of rationality, do you begin to appreciate these words of Miyamoto Musashi:
Don't mistake this for a specific teaching of rationality. It describes how you learn the Way, beginning with a desperate need to succeed. No one masters the Way until more than their life is at stake. More than their comfort, more even than their pride.
You can't just pick out a Cause like that because you feel you need a hobby. Go looking for a "good cause", and your mind will just fill in a standard cliche. Learn how to multiply, and perhaps you will recognize a drastically important cause when you see one.
But if you have a cause like that, it is right and proper to wield your rationality in its service.
To strictly subordinate the aesthetics of rationality to a higher cause, is part of the aesthetic of rationality. You should pay attention to that aesthetic: You will never master rationality well enough to win with any weapon, if you do not appreciate the beauty for its own sake.