What was it? AI?

I get an uncomfortable feeling, Eliezer, that this work is to ultimately lead to a mechanism to attract:

• people of libertarian bent

• people interested in practically unbounded longevity of consistent, continual consciousness

and also lead to a mechanism to tar people disinclined to those two goals; tar them with the label "sentimentally irrational".

Rationality to me is simply a tool. I would have absolutely no confidence in it without the ongoing experiences of applying it iteratively, successfully to specific goals.

And of course, no matter

The success of science was and is because it is useful, and similarly for rationalism. But one of the critiques of rationalism and of the overcoming-bias program is that it is sometimes counterproductive. The unbiased tend to be unhappy and/or insane. If someone's goals are to be happy and successful in life, he does best not to be fully rational. Irrationality is the most useful policy if these are your goals.

Your argument suggests that this is true only because this is setting the goalposts too low. For someone who merely seeks happiness, yes, irrational... (read more)

I rarely post, only read in hopes of learning. Today, I comment: I appreciate the beauty of this post.

Thank you, Eliezer.

I am often confused by your writing, because I don't see where you have "skin in the game". Where are you exercising your tools of rationality?

If I'd went ahead and said that within the post, it would've transformed a piece on rationality into overt propaganda, destroying its internal aesthetics. Read my website.

I have touched before on the idea that a rationalist must have something they value more than "rationality"

What a terrible idea... then whenever rationality comes in conflict with that thing, rationality will be discarded.

We already see lots and lots of this behavior. It's the human norm, in fact: use rationality as a tool as long as it doesn't threaten X, then discard it when it becomes incompatible with X.

Perhaps I am one of the "sentimentally irrational," but I would pick the 400 certain lives saved if it were a one-time choice, and the 500 @ 90% if it were an iterated choice I had to make over, and over again. In the long run, probabilities would take hold, and many more people would be saved. But for a single instance of an event never to be repeated? I'd save the 400 for certain.

Your 80% and 90% figures don't really add up either. You don't describe how many people in total will die, regardless of you decision. If the max death number poss... (read more)

Caledonian, I think you're misreading him. He's not saying: the cause is the one thing you never think rationally about. He's saying: the cause is good (rationally good) and to protect/preserve it you have to pull yourself into conformance with the real world, because that's where the action is. To achieve that you have to hold up what you (perhaps mistakenly) think of as "reason" against the real world, and be prepared to re-evaluate if it doesn't work. What your re-evaluation seeks is better techniques of reason - not to throw reason away.

"Rationality by its nature cannot be only a means towards an end."

Rationality is conformance to reality. You can conform to reality for a cause. (You're saying, you can't mold reality to your cause - I agree, but that's not what he was meaning.) He was meaning that people have thought themselves rational when applying formal, skillful, pedigreed academic techniques that DON'T WORK, such as Jesuit style casuistry. So you have to hold the technique up against reality. You won't do that if you put the technique first by saying "I serve reason&q... (read more)

Julian, Caledonian is a well-known troll on OB. We've decided against censorship for now, but you might not want to waste too much time. I generally don't respond to Caledonian unless I see someone else agreeing with him.

I totally agree with "Anon", and others who made similar points in the Circular Altruism post. Context matters! Is it a one-time choice, or an iterated choice? Is there an upper limit to the number of deaths, or no limit? Are the 500 the number of people on the sinking ship/last people on planet earth, or possible victims from a much larger pool? You can only do the math and make a rational decision when you have ALL the numbers from the relevant context.

The first steps of rationality lie not in separating problems from their context, but in determining what context is relevant.

Good question, Nominull. Unfortunately I lack the ability to answer your question from personal experience. Mine just fell into my lap.

But is believing true things what you most desperately want, in all the world?

Caledonian, I gather Eliezer put "rationality" in quotes because people may believe they are committed to rationality when in fact they are not. If they have a goal which is contingent on rationality that will help them from straying from the path.

Anon: do you suggest that others follow your policy as well? Then when many people have individual made isolated choices like that, far fewer lives will have been saved. And in the whole history of the world, choices like that must have been made many times. Why does it matter whether it is you who are repeating the choice or other people?

The question about whether the 500 are that last people in the world is adding other utilities into the issue, such as preserving the human race, and so on. In that case you have a different comparison; naturally, you may have to consider other factors besides the utility of the lives. But as long as you consider only the lives, Eliezer is right.

Caledonian: "I think rationality has to be the starting point."

Can you expand on this? A rationalistic moral relativist might say that actions require goals, ultimate goals are arbitrary, and so rationality cannot be the starting point there. In the real world, by the time one is able to entertain ideas like 'choosing to be more rational', you're already going to have goals, preferences, ideas about how you should live your life. So it could be countered that 'rationality' never has to supply everything; its purpose will largely be to critique ex... (read more)

Anon, Wendy:

Certainly finding out all of the facts that you can is good. But rationality has to work no matter how many facts you have. If the only thing you know is that you have two options:

1. Save 400 lives, with certainty
2. Save 500 lives, 90% probability; save no lives, 10% probability. then you should take option 2. Yes, more information might change your choice. Obviously. And not interesting. The point is that given this information, rationality picks choice 2.

"It takes visceral panic, channeled through cold calculation, to cut away all the distractions." - this just made it to my quotes file.

If i understand Eliezer's point correctly in terms of the map/territory analogy, what he says is that having somewhere to go and actually needing to put your map to use will motivate you to make that map as accurate as possible, if you care about your destination more than you 'believe in' the current iteration of your map and/or the techniques used to derrive it.

A rationalistic moral relativist might say that actions require goals, ultimate goals are arbitrary, and so rationality cannot be the starting point there.

Lots of things act without having any sort of goals. Does fire have a goal of reducing high-energy compounds into oxidized components and free energy? No, but it does it anyway.

You can limit 'action' to intentional events only, I suppose.

However, how does declaring that goals are arbitrary rule out assertions about necessary starting points?

So it could be countered that 'rationality' never has to su

Formerly, I think I would have chosen #2 in the first case and #1 in the second. But Eliezer has converted me. Now I choose #2 in both cases. But would he do that himself?

Isn't that implicitly what he does for a living? Eliezer could become a firefighter or emergency medical technician, or work for clean drinking water in rural Africa, with a near-certainty of preventing several deaths in the next year. Meanwhile, there is a very small chance of someone creating an non-Friendly AI in the next year. We can argue about the probabilities (of the problem arising, of successfully presenting it), but Eliezer has already chosen the existential threat.

"So I hereby retract my argument against voting, Pascal's Mugging, and Pascal's Wager. In the particular Mugging we discussed, there may have been anthropic reasons to make it proportionally improbable. But without such reasons, it should be accepted."

I'm certainly glad you think so, Unknown, because I was just contacted by the Dark Lords of the Matrix. It turns out that we are living in a simulation. I have no idea what the physics of the world outside are like, but they're claiming that unless you personally send $100 to SIAI right now, they're... (read more)

(same anon from above who asked about the context of the 400/500 problem being an issue)

In response to GreedyAlgorithm who said:

Certainly finding out all of the facts that you can is good. But rationality has to work no matter how many facts you have. If the only thing you know is that you have two options:

1. Save 400 lives, with certainty
2. Save 500 lives, 90% probability; save no lives, 10% probability. then you should take option 2. Yes, more information might change your choice. Obviously. And not interesting. The point is that given this information, rat

Z. M. Davis, given the existence of that many people, and given that threat, the probability that I personally would be the one threatened in that must be multiplied by one over the number of people, since it could have been anyone else. So the expected disutility from your mugging is one dust speck multiplied by the probability that the Matrix scenario is actually true. This probability is very low, and even if it were unity, the disutility of one dust speck isn't going to get me to pay $100.

So again, I said "without such reasons, it should be accept... (read more)

It's probably just that I'm stupid, but I don't understand the anthropic solution to Pascal's Mugging. Why does it matter that other people could have been asked? What if it were stipulated that the mugger threatens everyone?

Maybe I should actually study Kolmogorov complexity before trying to grapple with such matters.

Viz. the dilemma posed in Circular Altruism, what should we do? When forced to "Shup up and multiply", we have forgone our intuitions and picked the choice based upon our mathematics. However, we are not just overcoming our own intuitions, but also the intuitions of everyone who does not simply "Shut up and multiply". We are held accountable not by those who knows the math, but by those who have intuitions like ourselves.

If we save everyone, we are heroes. If we do not, we are held accountable not for the math, but for the very intuitio... (read more)

RS, if that really bothers you, you haven't found your something to protect yet.

So, is your point that we need a cause against which to evaluate the success of our mathematics? That perhaps this sort of feedback that, persumably, you encounter on a daily basis, is something that does not come through rationality itself, but through the very real feedback of what you have chosen to protect?

I guess my previous post was a reflection that I am just a budding rationalist, and also that my skills have not been sharpened against the proper stone.

"The point is that given this information, rationality picks choice 2." - Posted by: GreedyAlgorithm

Sorry, no. Given this information, rationality says that there is not enough information to make an appropriate decision, and demands to know the context. If contextual information isn't available, rationality will say that either option 1 or 2 may be right, depending on circumstances.

Rationality never dismisses context as irrelevant just because it isn't known. If unknown factors make the right answer uncertain, then you must accept that it is unc... (read more)

For some reason this post reminds me of the Buddhist parable asceticsim now, nymphs later.

I don't think it's all that uncommon to begin cultivating an art for some specific purpose, proceed to cultivate it largely for its own sake, and eventually to abandon the original purpose.

Under Multiple Worlds, aren't you condemned, whatever you do or don't do, to there being a number tending to infinity of worlds where what you want to protect is protected, and a number tending to infinity where it is not ?

Caledonian: Let's distinguish between the aesthetics of rationality and the pragmatics of rationality. Is my model of the world consistent, do my goals make sense - that's pragmatics. Aesthetics is by comparison nebulous and subtle, but perhaps it encompasses both admiration for the lawlike nature of reality and self-admiration for one's own relationship to it. :-)

It seems to me that you are taking issue with the idea that the pragmatics of rationality should be trumped by a higher cause. This essay says nothing about that. It says, first, that it's a psyc... (read more)

You are not alone, Z. M. Davis: I disagree with Eliezer over whether Robin's anthropic solution is a satisfactory solution to Pascal's Mugging. (Eliezer repeated his endorsement of Robin's anthropic solution here a few weeks ago.) Since I started reading Eliezer 6 years ago, this is the first time I can recall disagreeing with him on a question of fact. (As I have pointed out many times in the comments here, I disagree with him significantly on terminal values.) If anyone wants to reply to this, I humbly suggest doing so by clicking on my name below.

For those saying they have nothing to protect or still need to find something to protect, remember that you are human and, unless you have no natural family or reproductive ties, you always have the people you love to protect. It may seem counterintuitive if you've bought into Hollywood rationality, but love is a powerful motivational force. If you think that, in theory, being more rational is good, but don't see how you can effect greater rationality in your mind, consider the many benefits of your increased rationality (again, not Hollywood rationality... (read more)

Excellent point by Worley. Since I have assumed the role on this blog of pointing out that happiness is not the meaning of life, let me hasten to add that happiness is a very useful barometer. Whether you are happier on average now than you were 10 years ago is for example probably a more reliable barometer of whether your life is on a better track than it was 10 years ago than change in financial net worth over those 10 years (though net worth is an important barometer too). And the one situation in which happiness is least likely to steer you wrong is... (read more)

Hollerith, if 'most psychologists are idiots', I wonder how they discovered all the cognitive biases ?

if 'most psychologists are idiots', I wonder how they discovered all the cognitive biases

He said 'most', not 'all'. And just because someone is an idiot doesn't mean everything they do is wrong. Even Freud managed to do some good descriptive work before descending into madness and delusion.

I mentioned psychologists in a particular context, namely, how to apply the skills of rationality to the project of nuturing and supporting your friends, lovers and family. Worley and I think rationality can be applied to that project. But I thought just leaving it at that would mislead some of the readers who have not had a lot of practical experience in life: unlike many of the other projects rationality is typically applied to, this project is different in that you cannot just travel to your nearest bookstore and by browsing the shelves expect to fin... (read more)

Caledonian: "I don't believe I've ever heard anyone speak of the aesthetic aspects of rational thought before."

It's funny - the phrase "aesthetics of rationality" appears in the final paragraph of Eliezer's post; apparently it's what the whole thing was about. But I didn't notice it either, until I was seriously casting about for some way to show that Caledonian person why their criticism was off the mark. I think Eliezer's point may be something like this: the aesthetics of rationality are all that could truly make it an end in itself;... (read more)

Aesthetics are rarely a topic when rationality is discussed. Mostly because they're only relevant to ancient-Greek-style thought.

On the list of things likely to cause unreasonable attachment, it's pretty far down. Love of familiarity, wanting to appear intelligent to others, wanting to appear intelligent to oneself, unwillingness to face conclusions that one finds unpalatable, general inflexibility... these are all plausible causes of failure. But aesthetics?

I think you're pretty close to the core of this one. You identified that having something to protect gives you strength. And having a worthy cause to work for, for the same reason.

But what is that reason? What is it that gives you strength? What is the underlying cause of us gaining strength from certain causes?

I'm not certain I understand the topic well enough myself, but I think I have something that you might find insightful here.

Moral Idealism. That's where your power comes from. Whether you're fighting to protect a loved one, or you're fighting to pro... (read more)

Personally, I find aesthetic purity to be a very strong source of attachment for me. It's certainly caused 'unreasonable attachments', like being stuck on being "right" and ascribing a purity to it (eg. I am right about this and you are wrong, therefore I will absolutely refuse to do this small nitpicky thing and I don't care if I jam up the whole process because it's MORALLY WRONG not to do so. I am the lone voice of dissent!). Oh, school..

I came across the same hack, or coping trick. Just remap the definition of what you're being pure about to "winning" or "rationality".

Pretty sure I'm displaying that I missed the point somehow.

The proper choice between (1) certainly save 400 lives and (2) 90% probability of saving 500 lives with 10% probability of saving no lives, depends on your utility function, which depends on the circumstances. If your utility is proportional to the number of lives saved, then sure, go with (2).

On the other hand, suppose that some cataclysm has occurred, those 500 lives are all that remains of the human race, and extinction of the human race has such an extremely negative utility for you that all other considerations amount to rounding error in the utility... (read more)

Holden made a similar point.

All right, I'd like to attempt a summary to make sure that I am understanding this post, if anyone see's some mistake in my interpretation, I'd appreciate it if they let me know.

Virtually everyone wants their beliefs to be true, this amounts to practically everyone wants to be epistimically rational. Rationality is a rare trait, so obviously that desire is not enough to make you epistimically rational. But that desire mixed with the rare desire to have all of your beliefs make useful predictions about whatever they talk about, is enough, provided that you... (read more)

You will never master rationality well enough to win with any weapon, if you do not appreciate the beauty for its own sake.

I have a low prior for this statement, but I don't have any data. I wonder why Eliezer thinks this is the case.

Here I have a question that is slightly unrelated, but I'm looking for a good cognitive science science fair project and I'm having trouble thinking of one that would be not completely impractical for a high-schooler to do, won't take more than a few months, and would be interesting enough to hold people's attention for at least a few minutes before they head off to the physics and medical research projects. No one ever does decent cognitive science projects and I really want to show them that this branch of science can be just as rigorous and awesome as the other ones. Does anyone have any ideas?

I've been coming back to this post for 7 years or so, and the whole time it's obvious that I don't have something to protect, and haven't found one, and haven't yet found a way to find something to protect. It seems pretty cool though - and accurate that people who really care about things are able to go to great lengths to improve the way they think about the thing and their ability to to solve it.

I can say that once I realized I cared about wanting to care about something, that helped me quite a bit and I started improving my life.

Very interesting. I can't help feeling that "trying to be a better rationalist" is somehow a paradoxical aim.

Roughly speaking I would say that we have preferences, and their is no rational way of picking preferences. If you prefer pizza to icecream, or pleasure to pain, or living to dying, then that is that. Rationality is a mechanism for effectively seeking your preferences, ordering pizza, not putting your had in a fire etc. You can't pick rational preferences (goals), you can pick a rational route towards those goals.

If you adopt "I want to be more rati... (read more)

I savor the succulent choleric chaos of declaring that I value mere phlegm above yellow bile. That is almost a contradiction, but not quite; and the resulting blend has a choleric quality as well: a delicious humor.