Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

If reason told you to jump off a cliff, would you do it?

-12 Post author: Shalmanese 21 December 2009 03:54AM

In reply to Eliezer's Contrarian Status Catch 22 & Sufficiently Advanced Sanity. I accuse Eliezer of encountering a piece of Advanced Wisdom.

Unreason is something that we should fight against. Witch burnings, creationism & homeopathy are all things which should rightly be defended against for society to advance. But, more subtly, I think reason is in some ways, is also a dangerous phenomena that should be guarded against. I am arguing not against the specific process of reasoning itself, it is the attitude which instinctually reaches for reason as the first tool of choice when confronting a problem. Scott Aaronson called this approach bullet swallowing when he tried to explain why he was so uncomfortable with it. Jane Galt also rails against reason when explaining why she does not support gay marriage.

The most recent financial crisis is a another example of what happens when reason is allowed to dominate. Cutting away all the foggy noise & conspiracy theories, the root cause of the financial crisis was that men who believed a little too much in reason became divergent from reality and reality quickly reminded them of that. The problem was not that the models were wrong, it was that what they were trying to accomplish was unmodelable. Nassim Nicholas-Taleb's The Black Swan explains this far better than I can.

A clear-eyed look back on history reveals many other similar events in which the helm of reason has been used to champion disastrous causes: The French Revolution, Communism, Free Love Communes, Social Darwinism & Libertarianism (I am not talking about affective death spirals here). Now, one might argue at this point that those were all specific examples of bad reasoning and that it's possible to carefully strip away the downsides of reason with diligent practice. But I don't believe it. Fundamentally, I believe that the world is unreasonable. It is, at it's core, not amenable to reason. Better technique may push the failure point back just a bit further but it will never get rid of it.

So what should replace reason? I nominate accepting the primacy of evidence over reason. Reason is still used, but only reluctantly, to form the minimum span possible between evidence & beliefs. From my CS perspective, I make the analogy between shifting from algorithm centered computation to data centric. Rather than try to create elaborate models that purport to reflect reality, strive to be as model-free as possible and shut up and gather.

If the reason based approach is a towering skyscraper, an evidence based approach would be an adobo hut. Reasoning is sexy and new but also powerful, it can do a lot of things and do them a lot better. The evidence based approach, on the other hand, does just enough to matter and very little more. The evidence based approach is not truth seeking in the way the reason based approach is. A declaration of truth, built on a large pile of premises is worse than a statement of ignorance. This, I think is what Scott Aaaronson was referring to when he said "What you've forced me to realize, Eliezer, and I thank you for this:  What I'm uncomfortable with is not the many-worlds interpretation itself, it's the air of satisfaction that often comes with it."

Comments (41)

Comment author: anonym 21 December 2009 09:12:19AM *  17 points [-]

If reason told you to jump off a cliff, would you do it?

Yup, because there are probably some pretty damned good reasons -- no pun intended -- for jumping off the cliff, like that I have a parachute strapped to my back and an impending explosion will kill all people still on the cliff.

If there weren't good reasons that were similarly persuasive, then it wouldn't be reasonable.

Comment author: wedrifid 21 December 2009 01:46:16PM 3 points [-]

Yup, because there are probably some pretty damned good reasons -- no pun intended -- for jumping off the cliff, like that I have a parachute strapped to my back and an impending explosion will kill all people still on the cliff.

That's a good example. I was going for "hanglider, chased by a guy wielding a claymore".

Comment author: Nic_Smith 21 December 2009 05:47:50PM 2 points [-]

Would "bungee cords firmly attached to body, cutie of the interesting sex looking on anxiously" count as reason or as misguided social signaling?

Comment author: Cyan 21 December 2009 06:14:28PM 2 points [-]

You missed the rather important "cliff has substantial overhang".

Comment author: Blueberry 21 December 2009 09:28:34PM 1 point [-]

Depends on the utility you get from bungee jumping and the probability of the cords breaking.

Comment author: aausch 26 December 2009 04:17:11AM 1 point [-]

Also, on your relative social position to the cutie, your general charm and good looks, etc .. - in other words, your perceived odds of making it with her if you don't jump off the cliff.

Comment author: Kaj_Sotala 21 December 2009 09:17:03AM 14 points [-]

You're employing the classical division of reason versus empiricism. While I do have criticized this forum's emphasis on reason before, your critique seems to be off the mark. This site is devoted to rationality, not pure reason as such, and rationality uses whatever mix of reason and empiricism that achieves the best results. You're making the assumption that Eliezer relies only on reason and not enough on evidence, but you (ironically given your position) aren't providing any evidence to back up the claim.

Suppose there is an optimal mix of reason and empiricism for a specific domain. If you suspect somebody's mix of the two seems to be wrong, you need to first show that the optimal mix for that domain is what you think it to be, and then that your opponent is leaning too much in one direction. You can't just say "my opponent is wrong" and then parade a long list of failures which you believe were caused by too much reason. That's a fully general counterargument.

Comment author: grouchymusicologist 21 December 2009 04:29:33AM 5 points [-]

Jane Galt also rails against reason when explaining why she does not support gay marriage.

Seems like a fairly blatant category error to even mention that in this context.

Comment author: Bo102010 21 December 2009 01:30:20PM *  1 point [-]

I don't know about that, but while "[McArdle] does not support gay marriage" is technically true, it's a bit misleading. McArdle "does not support one side or the other," but only mentioning that she doesn't support one side sort of implies that she supports the other.

Comment author: Blueberry 21 December 2009 10:38:48AM 1 point [-]

Why? I think that the OP meant that people blindly support gay marriage when thinking purely in terms of abstract logic, instead of looking at the empirical world, where there are marginal cases where gay marriage may make things worse, as Jane Galt discusses.

Comment author: Psychohistorian 21 December 2009 05:31:38AM *  12 points [-]

"Reason" does not appear to be the right term here. "Cleverness" comes to mind as a better substitute, though I suspect there are better terms. The banking crisis occurred because people thought they were too clever. The various problematic causes you mention all appear to overestimate their own cleverness. It's also unclear to me what it would mean for them to rely on reason less, and how this would cause their worldview to better match reality.

I think this might be best phrased as an objection to an overreliance on clever theories and a tendency to eschew evidence in favor of cleverness. Insofar as that is your point, it is an excellent if not novel one. But the way this is phrased is a bit more antagonistic than I think is merited, and seems to attack a type of thought rather than a specific error that a type of thought is prone to.

If my semantic distinction does not make sense, let me just explain my connotations. When I hear "reason," I tend to think of it much like "rational;" one definitionally cannot make a mistake through being too rational, in that rationality is the thing that having more of it causes you not to make mistakes. "Cleverness," on the otherhand, brings the same intellectual sleight-of-hand without any connotation of accuracy. The sentence, "Bob lost all his money because he was too reasonable," does not really make sense, whereas, "Bob lost all of his money because he was too clever," does. A good example of being too clever would be the demise of Vizzini from the Princess Bride.

Comment author: simpleton 21 December 2009 07:04:32AM 4 points [-]

Yes.

The things Shalmanese is labeling "reason" and "evidence" seem to closely correspond to what have been previously been called the inside view and outside view, respectively (both of which are modes of reasoning, under the more common definition).

Comment author: RobinHanson 21 December 2009 05:30:59PM 4 points [-]

Yes, that was going to be my comment. The outside view also uses "reason" but with wider and shallower chains of reasoning. The inside view is more fragile, requiring more assumptions and longer chains of reasoning.

Comment author: cousin_it 22 December 2009 02:59:18PM 0 points [-]

Thanks for that comment, it made something click for me.

Comment author: abigailgem 22 December 2009 02:26:49PM 3 points [-]

"cleverness" comes to mind as a better substitute

Or "Hubris". In the examples, the people go wrong not because they are using reason and they should not use reason, but because they falsely imagine they are capable of using reason sufficiently to deal with the particular issue.

Comment author: Technologos 21 December 2009 11:04:37AM 2 points [-]

I'd add that "reason" is here taken as something like "abstract deduction."

Far from reason and evidence being in conflict on occasion, the former strictly requires the latter.

Indeed, it is precisely when we start thinking of reason as disconnected from evidence that it turns into cleverness. Reason is attached to no particular ritual of cognition but rather to Winning.

Comment author: Dustin 21 December 2009 07:36:27PM 1 point [-]

Voted up, because it so well voices the thoughts I had when I read the OP in my RSS reader.

Comment author: smoofra 22 December 2009 07:03:10PM 4 points [-]

You're confusing "reason" with inappropriate confidence in models and formalism.

Comment author: PhilGoetz 21 December 2009 06:12:36AM *  4 points [-]

The most recent financial crisis is a another example of what happens when reason is allowed to dominate. Cutting away all the foggy noise & conspiracy theories, the root cause of the financial crisis was that men who believed a little too much in reason became divergent from reality and reality quickly reminded them of that.

How can you say reason failed them, when they made so much money?

Comment author: Technologos 21 December 2009 07:35:45PM 2 points [-]

Made and continue to make. One wonders if they might not actually be rather prime examples of Winning in a practical though perhaps corrupted sense.

Comment author: wedrifid 21 December 2009 05:53:25AM 4 points [-]

In reply to Eliezer's Contrarian Status Catch 22 & Sufficiently Advanced Sanity. I accuse Eliezer of encountering a piece of Advanced Wisdom.

This post is indistinguishable from 'Advanced Wisdom'. I accuse you of not really understanding that of which you speak. Even if you are, in fact, advanced and wise you will need to do some background reading here in order to understand the posts here that you may enlighten the authors.

By way of answer to your rhetorical question: Yes, I would jump of a cliff if reason told me to.

Comment author: RichardKennaway 21 December 2009 04:15:31PM 3 points [-]

The problem was not that the models were wrong, it was that what they were trying to accomplish was unmodelable. Nassim Nicholas-Taleb's The Black Swan explains this far better than I can.

My reading of Taleb's book was the opposite: he says quite explicitly that the finance models behind the crash were wrong, being based on Gaussian distributions, which did not fit the phenomena being modelled. He contrasts the Gaussian with power-law distributions, although I admit I don't know if he has done any mathematical modelling of the finance markets using those.

Comment author: Technologos 21 December 2009 07:33:18PM 0 points [-]

I don't know what exactly he has done, modeling-wise, but he seems to have made a hell of a lot of money during the financial collapse. Whether this is long-term better modeling than the kind he criticizes or largely due to sampling coincidences, remains to be seen.

Comment author: Vladimir_Nesov 21 December 2009 04:37:57AM 3 points [-]

In many situations, there is a "no action" safe default action, which you can take as a matter of epistemic hygiene. Then, overconfidence is dangerous and doubt is a virtue. In other cases, this heuristic is not as commendable.

Comment author: cousin_it 21 December 2009 08:21:28AM *  2 points [-]

I think you really meant to argue that (outside of mathematics) each link in a chain of inference has a small chance of going wrong, so longer chains are exponentially less likely to be correct than shorter ones. This is well known around here, though hubris does take the better of us sometimes!

Comment author: komponisto 21 December 2009 04:20:38AM *  2 points [-]

The problem was not that the models were wrong, it was that what they were trying to accomplish was unmodelable.

[...]

Fundamentally, I believe that the world is unreasonable. It is, at it's core, not amenable to reason. Better technique may push the failure point back just a bit further but it will never get rid of it.

Since you mentioned Scott Aaronson, you might want to have a look at this post.

Comment author: Eliezer_Yudkowsky 21 December 2009 05:27:55AM 3 points [-]

Read through the sequences, dammit.

Comment author: mattnewport 21 December 2009 05:31:14AM 15 points [-]

Indeed. Maybe start with Rationality is Systematized Winning. Giving a few examples of people failing at rationality is not an effective criticism of rationality.

Taleb's books are interesting and he makes quite a few good points but the man has a lot of flaws. Eric Falkenstein sums up the case against him quite well.

Comment author: Technologos 21 December 2009 10:57:36AM 1 point [-]

I maintain that, of all the paradigm-altering and life-improving concepts I've gained from the sequences, the one derived from that post was the most important.

Literally nothing else we do here makes sense without it.

Comment author: PhilGoetz 21 December 2009 06:16:10AM *  1 point [-]

And I'm voting this up, for the same reasons.

Comment author: PhilGoetz 21 December 2009 06:11:12AM *  16 points [-]

I'm voting this down because you so often respond to people with something like "Read the sequences" without deigning to mention any specific problems. That's high-handed and cult-leader-like. It's also unhelpful to someone even if they take your advice and read the sequences, because they're not likely to figure out the connections between them and what they said in their post.

At the very least, recommending one specific post to read would a great improvement.

Comment author: MrHen 22 December 2009 06:56:46PM 3 points [-]

I upvoted EY's comment, but very much agree with your comment. I am still plugging through the sequences and appreciate the, "This was answered, dummy." response, but I prefer, "This was answered over here, dummy."

In terms of actually increasing rationality in other people, getting short with them is probably counter-productive. But I get the frustration of dealing with noobs like me. I prefer the harshness because the feedback is clearer but I imagine that not everyone is like me.

Comment author: wedrifid 21 December 2009 06:20:44AM *  3 points [-]

Downvoted the parent (I have no doubt someone will rectify the temporary negative total). I hate high handed behaviour and so crying wolf over cult like high handedness is an instrumental detriment to me.

It's also unhelpful to someone even if they take your advice and read the sequences, because they're not terribly likely to figure out the connections between them and what they said in their post.

I would have upvoted if the expressed reasoning was "Downvoted for not providing the necessary reference to the sequence that would make the confusion obvious. It would not be hard for you to find the link matthew found and leaving it off is presumptive and makes you look kinda like a tool."

(ETA: Good change. Vote now neutral.)

Comment author: bogus 21 December 2009 12:28:08PM *  0 points [-]

It's quite hard to evaluate this post with any precision, but I find it reasonable to assign 23.432774% probability that it is a genuine piece of Advanced Wisdom. Accordingly I choose +1, with an estimated expected benefit of 235.74637 utilons plus 54.747746 warm fuzzies.

Comment author: MatthewB 24 December 2009 08:21:28PM *  0 points [-]

Maybe someone could clear this up a bit for me, but...

I had always thought that reason was a method of formalized thinking that weighed evidence and logic in coming to the most likely or most preferable conclusion.

Whereas...

Rationality is reasoning within a formal system, and following the consequences of the beliefs of that formal system.

I realize that my attempt to define these two terms is probably flawed to a degree, but I am still new at much of the terminology.

Edit: is there some form of coding by which I might use italics, bold, underlining, quoting, links, etc.? I see that others have used these, yet the norms for these styles don't seem to work here.

Comment author: Alicorn 24 December 2009 08:35:17PM *  1 point [-]

Click the "help" link that appears when you compose a comment to get a style guide. It'll tell you how to do bold, italics, links, etc.

Comment author: MatthewB 24 December 2009 09:29:08PM 0 points [-]

Click the "help" link that appears when you compose a comment to get a style guide. It'll tell you how to do bold, italics, links, etc.

Like this?

Thanks much. That will help a lot, as I tend to be a stickler for style (I also appreciate the heads-up on my rather embarrassing reliance upon the word selection software that caused a few gaffs earlier)

Comment author: Eliezer_Yudkowsky 24 December 2009 08:32:40PM 1 point [-]
Comment author: MatthewB 24 December 2009 09:26:19PM 0 points [-]

Thank you (much).

I do have to say that for one who claims to have a hard time writing, you are as prolific as one who claims to have no problem writing (now, as to what you mean by this, it may be up to a definition of goals of the writing).

The differences between Epistemic and Instrumental Rationality helped a great deal.

In my discussions with Steve Omohundro about Rationality, I get the feeling that he tends to think of rationality in the Instrumental sense. So, I have had a tendency to lean more toward this definition, even though I have often felt that I should be more inclusive of the epistemic type.

This leaves the question of defining Reason. Following what I have read, reason is just a toolbox (with many tools inside) to allow us to make rational decisions.

Is that correct? (note: I have not yet read all of the "Map and the Territory". That's a lotta words in that link, and it will take me some time to get through them all)

Comment author: LucasSloan 21 December 2009 06:43:48AM 0 points [-]

If reason told me to jump off a cliff would I?

Interesting question. I'm tempted to say yes. If by jumping off a cliff, I can save the lives of 20 other people, then yes. I certainly should. I'm not going to say that I would, I'm a much a victim of acrasia as anyone.

Comment author: wedrifid 21 December 2009 01:48:54PM 0 points [-]

If by jumping off a cliff, I can save the lives of 20 other people, then yes. I certainly should.

Really? Well, so long as you don't try to tell me I should jump off a cliff to save 20 other people then by all means go ahead. The moment you tried to extend that norm beyond yourself I would reject your attempt.