= 27d567bc2d48d9f40e9ccc20ea370b15)
The Ten Commandments of Rationality
(Disclaimer/TL;DR: This article, much like Camelot, is a silly place/post. Nonetheless I think it presents a pretty solid list of 10 rationality lessons to take away from Less Wrong which must not be forgotten upon pain of eternal damnation/irrationality.)
In a realm not far from here, somewhere within a bustling metropolis, there lies an old and dusty book. It is placed in a most conspicuous location; in the middle of a busy street where countless citizens walk by it every day. Yet none pick it up, for it is placed on a pedestal just high enough that it cannot be reached or seen easily, and the slight inconvenience of standing on one’s toes to reach for it is sufficient to deter most. Yet if a traveller were sufficiently aware to look up and see the book, and curious enough to reach for it, and willing to suffer the slight discomfort of having to touch its muddy cover to open and read its ancient pages, that one would find within a wealth of wisdom and rationality that would transform the reader’s life forever. For this is the most holy Book of Bayes, and its first and last pages both read thusly:
The Ten Commandments of Rationality
1) Thou shalt never conflate the truth or falsehood of a proposition with any other characteristic, be it the consequences of the proposition if it be true, or the consequences of believing it for thyself personally, or the pleasing or unpleasant aesthetics of the belief itself. Furthermore, thou shalt never let thy feelings regarding the matter overrule what thy critical faculties tells thee, or in any other way act as if reality might adjust itself in accordance with thine own wishes.
2) Thou shalt not accept any imperfect situation if it may be optimized, nor shalt thou abstain from improving upon a situation by imagining ever better options without acting on any of them, nor must thee allow thyself to be paralyzed with fear or apathy or indecision when any action is still superior to doing nothing at all. Thus let it be said: Thou shalt not allow thyself to be beaten by a random number generator.
3) Thou shalt not declare any matter to be unscientific, or inherently irrational, or a false question, or with any other excuse wilfully close thine own eyes and expel all curiosity regarding the matter before thou hast even asked thyself whether the question is worth answering. To transgress thusly is to forfeit any chance to update thy own beliefs on a matter that is truly unusual to thee.
4) Thou shalt not hold goals or beliefs which conflict with each other, in such a manner as to violate most divine transitivity, and thereby set thyself up for most ignominious defeat, and rest easy in knowing this fact. Rather shalt thou engage in mindfulness and self-reflection, and in doing so find thy own true priorities, and solve any inconsistencies in a utility maximising manner so that thou may not fall prey to the wrath of the most holy Dutch Book, which is merciless but just.
5) Thou shalt never engage in defeatism, nor wallow in ennui or existential angst, or in any other way declare that thy efforts are pointless and that exerting thyself is entirely without merit. For just as it is true that matters may never get to the point where they cannot possibly get any worse, so is it true that no situation is impossible to improve upon.
6) Thou shalt never judge a real or proposed action by any metric other than this: The expected consequences of the action, both direct and indirect, be they subtle or blatant, taking into account all relevant information available at the time of deciding and no more or less than this.
7) Thou shalt never sit back on thy lazy laurels and wait for rationality to come to thee, nor shalt thou declare that thy beliefs must be correct as all others have failed to convince thee of the contrary: The cultivation of thy rationality and the falsification of thine beliefs is thine own most sacred task, which is eternal and never finished, and to leave it to others is to invite doom upon the validity of thine own beliefs and actions, for in this case others will never serve thee as well as thou might serve thyself.
8) Thou shalt never let argumentation stand in the way of knowledge, nor let knowledge stand in the way of wisdom, nor let wisdom stand in the way of victory, no matter how wise or clever it makes thee feel. Also shalt thou never conflate exceptions for rules or rules for exceptions when arguing any issue, nor bring up minutiae as if they were crucial issues, nor allow oneself to be swept away in arguing for the sake of argumentation, nor act to score cheap and yea also easy points, nor present thy learnings in a needlessly ambiguous manner such as this if it can be helped, or in any other way allow oneself to lose sight of thine most sacred goal, which is victory.
9) Thou shalt never assign a probability exactly equal to 0 or 1 to any proposition, nor declare to the skies that thy certainty regarding any matter is absolute, nor any derivation of such, for to do so is to declare thyself infallible and is placing thyself above thine most holy lord, Bayes.
10) Thou shalt never curse thy rationality, and wish for ye immediate satisfaction over thy eventual victory, all for the sake of base emotion, which is transient whereas victory is transcendent. Let it be known that it is an unspoken truth amongst rationalists -indeed it is the first and most elementary rule of rationality and yet oft forgotten by those practiced in the art- that base impulse and most holy reason are as a general rule incompatible, as there cannot be two skies.
Such are the Ten Commandments of Rationality. And Lo! If one abides by these rules, then let it be said that they act virtuously, and the heavens shall reward them with the splendour of higher expected utility relative to the counterfactual wherein they did not act virtuously. But to those who do not act virtuously, but rather act with irrationality in their minds and biases in their thinking, and who in doing so break any of the Commandments of Rationality, to them let it be said that they have transgressed against thy lord Bayes, and they shall be smitten by the twin gods of Cause and yea also Effect as surely as if they had smitten themselves. For let it be said: The gods of causality may be blind, but their aim doth be excellent regardless.
(All silliness aside, what do you all think? Is this a good list of 10 things to take away from Less Wrong? Do you have a better list? Are posts like these a waste of time? Or, Bayes forbid, did I get my thees and thous wrong somewhere? Let me know in the comments.)
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Comments (73)
Here, let me take a stab:
1) Don't confuse beliefs and values.
2) Be agenty.
3) Never leave information on the table.
4) Strive for consistent beliefs.
5) There is actually a Christian formulation of this one: "thou shalt not blaspheme against the Holy Spirit" (Acquinas interpretation). Judaism and Catholicism (perhaps Sufism also, but I am not very familiar with the Sufi tradition so will not comment) have many elements of "proto-rationality", for a number of reasons, one of them being that at one point studying religion was "academia" -- where smart people went.
6) Use CDT :).
7) Having accurate beliefs and completed goals takes work. Remember to work for what you want.
8) Argue collaboratively.
9) Never be certain of anything.
10) Remember to integrate your utility with respect to time.
You know, the difference between people like Dennett or Dawkings and the LW crowd, is that while all are atheists, Dennett and Dawkings genuinely do not miss God or religion. I get the feeling you guys do, with your commandments, and virtues, and solstices, and wedding ceremonies.
I disagree with 4). I think our cognitive architecture is not consistent, and I think wishing it were so is not really very productive. "Man, to thyself be true."
Thank you for that brevity. It makes clear, what can then also be seen in the original, a striking omission: any injunction to pursue the truth, to make one's beliefs correspond with reality. Which highlights the problem with (4): updating towards consistency -- also called decompartmentalising -- while neglecting to update towards reality is a short road to crackpottery.
Dawkins who wants that we call ourselves brights and who preaches militant atheism isn't that from from religion either.
The Brights movement explicitly rejected priests, gurus, and ritual. I sometimes wonder why lesswrong cannot seem to let these things go, as well. Some of this attachment is disguised as humor, but us homo sapiens love to use humor as a cover. Personally, I find this attachment one of the creepier features of the lesswrong community.
Aggressively competing in the marketplace of ideas is not the same thing as religion, many clearly unreligious sets of ideas compete quite aggressively.
What do you mean with "priest" and "guru". For what definition of those terms do they fit for someone in the LW community but don't fit for Dawkins or the other horsemen?
As far as rituals go, you might have a point. I think it's comes out of the rationalism is about winning idea of lesswrong. Rituals are simply very useful tools. Not using them means choosing a suboptimal strategy.
Policy Debates Should Not Appear One-Sided. The argument for or against using rituals shouldn't appear one-sided.
Competing in the market place of ideas doesn't mean that you have to self-identify with a label that signals group loyality. There are a bunch of people in the new atheist community who say things like the purpose of life is to spread one's genes.
Sorry for not being clear -- I am not saying that lesswrong should stop using rituals. That is, this is not a policy debate. I consider myself an outsider to this community, and it would be rude for me to impose. I may find it creepy, but who cares what I think?
If lesswrong wants to use rituals, for whatever reason, it should. But I think it is rather curious in the sense of being an irreligious org that uses religious trappings. This is what I mean when I say that lesswrong misses religion.
I mean "guru" in the sense that EY is considered a guru (and, I believe, deliberately cultivates an air of a guru).
Given that you have 2000 karma I don't think you are truely an outsider.
The Jusos are the youth organisation of Germany's SPD, which is the left party that's currently part of the government.
At one Jusos meeting I attended we sang the Internationale. It's a ritual. It's useful for group bonding. That doesn't make it religious.
Could you be more specific? What behaviors are you talking about?
You know, Israel defines a Jew to be someone who considers themselves a Jew.
I ignore karma. I am not convinced by the idea of rationality or consistent beliefs. I am not a Bayesian with a capital B. I don't subscribe to the Everett interpretation of quantum mechanics. I am not an atheist. I don't believe UFAI is an issue worth spending a lot of resources on at the moment. I attended a total of 1 LW meetup (in Boston -- I think Scott Aaronson and Michael Vassar were there).
I do think LW is a good and valuable community, and I think there are many very useful concepts in circulation here (for example tabooing and steelmanning -- these are useful enough to have been reinvented elsewhere), which is why I participate here. Also some folks connected to LW think about and write about interesting things.
Things like point 1 in this post:
http://lesswrong.com/lw/jne/a_fervent_defense_of_frequentist_statistics/ajwa
A guru speaks from a position of authority, a scientist communicates/argues with peers. I think the modern academic approach has a lot less failure modes than the guru approach (which has been tried extensively in the past of our species).
edit: To clarify my thinking a bit. A "guru" is a kind of memetic feudal lord. I am suspicious of attempts to revert to feudalism, our species does it far too easily. I think we can do better than feudal forms.
Being an outsider is more than just not subscribing to the label of belonging to an ingroup.
I don't think disagreeing on things like consistent beliefs makes you an outsider. I think the last longer post on the topic even argued against having consistent beliefs.
I don't think what makes a true citizen of Lesswrong is that a person treats Eliezer as his guru and simply copies his beliefs.
If you decide that this community is good, that you find participation valuable and think for yourself that makes you perfect member.
Then what are you and if you already like religion why do you see a problem with the same pattern appearing in LW?
I don't think it's useful to pretend that everyone understand what you mean with a concept. It can seem authoritative to say that the person you are talking to just doesn't understand what you mean, but it often directly addresses the core issue of a disagreement.
Communication is also not something where you have to pick one style for all your communication needs. One day you can be more intellectual and the other day you can use more simple language.
Would you mind to elaborate on this?
Absolutely -- I think things like (a)theism, and things like interpretations of QM are "questions of taste." I think it is a waste of time to argue about taste. I also think that tolerance of diverse tastes that agree on all empirical predictions (and agree that empirical predictions is how we go about evaluating things) has advantages.
Thanks. Outside of communities that entertain ideas such as acausal trade and ancestor simulations, I mostly interpret "atheism" to be an imprecise but useful term to communicate the beliefs that (a) any given religion has a negligible probability of being true, and that (b) empirical predictions is how we should go about evaluating things.
This niche has already been filled by the Twelve Virtues.
Twelve Virtues would be a great first chapter in the book. Ten Commandments would be a great last chapter.
The former is about the mindset one needs to start reasoning about rationality; the latter is a list of conclusions to remember.
Ilya's right, it's too long. For example, in Exodus it is written, "Thou shalt not commit adultery." The author doesn't bother defining adultery because his audience shares enough history and culture to know pretty what what's meant. I suspect that you're trying to defend yourself against corner cases and nitpicking. That's a reasonable thing to want (especially in this crowd!) but that's what commentaries are for.
Come to think of it, that division between commandment and commentary might be useful to you. For example, I would rewrite your ninth commandment as, "Thou shalt not assign probability of 0 or 1", and appeal to Rebe Yudkowsky's writings for questions like "but what about epsilon?"
I like that it's self-contained, not a maze of hyperlinks. Could be a bit shorter, though.
Perhaps each point could start with the essence (in bold) and follow with an explanation, like this:
Actually, the Old Testament has three versions of the commandments, each one of different length (Exodus 20, Deuteronomy 5, and the third one I forgot. Fun fact: I learned that at literature lessons in high school, not at any kind of religious lessons). The shorter commandments are the same, but the longer ones differ - maybe it was too difficult even for ancient izraelites to remember them exactly?
Let's try to make some other points shorter.
Number 10. Thou shalt meekly accept battles lost in pursuit of wars won
Number 7. Thou shalt not cease falsificating thine beliefs
I'm only familiar with the two versions of the commandments given in Exodus and Deuteronomy: I specified Exodus specifically to clarify that distinction, then wound up using an example that's the same in both of them. Oh well. I've never heard of a third, though; can you remember any other context?
I'd expect there to be exactly two versions, for the same reason that there are two creation stories in Genesis: the early books of the Bible are the first written form of a faith with two competing (though closely related!) oral traditions.
Anyway, now that I've thought about it more I think this concept would work better as a riff on the book of Proverbs.
Of course, that shows that the Ten Commandments cannot possibly be a basis, rather than summary, of morality, since the commandment "Thou shalt not commit adultery" does not put forth a moral rule, but rather reminds the audience to follow a previously existing moral rule.
Yes. People who say that the ten commandments form the basis of Western morality are not just wrong, but incoherent; there exists no possible history in which that is true.
Absolute-certainty/universal applicability red flag raised.
Silver-lining claim red flag raised.
And by far, most importantly: map-territory conflation red flag raised.
Some possible situations truly can't be improved upon. The fact that you must always be uncertain about whether you are really in one is no help. Just a guarantee that in such a situation a rationalist will always have a little bit of false hope.
Upvoted anyway, most of these are good.
Okay, I acknowledge that "no situation is impossible to improve upon" is not strictly speaking true for literally every conceivable situation, but if ever there is a time where it's acceptable to leave out the ol' BOCTAOE for the sake of prose, I'd say a post including this many thees and thous would be it.
I don't think I conflated map and territory though. The statement "There is never cause for complete hopelessness and despair" is a policy recommendation (read it as: "complete hopelessness and despair is never useful"), not a statement about the territory.
I think "Grant me the serenity to accept the things I cannot change, the courage to change the things I can, and wisdom to know the difference." is much better than wanting to optimize everything.
Too long.
And too purple-prosy. Lines like this:
Need to be rewritten / removed.
The map is not the territory. Rationality is about making effective decisions.
If you as an American ask me whether I come from Berlin, I'm going to say "Yes." I have been born in Berlin. If someone from Berlin asks me I could say: "No. I have been born in Spandau." Spandau is a district of Berlin and there a complex history. Both answer are true because it depends on the context in which the question is asked.
When doing biological modeling there often a tradeoff between complexity of the model and accuracy. Which model you want to use depends on the purpose. If you want to model a whole brain you are going to use a less complex model of a neuron than when you want to model 100 neurons and how those neurons interact with each other.
Beauty is a guiding principle in theoretical physics.
Feeling are a valuable source of information. Shutting down any source of information is no good idea.
Basically you are saying that Eliezer is wrong with Timeless decision theory.
I profess I entirely fail to see how your post refutes the quoted paragraph. Yes, using models is useful, but that is in no way the same as falling prey to wishful thinking. I keep trying to re-read that paragraph to see how it might be interpreted in a way that makes your reply seem natural, but my best guess is that you might have read "Do not let feelings overrule critical thinking or in any other way engage in wishful thinking" as "ignore your feelings". And I still don't see how saying models are useful flows from there.
As far as I know, that sequence is meant to detail ways in which your actions might have indirect/timeless/acausal consequences, and therefore supplements rather than contradicts consequentialism. If I'm wrong, please explain how and why.
Your paragraph doesn't mention anything about wishful thinking. Wishful thinking might be the only thing that comes to mind for you if you think about allowing feelings override critical thinking, but it isn't.
If a sudden feeling of fear triggers in myself and I can't explain with rational thought why a given situation is dangerous or why I would feel fear, I still remove myself from the situation.
There are studies in nurses, that if a nurse get's a feeling that a patient is in a critical situation but the nurse has no evidence that the patient is in a critical situation the patient should still get extra supervision. There good evidence that the nurse should let her intuitive feelings overrule critical thinking if the cost of a false positive is low but the cost of a false negative is high.
In case you want to argue that you can make a rational decision by making an utility calcuation in your head, that might work in the case of the nurses but there are plenty of situation where the time to do that calculation isn't available and it's very useful to respond immediately.
If I dance intimitely with a woman who's a stranger than it's very important that I immediately act when I get the feeling that something isn't right. When I started dancing I tried to get a rational model of what intimicy is or isn't okay and act based on mental rules. It doesn't work that way.
That requires that I can tell the feeling of "touching a woman feels good" apart from "this interaction doesn't flow well, it's better to reduce intimacy". Understanding emotions and being able to tell different ones apart is useful. There are feelings that you should allow to override critical analysis in specific situations, there are other feelings that you shouldn't allow to override critical analysis.
In biological modeling feelings of the person doing the modeling aren't so central that they should override critical thought, but the model still get's optimized for a certain use case and good models often trade some accuracy for simplicity. Simple models are more beautiful and simply beautiful models should be preferred over ugly complicated one if both models predict reality equally well.
It not about the indirect consequences of the action but about the consequences of being the kind of person that engages in specific actions.
Perhaps "consequences" needs to be tabooed. A consequence of something is something that is caused by it, but what does "cause" mean? That's part of what makes Newcomb so paradoxical: it's generally accepted that cause must precede effect, but the hypothetical is set up to treat Omega's actions as depending on a decision after those actions. Are the contents of the boxes included in the category of "consequences" of the choice of how many boxes to take?
I think most people actually mean consequence when they say the word. The difference between someone who practices TDT and someone who does CDT is more than a bunch of semantics. The paragraph describes CDT.
Beware of blaming semantics when you should update one of your core beliefs instead.
Who here actually knows exactly what TDT is? (I am not sure I do -- it was never written down fully -- and I thought about these issues a lot). Are you just assuming people got TDT right? TDT might be "conceptual vaporware". I read an old paper on it, but I didn't like the paper (nor did that paper have a full description).
I think the wiki does contain a written down definition:
I think what Sophronius describes in the paragraph would is what's "intuitively labeled as rational".
I think that's sort of the problem with the post. It's a list of 10 things that intuitively feel like they are the things rational people should do.
It's not a list that tries to describe what reasoned principles about rationalism Lesswrong did come up with. TDT is sort of the LW house decision theory. It's about moving beyond the intuitive idea of rationalism that popular out there. LW rationality is on the other hand supposed to be about winning.
I think the example of reacting when fear comes up is a good example. A nurse should follow the algorithm that if she feels a given patient is in a critical condition the patient gets extra supervision.
The intuitive rational belief that the nurse should have good reasons that she can explain to other people about why a patient needs supervision. The intuitive rational belief is that there should be reasons besides the emotions of the nurse to give the patient extra supervision.
We do have studies that validate the abstract heuristic that the nurse should let her feeling overrule her intellectual analysis of the situation.
If you read the original paper from two decades ago that introduces the concept of evidence-based medicine you find that it's about getting medical professionals to read more scientific papers and deemphasized intuitive decision making.
We learned something in those two decades. We decided that rationality should be about winning. We don't know everything but we can at least make an effort to be less wrong. We know that specific choices are well made with intuition than it would be stupid to not go the winning way and instead try to analyse the situation intellectually. Of course the nurse should still learn medical science but she should also listen to her intuition.
We are in the 21st century and not anymore in the 20st. End 20st century ideology is outdated and it's useful to update. To get less wrong.
Is TDT the best way to think about making decisions? It's still in it's infancy and there still room to refine it. Let's run CFAR workshop to see what heuristics are actually practical when you teach them to humans.
There are a bunch of folk rationality beliefs.
I am sorry, but that is not specified at all. If I give you a specific problem (I have a list of them right here!), will you be able to tell me what "the TDT answer" should be? The way people seem to use TDT is as a kind of "brand name" for a nebulous cloud of decision theoretic ideas. Until there is a paper and a definition, TDT is not a defensible point. It has to be formally written down in order to have a chance to be wrong (being wrong is how we make progress after all).
If it's a set of related decision theories, fine -- tell me what the set is! Example: "naive EDT" is "choose an action that maximizes utility with respect to the distribution p(outcome | action took place)." This is very clear, I know exactly what this is.
A list of exact things at the core of rationality is well-traveled ground; however, if you focus exactly on "things to take away from Less Wrong" I'd write a different list.
I hate flowery langauge and "commandment" style writing, so these are more along the lines of ten Oblique Strategies than ten commandments.
I would say more than anything else these are the things I've taken from Less Wrong, not from the Sequences, nor from the impulse-towards-rationality within all interactions, but from the veneer of community stretched on top of those things.
Really? What if the thing you protect is "all sentient beings," and that happens to be the same as the thing the person who introduced it to you or a celebrity protects? There're some pretty big common choices (Edited to remove inflationary language) or what a human would want to protect.
Beware value hipsterism.
Or, if by "thing to protect", you really mean "means to protect", and you're warning against having the same plan to protect the thing as a celebrity or person who introduced the idea to you, this sounds like "Celebrities and people who introduce people to the idea of means to protect things are never correct and telling the truth about the best available means to protect", which is obviously false.
You, personally, probably don't care about all sentient beings. You probably care about other things. It takes a very rare, very special person to truly care about "all sentient beings," and I know of 0 that exist.
I find it very convenient that most of Less Wrong has the same "thing-to-protect" as EY/SigInst, for the following reasons:
Taken in concert with this quote from the original article:
...it seems obvious to me that most people on LW are brutally abusing the concept of having a thing-to-protect, and thus have no real test for their rationality, making the entire community an exercise in doing ever-more-elaborate performance forms rather than a sparring ground.
I care about other things, yes, but I do care quite a bit about all sentient beings as well (though not really on the level of "something to protect", I'll admit). And I have cared about them before I even heard of Eliezer Yudkowsky. In fact, when I first encountered EY's writing, I figured he did not care about all sentient beings, that he in fact cared about all sapient beings, and was misusing the word like they usually do in science fiction, rather than holding some weird theory of what consciousness is that I haven't heard of anyone else respectable holding, that the majority of neuroscientists disagree with, and that unlike tons of other contrarian positions he holds, he doesn't argue for publicly (I think there might have been one facebook post with an argument about it he made, but I can't find it now).
Something I neglected in the phrase "all sentient beings" is that I care less about "bad" sentient beings, or sentient beings who deliberately do bad things than "good" sentient beings. But even for that classic example of evil, Adolf Hitler, if he were alive, I'd rather that he be somehow reformed than killed.
I may not be able to do FAI research, but I can do what I'm actually doing, which is donating a significant fraction of my income to people who can. (slightly more than 10% of adjusted gross income last tax year, and I'm still a student, so as they say, "This isn't even my final form").
What I've really taken from the person who taught me the concept of a thing-to-protect, is a means-to-protect. If I hadn't been convinced that FAI was a good plan for achieving my values, I would be pursuing lesser plans to achieve my values. I almost started earning to give to charities spreading vegetarianism/veganism instead of MIRI. And I have thought pretty hard about whether this is a good means-to-protect.
Also, though I may not be "thing-to-protect"-level altruistic yet, I'm working on it. I'm more altruistic than I was a few years ago.
This isn't even my final form.
Examples?
I wonder whether you have an understanding of what "solipsism" means that is different from mine. How can a rationalist commandment include a proscription against considering a particular hypothesis?
Disregarding the style of writing, I would just like to say that I think most people on this website could generally do very well with taking commandment number 8 seriously. Meditating on it every night before going to bed might help, or maybe tattooing it on their foreheads so they see it in the mirror when they wake up... 6 foot flaming letters in the sky spelling it out, maybe.
In all seriousness, I cannot count the number of times I get the urge to bash some heads together and yell "Stop being clever! You're here to learn things, not win arguments or score points!"
If anyone has a good idea for how to get this message across in a way that doesn't seem overly combattative, let me know.
On occasion I do get the urge to bash some heads and yell "Stop telling me what I'm here for! My goals are not necessarily what you think they are or what you think they should be!".