Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Rationality is Systematized Winning

48 Post author: Eliezer_Yudkowsky 03 April 2009 02:41PM

Followup toNewcomb's Problem and Regret of Rationality

"Rationalists should win," I said, and I may have to stop saying it, for it seems to convey something other than what I meant by it.

Where did the phrase come from originally?  From considering such cases as Newcomb's Problem:  The superbeing Omega sets forth before you two boxes, a transparent box A containing $1000 (or the equivalent in material wealth), and an opaque box B that contains either $1,000,000 or nothing.  Omega tells you that It has already put $1M in box B if and only if It predicts that you will take only box B, leaving box A behind.  Omega has played this game many times before, and has been right 99 times out of 100.  Do you take both boxes, or only box B?

A common position - in fact, the mainstream/dominant position in modern philosophy and decision theory - is that the only reasonable course is to take both boxes; Omega has already made Its decision and gone, and so your action cannot affect the contents of the box in any way (they argue).  Now, it so happens that certain types of unreasonable individuals are rewarded by Omega - who moves even before they make their decisions - but this in no way changes the conclusion that the only reasonable course is to take both boxes, since taking both boxes makes you $1000 richer regardless of the unchanging and unchangeable contents of box B.

And this is the sort of thinking that I intended to reject by saying, "Rationalists should win!"

Said Miyamoto Musashi:  "The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means.  Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement.  It is essential to attain this.  If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him."

Said I:  "If you fail to achieve a correct answer, it is futile to protest that you acted with propriety."

This is the distinction I had hoped to convey by saying, "Rationalists should win!"

There is a meme which says that a certain ritual of cognition is the paragon of reasonableness and so defines what the reasonable people do.  But alas, the reasonable people often get their butts handed to them by the unreasonable ones, because the universe isn't always reasonable.  Reason is just a way of doing things, not necessarily the most formidable; it is how professors talk to each other in debate halls, which sometimes works, and sometimes doesn't.  If a hoard of barbarians attacks the debate hall, the truly prudent and flexible agent will abandon reasonableness.

No.  If the "irrational" agent is outcompeting you on a systematic and predictable basis, then it is time to reconsider what you think is "rational".

For I do fear that a "rationalist" will clutch to themselves the ritual of cognition they have been taught, as loss after loss piles up, consoling themselves:  "I have behaved virtuously, I have been so reasonable, it's just this awful unfair universe that doesn't give me what I deserve.  The others are cheating by not doing it the rational way, that's how they got ahead of me."

It is this that I intended to guard against by saying:  "Rationalists should win!"  Not whine, win.  If you keep on losing, perhaps you are doing something wrong.  Do not console yourself about how you were so wonderfully rational in the course of losing.  That is not how things are supposed to go.  It is not the Art that fails, but you who fails to grasp the Art.

Likewise in the realm of epistemic rationality, if you find yourself thinking that the reasonable belief is X (because a majority of modern humans seem to believe X, or something that sounds similarly appealing) and yet the world itself is obviously Y.

But people do seem to be taking this in some other sense than I meant it - as though any person who declared themselves a rationalist would in that moment be invested with an invincible spirit that enabled them to obtain all things without effort and without overcoming disadvantages, or something, I don't know.

Maybe there is an alternative phrase to be found again in Musashi, who said:  "The Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size."

"Rationality is the spirit of winning"?  "Rationality is the Way of winning"?  "Rationality is systematized winning"?  If you have a better suggestion, post it in the comments.

Comments (252)

Comment author: iocompletion 14 November 2013 07:58:00PM *  0 points [-]

Rationality leads directly to effectiveness. Or: Rationality faces the truth and therefore produces effectiveness. Or: Rationality is measured by how much effectiveness it produces.

Comment author: Troshen 13 April 2013 06:57:37PM 1 point [-]

I'm not sure if it's better, but here's one that works well. Similar to the phrase, "Physician, heal thyself!" another way to say rationalists should win is to say, "Rationalist, improve thyself!"

If you aren't actually improving yourself and the world around you, then you aren't using the tools of rationality correctly. And it follows that to improve the world around you, you first have to be in a position to do so by doing the same to yourself.

Comment author: katydee 29 March 2013 11:12:08AM 0 points [-]

Typo report: "hoard of barbarians" should be replaced by "horde of barbarians."

Comment author: TheOtherDave 29 March 2013 01:23:13PM 2 points [-]

Hey, you never know when you might need a barbarian... you don't want to run out!

Comment author: CharlieSheen 18 April 2011 11:51:44PM 3 points [-]

"Rationalists should win." is what sold me on this site. Its a good phrase.

Comment author: AdShea 02 December 2010 02:51:48AM 0 points [-]

It seems that most the discussion here is caught up on Omega being able to "predict" your decision would require reverse-time causality which some models of reality cannot allow to exist.

Assuming that Omega is a "sufficiently advanced" powerful being, then the boxes could act in exactly the way that the "reverse time" model stipulates without requiring any such bending of causality through technology that can destroy the contents of a box faster than human perception time or use the classical many-worlds interpretation method of ending the universe where things don't work out the way you want (the universe doesn't even need to end, something like a quantum vacuum collapse would have the same effect of stopping any information leakage in non-conforming universes).

This makes the not-quite-a-rationalist argument of "the boxes are already what they are so my decision doesn't matter, I'll take both" no longer hold true.

Comment author: David_Gerard 02 December 2010 08:55:30AM -1 points [-]

Your assumptions mean that the more likely answer is "Omega is sufficiently powerful to mess with me any way it likes; why am I playing this game?"

That is, problems containing Omega are more contrived and less relevant to anything resembling real life the more one looks at them.

Note that thinking too much about Omega can lead to losing in real life, as one forgets that Omega is hypothetical and cannot possibly exist, and actually goes so far as to attributes the qualities of Omega to what is in fact a manipulative human. One example that I found quite jawdropping. This is a case I think could quite fairly be described as reasoning oneself more ineffective. People who act like that are a reason to get out of the situation, not to invoke TDT.

Comment author: tel 21 December 2009 05:59:59AM 2 points [-]

Perhaps it's not about the ad hominem.

"Rationality is whatever wins."

If it's not a winning strategy, you're not doing it right. If it is a winning strategy, overall in as long of terms as you can plan, then it's rationality. It doesn't matter what the person thinks: whether they'd call themselves rationalists or not.

Comment author: nazgulnarsil 16 April 2009 08:19:05PM 1 point [-]

I one box newcombs problem because the payoffs are too disproportionate to make it interesting. how about this? if omega predicted you would two box they are both empty if omega predicted you would one box both boxes have $1000

Comment author: Normal_Anomaly 08 June 2013 08:55:55PM 0 points [-]

That payoff matrix doesn't preserve the form of the problem. One of the features of the problem is that whatever is in box B, you're better off two-boxing than one-boxing if you ignore the influence of Omega's prediction. A better formulation would be that box A has $1000, and Box B has $2000 iff Omega believes you will one-box. Box B has to potentially have more than box A, or there's no point in one-boxing what ever DT you have.

Comment author: Nebu 14 April 2009 05:25:11PM 0 points [-]

One problem is that "Rationalists should win" has two obvious interpretations for me:

  1. Rationalists should win, therefore if rationalists aren't winning, there's something wrong with the world.
  2. Rationalists should win, therefore if you aren't winning, you're not a rationalist.

Compare with:

  1. People should donate money to Africa, therefore if people aren't donating to Africa, there's something wrong with the world.
  2. People should donate money to Africa, therefore if you don't donate to Africa, you're not a person.

and

  1. Protons should have an electric charge of 1, therefore if protons don't have an electric charge of 1, there's something wrong with the world.
  2. Protons should have an electric charge of 1, therefore if you don't have an electric charge of 1, you are not a proton.
Comment author: Vladimir_Nesov 14 April 2009 05:48:01PM *  2 points [-]

Shouldness in "Rationalists should win" is a much more detailed notion than correspondence to winning situations. It refers to a property of achieving goals, as seen under uncertainty, in our case implemented by cognitive algorithms that search the solution-space for the right plans. Rationalists should-win, have a good measure of win-should-ness.

Looking over it all again, I should add that "rationality is about winning" is also an immensely simpler sentiment, that still seems to retain the gist of the message for which the "rationalists should win" motto was devised.

Comment author: ciphergoth 14 April 2009 05:32:46PM 2 points [-]

What "Rationalists should WIN" needs is a stake through its heart; it is misinterpreted so much more often than it is correctly used that we may need to do without it altogether.

Comment author: JGWeissman 05 April 2009 03:36:56AM 3 points [-]

Alleged rationalists should not find themselves envying the mere decisions of alleged nonrationalists, because your decision can be whatever you like.

Eliezer said this in the Newcomb's Problem post which introduced "Rationalists should win".

Perhaps for a slogan, shorten it to: "Rationalists should not envy the mere decisions of nonrationalists." This emphasizes that rationality contributes to winning through good decisions.

A potential problem is that, in some circumstances, an alleged rationalist could find a factor that seems unrelated to their decisions to blame for losing, and therefore argue that their being rational is consistent with the slogan. For example, a someone who blames losing on luck might need to reconsider their probability theory that is informing their decisions. Though this should not be a fully general counterargument, someone who wins more often than others in the same situation is likely doing something right, even if they do not win with probability 1.

Comment author: timtyler 04 April 2009 12:44:07PM 5 points [-]

Re: "First, foremost, fundamentally, above all else: Rational agents should WIN."

In an attempt to summarise the objections, there seem to be two fairly-fundamental problems:

  1. Rational agents try. They cannot necessarily win: winning is an outcome, not an action;

  2. "Winning" is a poor synonym for "increasing utility": sometimes agents should minimise their losses.

"Rationalists maximise expected utility" would be a less controversial formulation.

Comment author: MarkusRamikin 05 May 2012 10:19:29AM *  0 points [-]

I much prefer "rationalists should win" because it's simple, accessible language. Makes this article more powerful than it would otherwise be. Everyone gets winning; how many people find terms like expected utility maximisation meaningful on a gut level?

Comment author: timtyler 05 May 2012 11:12:43AM -1 points [-]

Charlie Sheen rationality.

Comment author: ciphergoth 04 April 2009 01:13:32PM 3 points [-]

I agree with your two problems, but the problem with your alternative and so many others presented here is that it doesn't so strongly speak to the distinction which EY means to draw, between wanting to be seen to have followed the forms for maximising expected utility and actually seeking to maximise expected utility.

Also, of course, one who at each moment makes the decision that maximises expected future utility defects against Clippy in both Prisoner's Dilemma and Parfit's Hitchhiker scenarios, and arguably two-boxes against Omega, and by EY's definition that counts as "not winning" because of the negative consequences of Clippy/Omega knowing that that's what we do.

Comment author: Normal_Anomaly 08 June 2013 08:59:49PM 0 points [-]

Also, of course, one who at each moment makes the decision that maximises expected future utility defects against Clippy in both Prisoner's Dilemma and Parfit's Hitchhiker scenarios, and arguably two-boxes against Omega, and by EY's definition that counts as "not winning" because of the negative consequences of Clippy/Omega knowing that that's what we do.

I think I'm misunderstanding you here because this looks like a contradiction. Why does making the decision that maximizes expected utility necessarily have negative consequences? It sounds like you're working under a decision theory that involves preference reversals.

Comment author: ciphergoth 09 June 2013 09:44:25AM 1 point [-]

I'm talking about the difference between CDT, which stiffs the lift-giver in Parfit's Hitchhiker and so never gets a lift, and other decision theories.

Comment author: Normal_Anomaly 10 June 2013 10:40:07PM 0 points [-]

Oh, I see. I thought you were saying an optimal decision theory stiffed the lift-giver.

Comment author: ciphergoth 11 June 2013 09:58:45AM 0 points [-]

I hope I've become clearer in the four years since I wrote that!

Comment author: Normal_Anomaly 03 July 2013 08:02:23PM 0 points [-]

. . . did not notice the date-stamp. Good thing thread necros are allowed here.

Comment author: timtyler 04 April 2009 01:42:00PM 1 point [-]

Re: "it doesn't so strongly speak to the distinction which EY means to draw"

I wasn't trying to do that. It seems like a non-trivial concept. Is it important to try and capture that distinction in a slogan?

Re: "one who at each moment makes the decision that maximises expected future utility defects"

Expected utility maximising agents don't have commitment mechanisms, and can't be trusted to make promises? I am sceptical. In my view, you can express practically any agent as an expected utility maximiser. It seems easy enough to imagine commitment mechanisms. I don't see where the problem is.

Comment author: Technologos 21 December 2009 11:08:46AM 0 points [-]

In the Least Convenient Possible World, I imagine nobody has a commitment mechanism in the Prisoner's Dilemma.

Comment author: timtyler 21 December 2009 05:17:19PM 0 points [-]

You can't claim commitment mechanisms are not possible when in fact they evidently are. "Always cooperate" is an example of a strategy which is committed to cooperate in the prisoner's dilemma.

Comment author: Technologos 21 December 2009 07:14:30PM 0 points [-]

"Commitment mechanism" typically means some way to impose a cost on a party for breaking the commitment, otherwise it is, in the game theorist's parlance, "cheap talk" instead. In the one-shot PD, there is by definition no commitment mechanism, and it was in this LCPW that Eliezer's decision theories are frequently tested.

You're talking about the repeated PD with "always cooperate," rather than the one-shot version, which was the scenario in which we found ourselves with Clippy. Please understand--I'm not saying EU-maxing agents do not have commitment mechanisms in general, just that the PD was formulated expressly to show the breakdown of cooperation under certain circumstances.

Regardless, always cooperate definitely does not maximize expected utility in the vast majority of environments. Indeed, it is not part of literally any stable equilibrium in a finite-time RPD. But more to the point, AC is only "committed" in the sense that, if given no opportunities afterward to make decisions, it will appear to produce committed behavior. It is unstable precisely because it requires no further decision points, where the RPD (in which it is played) has them every round.

Comment author: timtyler 21 December 2009 07:54:53PM *  0 points [-]

You and I are using different definitions of "commitment mechanism", then.

The idea I am talking about is demonstrating to the other party that you are a nice, cooperative agent. For example by showing the other agent your source code. That concept has nothing to do with crime and punishment.

The type of commitment mechanism I am talking about is one that convincingly demonstrates that you are committed to a particular course of action under some specified circumstances. That includes commitment via threat of retribution - but also includes some other things.

AC's stability is tangential to my point. If you want to complain that AC is unstable, perhaps consider TFT instead. That is exactly the same as AC on the first round.

Comment author: SoullessAutomaton 04 April 2009 01:06:06PM 2 points [-]

"Rationalists maximise expected utility" would be a less controversial formulation.

But, alas, less catchy.

Comment author: timtyler 04 April 2009 01:22:38PM 5 points [-]

As contagious memes go, "rationalists should win" seems to be rather pathogenic to me. A proposed rationalist slogan shouldn't need so many footnotes. For the sake of minds everywhere, I think it would be best to try to kill it off in its early stages.

Comment author: haig 04 April 2009 08:25:54AM *  1 point [-]

Let's use one of Polya's 'how to solve it' strategies and see if the inverse helps: Irrationalists should lose. Irrationality is systematized losing.

On another note, rationality can refer to either beliefs or behaviors. Does being a rationalist mean your beliefs are rational, your behaviors are rational, or both? I think behaving rationally, even with high probability priors, is still very hard for us humans in a lot of circumstances. Until we have full control of our subconscious minds and can reprogram our cognitive systems, it is a struggle to will ourselves to act in completely rational ways.

To spread rationality, amongst humans at least, we might want to consider a divide and conquer approach focusing on people maintaining rational beliefs first, and maximizing rational behavior second.

Comment author: Thomas 04 April 2009 08:22:55AM 3 points [-]

Both boxes might be transparent. In this case, you would see the money in both boxes only if you are rational enough to understand, that you have to pick just B.

Wouldn't that be an irrational move? Not all! You have to understand that to be rational.

Comment author: Vladimir_Nesov 04 April 2009 06:54:32PM 2 points [-]

That's brilliant! (I'm not sure what you mean by understand though.)

In other words, Omega does one of the two things: it either offers you $1000 + $1, or only $10. It offers you $1000 + $1 only if it predicts that you won't take the $1, otherwise it only gives you $10.

This is a variant of counterfactual mugging, except that there is no chance involved. Your past self prefers to precommit to not taking the $1, while your present self faced with that situation prefers to take the 1$.

Comment author: Thomas 28 April 2009 08:09:02AM 0 points [-]

You have to understand this twist, to be able to call yourself rational, by my book.

You understood the twist, as I see.

Comment author: Vladimir_Nesov 28 April 2009 11:27:50AM 1 point [-]

This reply is too mysterious to reveal whether you got the criterion right.

Comment author: Vladimir_Nesov 04 April 2009 07:53:39PM *  0 points [-]

Hmmm... It looks like the decision to take the $1 determines the situation where you make that decision out of reality. Effects of precommitment being restricted to the counterfactual branches are a usual thing, but in this problem they stare you right in the face, which is rather daring.

Comment author: Vladimir_Nesov 04 April 2009 08:21:07PM *  5 points [-]

Another variation, playing only on real/counterfactual, without motivating the real decision. Omega comes to you and offers $1, if and only if it predicts that you won't take it. What do you do? It looks neutral, since expected gain in both cases is zero. But the decision to take the $1 sounds rather bizarre: if you take the $1, then you don't exist!

Agents self-consistent under reflection are counterfactual zombies, indifferent to whether they are real or not.

Comment author: Will_Newsome 02 July 2011 07:53:01PM -1 points [-]

Seems roughly as disturbing as Wikipedia's article on Gaussian adaptation:

Gaussian adaptation as an evolutionary model of the brain obeying the Hebbian theory of associative learning offers an alternative view of free will due to the ability of the process to maximize the mean fitness of signal patterns in the brain by climbing a mental landscape in analogy with phenotypic evolution.

Such a random process gives us lots of freedom of choice, but hardly any will. An illusion of will may, however, emanate from the ability of the process to maximize mean fitness, making the process goal seeking. I. e., it prefers higher peaks in the landscape prior to lower, or better alternatives prior to worse. In this way an illusive will may appear. A similar view has been given by Zohar 1990. See also Kjellström 1999.

Comment author: Eliezer_Yudkowsky 04 April 2009 08:05:32PM 1 point [-]

If you want your source code to be self-consistent under reflection, you know what you have to do.

Comment author: anonym 04 April 2009 02:36:15AM *  1 point [-]

Rationality is the art of the optimal.

Rationality is optimally systematized systematizing. (:-))

Comment author: byrnema 04 April 2009 01:49:34AM 0 points [-]

Has it been settled then, that in this Newcomb's Problem, rationality and winning are at odds? I think it is quite relevant to this discussion whether or not they ever can be at odds.

My last comment got voted down -- presumably because whether or not rationality and winning are ever in conflict has been discussed in the previous post. (I'm a quick study and would like feedback as to why I get voted down.) However, was there some kind of consensus in the previous post? Do we just assume here that it is possible that rationality is not always the winning strategy? I cannot!

Looking through the comments, it sounds like many people think it is most rational to pick both boxes because of some assumption about how physical reality can't be altered. In a hypothetical reality where that assumption doesn't hold, it would be irrational to insist on applying it.

Comment author: Z_M_Davis 04 April 2009 01:46:35AM 6 points [-]

Rationality seems like a good name for the obvious ideal that you should believe things that are true and use this true knowledge to achieve your goals. Because social organisms are weird in ways whose details are beyond the scope of this comment, striving to be more rational might not pay off for a human seeking to move up in a human world---but aside from this minor detail relating to an extremely pathological case, it's still probably a good idea.

Comment author: knb 04 April 2009 05:33:55AM *  4 points [-]

Hmmm. Unless you are suggesting a different definition for rationality, I think I disagree. If an atheist has the goal of gaining business contacts (or something) and he can further this goal by joining a church, and impersonating the irrational behaviors he sees, he isn't being irrational. While behaviors that tend to have their origins in irrational thought are sometimes rewarded by human society, the irrationality itself never is. I think becoming more rational will help a person move up in a human status hierarchy, if that is the rationalist's goal. I think we have this stereotyped idea of rationalists as Asperger's-afflicted know-it-alls who are unable to deal with irrational humans. It simply doesn't have to be that way.

Comment author: Z_M_Davis 05 April 2009 02:14:36AM 2 points [-]

I denotatively agree with your conclusion, but I think that many if not most aspiring rationalists are incapable of that level of Machiavellianism. Suppose that your typical human cares about both social status and being forthright, and that there are social penalties for making certain true but unpopular statements. Striving for rationality in this situation, could very well mean having to choose between popularity and honesty, whereas the irrationalist can have her cake and eat it, too. So yes, some may choose popularity---but you see, it is a choice.

Comment author: byrnema 03 April 2009 11:42:21PM 0 points [-]

Am I missing something? I think this answer is very simple: rationality and winning are never at odds.

(The only exception is when a rational being has incomplete information. If information tells him that the blue box has $100 and the red box has $0, and it is the other way around, it is rational for him to pick the blue box even though he doesn't win.)

Comment author: HughRistik 04 April 2009 12:03:12AM 1 point [-]

The only exception is when a rational being has incomplete information

Even rational beings usually don't have complete information.

Comment author: byrnema 04 April 2009 02:18:34AM *  1 point [-]

Yes, I agree. I think being rational is always being aware that everything you "know" is a house of cards based on your assumptions. A change in assumptions will require rebuilding the house, and a false room means you need to challenge an assumption.

I'm just arguing that a false room never means that rationality (deduction) itself was wrong (i.e., not winning).

All a rational being can do is base decisions on the information they have. A question: is a rational position based upon incomplete information that leads to not winning really an example of "rationality" not winning? I think that in this discussion we are talking about the relationship between rationality and winning in the context of "enough" information.

Comment author: grobstein 03 April 2009 11:26:56PM *  5 points [-]

Rationality is made of win.

Duhhh!

(Cf.)

Comment author: Furcas 03 April 2009 09:41:12PM 0 points [-]

I already understood what you meant by "rationalists should win", Eliezer, but I don't find Newcomb's problem very convincing as an example. The way I see it, if you one-box you've lost. You could have gotten an extra $1000 but you chose not to.

Comment author: GuySrinivasan 03 April 2009 09:58:21PM *  2 points [-]

And yet those who one-box get $999000 more than those who don't. What gives? If there is a systematic, predictable thing that offers one-boxers $1000000 and offers two-boxers $1000, and there is not a systematic, predictable thing that provides some sort of countering offer to two-boxers, by one-boxing you still get more money.

I can't think of something of equal power level (examining your decisions, not your method of arriving at those decisions) which would be able to provide the countering offer to two-boxers.

Comment author: thomblake 03 April 2009 10:16:12PM 1 point [-]

Simple: most situations in real life aren't like this. If you believe Omega and one-box, you'll lose when he's lying. If your decision theory works better in hypothetical situations and worse in real life, then it doesn't make you win.

Comment author: orthonormal 04 April 2009 01:33:52AM *  8 points [-]

Apply the Least Convenient Possible World principle.

Also, I don't think Eliezer keeps harping on Newcomb's problem because he anticipates experiencing precisely that scenario. I see several important points that I don't think have been clearly made (not that I'm the one to do so):

  1. We can choose whether and when to implement certain decision algorithms, including classical causal decision theory (CCDT). This choice may in fact be trivial, or it may be subtle, but it is a worthy question for a rationalist.

  2. Although, for any fixed set of options, implementing CCDT maximizes your return, there are in fact cases where the options you have depend on the outcome of a model of your decision algorithm. I'm not talking about Omega, I'm talking about human social life. We base a large portion of our interactions with others on our anticipations of how they might respond. (This isn't often done rationally by anyone's standards, but it can be.)

  3. It gets confusing (in particular, Hofstadterian) here, but a plausibly better outcome might be reached in the Prisoner's Dilemma by selfish non-strangers mutually modeling the other's likely decision process, and recognizing that only C-C and D-D are stable outcomes under mutual modeling.

Of course, I still feel a bit uncomfortable with this line of reasoning.

Comment author: Furcas 03 April 2009 10:11:26PM *  2 points [-]

Of course the one-boxers get more money: They were put in a situation in which they could either get $1 000 000 or $1 001 000, whereas the two-boxers were put in a situation in which they could get $0 or $1000.

It makes no sense to compare the two decisions the way you and Eliezer do. It's like organizing a swimming competition between an Olympic athlete who has to swim ten kilometers to win and an untrained fatass who only has to swim a hundred meters to win, and concluding that because the fatass wins more often than the athlete, therefore fatasses clearly make better swimmers than athletes.

Comment author: JGWeissman 04 April 2009 04:03:41AM 3 points [-]

Of course the one-boxers get more money: They were put in a situation in which they could either get $1 000 000 or $1 001 000, whereas the two-boxers were put in a situation in which they could get $0 or $1000.

When faced with this decision, you are either in the real world, in which case you can get an extra $1000 by two boxing, or you are in a simulation, in which case you can arrange so your self in the real world gets and extra $1,000,000 by one boxing. Given that you can't tell which of these is the case, and that you are deterministic, you will make the same decision in both situations. So your choice is to either one box and gain $1,000,000 or two box and gain $1000. If you like having more money, it seems clear which of those choices is more rational.

Comment author: SoullessAutomaton 03 April 2009 10:33:33PM 1 point [-]

But if you were put into said hypothetical competition, and could somehow decide just before the contest began whether to be an Olympic athlete or an untrained fatass, which would you choose?

I think you're getting overly distracted by the details of the problem construction and missing the point.

Comment author: Furcas 03 April 2009 10:44:11PM 1 point [-]

If my only goal were to win that particular competition (and not to be a good swimmer), of course I'd choose to turn into a fatass and lose all my training. Likewise, if I could choose to precommit to one-boxing in Newcomb-like problems, I would, because pre-commitment has an effect on what will be in box B (whereas the actual decision does not).

The details are what makes Newcomb's problem what it is, so I don't see how it's possible to get "overly distracted" by them. Correct me if I'm wrong, but pre-commitment isn't an option in Newcomb's problem, so the best, the most rational, the winning decision is two-boxing.

Comment author: SoullessAutomaton 03 April 2009 11:11:02PM *  4 points [-]

Correct me if I'm wrong, but pre-commitment isn't an option in Newcomb's problem, so the best, the most rational, the winning decision is two-boxing.

By construction, Omega's predictions are known to be essentially infallible. Given that, whatever you choose, you can safely assume Omega will have correctly predicted that choice. To what extent, then, is pre-commitment distinguishable from deciding on the spot?

In a sense there is an implicit pre-commitment in the structure of the problem; while you have not pre-committed to a choice on this specific problem, you are essentially pre-committed to a decision-making algorithm.

Eliezer's argument, if I understand it, is that any decision-making algorithm that results in two-boxing is by definition irrational due to giving a predictably bad outcome.

Comment author: orthonormal 04 April 2009 12:59:58AM 1 point [-]

In a sense there is an implicit pre-commitment in the structure of the problem; while you have not pre-committed to a choice on this specific problem, you are essentially pre-committed to a decision-making algorithm.

That's an interesting, and possibly fruitful, way of looking at the problem.

Comment author: Furcas 03 April 2009 11:30:41PM *  1 point [-]

Pre-commitment is different from deciding on the spot because once you're on the spot, there is nothing, absolutely nothing you can do to change what's in box B. It's over. It's a done deal. It's beyond your control.

My understanding of Eliezer's argument is the same as yours. My objection is that two-boxing doesn't actually give a bad outcome. It gives the best outcome you can get given the situation you're in. That you don't know what situation you're in until after you've opened box B doesn't change that fact. As Eliezer is so fond of saying, the map isn't the territory.

Comment author: SoullessAutomaton 04 April 2009 12:59:58AM 1 point [-]

Pre-commitment is different from deciding on the spot because once you're on the spot, there is nothing, absolutely nothing you can do to change what's in box B.

If your decision on the spot is 100 percent predictable ahead of time, as is explicitly assumed in the problem construction, you are effectively pre-committed as far as Omega is concerned. You, apparently, have essentially pre-committed to opening two boxes.

My objection is that two-boxing doesn't actually give a bad outcome. It gives the best outcome you can get given the situation you're in.

And yet, everyone who opens one box does better than the people who open two boxes.

You seem to have a very peculiar definition of "best outcome".

Comment author: Furcas 04 April 2009 02:03:06AM *  0 points [-]

If your decision on the spot is 100 percent predictable ahead of time, as is explicitly assumed in the problem construction, you are effectively pre-committed as far as Omega is concerned. You, apparently, have essentially pre-committed to opening two boxes.

What I meant by 'pre-commitment' is a decision that we can make if and only if we know about Newcomb-like problems before being faced with one. In other words, it's a decision that can affect what Omega will put in box B. That Omega can deduce what my decision will be doesn't mean that the decision is already taken.

And yet, everyone who opens one box does better than the people who open two boxes.

And every fatass who competes against an Olympic athlete in the scenario I described above does 'better' than the athlete. So what? Unless the athlete knows about the competition's rules ahead of time and eats non-stop to turn himself into a fatass, there's not a damn thing he can do about it, except try his best once the competition starts.

You seem to have a very peculiar definition of "best outcome".

It seems too obvious to say, but I guess I have to say it. "The best outcome" in this context is "the best outcome that it is possible to achieve by making a decision". If box B contains nothing, then the best outcome that it is possible to achieve by making a decision is to win a thousand dollars. If box B contains a million dollars, then the best outcome that it is possible to achieve by making a decision is to win one million and one thousand dollars.

Well, I don't see how I can explain myself more clearly than this, so this will be my last comment on this subject. In this thread. This week. ;)

Comment author: Kenny 12 April 2009 04:59:59PM 2 points [-]

This exchange has finally imparted a better understanding of this problem for me.

If you pre-commit now to always one-box – and you believe that about yourself – then deciding to one-box when Omega asks you is the best decision.

If you are uncertain of your commitment then you probably haven't really pre-committed! I haven't tried to math it, but I think your actual decision when Omega arrives would depend on the strength of your belief about your own pre-commitment. [Though a more-inconvenient possible world is the one in which you've never heard of this, or similar, puzzles!]

Now I grok why rationality should be self-consistent under reflection.

Comment author: grobstein 03 April 2009 11:26:07PM -1 points [-]

Eliezer's argument, if I understand it, is that any decision-making algorithm that results in two-boxing is by definition irrational due to giving a predictably bad outcome.

So he's assuming the conclusion that you get a bad outcome? Golly.

Comment author: HughRistik 03 April 2009 11:35:45PM *  3 points [-]

True, we don't know the outcome. But we should still predict that it will be bad, due to Omega's 99% accuracy rate.

Don't mess with Omega.

Comment author: William 03 April 2009 11:31:02PM 1 point [-]

The result of two-boxing is a thousand dollars. The result of one-boxing is a million dollars. By definition, a mind that always one-boxes receives a better payout than one that always two-boxes, and therefore one-boxing is more rational, by definition.

Comment author: orthonormal 04 April 2009 01:36:58AM 1 point [-]

See Arguing "By Definition". It's particularly problematic when the definition of "rational" is precisely what's in dispute.

Comment author: Furcas 03 April 2009 11:41:32PM *  1 point [-]

The result of two-boxing is a thousand dollars more than you would have gotten otherwise. The result of one-boxing is a thousand dollars less than you would have gotten otherwise. Therefore two-boxing is more rational, by definition.

What determines whether you'll be in a 1M/1M+1K situation or in a 0/1K situation is the kind of mind you have, but in Newcomb's problem you're not given the opportunity to affect what kind of mind you have (by pre-commiting to one-boxing, for example), you can only decide whether to get X or X+1K, regardless of X's value.

Comment author: GuySrinivasan 04 April 2009 12:48:42AM 2 points [-]

Suppose for a moment that one-boxing is the Foo thing to do. Two-boxing is the expected-utility-maximizing thing to do. Omega decided to try to reward those minds which it predicts will choose to do the Foo thing with a decision between doing the Foo thing and gaining $1000000, and doing the unFoo thing and gaining $1001000, while giving those minds which will choose to do the unFoo thing a decision between doing the Foo thing and gaining $0 and doing the unFoo thing and gaining $1000.

The relevant question is whether there is a generalization of the computation Foo which we can implement that doesn't screw us over on all sorts of non-Newcomb problems. Drescher for instance claims that acting ethically implies, among other things, doing the Foo thing, even when it is obviously not the expected-utility-maximizing thing.

Comment author: thomblake 03 April 2009 10:36:55PM -1 points [-]

You're assuming that you can just choose how you go about making decisions every time you make a decision. If you're not granted that assumption, Furcas's analysis is spot on. Two-boxers succeed in other places and also on Newcomb; one-boxers fail in many situations that are similar to Newcomb but not as nice. So you need to decide what sort of decisions you'll make in general, and that will (arguably) determine how much money is in the boxes in this particular experiment.

Comment author: Nick_Tarleton 04 April 2009 07:50:09AM 2 points [-]

one-boxers fail in many situations that are similar to Newcomb but not as nice.

Such as?

(Is this meant to refer to failures of evidential decision theory? There are other options.)

Comment author: grobstein 03 April 2009 10:17:09PM 1 point [-]

This premise is not accepted by the 1-box contingent. Occasionally they claim there's a reason.

Comment author: byrnema 04 April 2009 02:22:40AM 1 point [-]

Can you please elaborate? I'm trying to catch up!

Comment author: Furcas 03 April 2009 10:30:42PM 1 point [-]

You mean they don't accept that the decision doesn't affect what's in box B?

Comment author: PeteG 03 April 2009 09:20:15PM 3 points [-]

Rationality is winning that doesn’t generate a surprise; randomly winning the lottery generates a surprise. A good measure of rationality is the amount of complexity involved in order to win, and the surprise generated by that win. If to win at a certain task requires that your method have many complex steps, and you win, non-surprisingly, then the method used was a very rational one.

Comment author: taw 03 April 2009 09:09:13PM -1 points [-]

Please ... Newcomb is a toy non-mathematizable problem and not a valid argument for anything at all. There must be a better example, or the entire problem is invalid.

Comment author: gwern 03 April 2009 10:09:32PM 6 points [-]

There must be a better example, or the entire problem is invalid.

I've long thought that voting in general is largely isomorphic to Newcomb's. If you cop out and don't vote, then everyone like you will reason the same way and not vote, and your favored candidates/policies will fail; but if you vote then the reverse might happen; and if you then carry it one more step... If you could just decide to one-box/vote then maybe everyone else like you will.

Comment author: cousin_it 04 April 2009 01:11:14AM *  1 point [-]

Sorry, in voting you don't play the singular boss role that you play in Newcomb's problem. But it's amusing how far democracy proponents will go to convince themselves that their vote matters. :-)

Comment author: gwern 04 April 2009 01:34:09AM 3 points [-]

I haven't worked it out rigorously (else you would've seen a post on it by now!), but it seems to me in close elections (Florida 2000, say) that thought process could be valid. Considering how small the margins sometimes are, and how much of the electorate doesn't vote, it doesn't strike me as implausible that there are enough people thinking like me to make a difference.

And of course we could just specify as a condition that you and yours are a bloc powerful enough to affect the election. (Maybe you're numerous, maybe there're only a few electors, whatever.)

But it's amusing how far democracy proponents will go to convince themselves that their vote matters.

The problem with irrelevant ad hominems is that they're very often based on flimsy evidence and so often wrong. I didn't even vote last year because I figured my vote didn't matter. I was not surprised.

Comment author: cousin_it 04 April 2009 10:56:07AM *  2 points [-]

In Newcomb's problem you're the boss, e.g. you can assign yourself a suitable utility function beforehand to keep the million and screw the thousand. Not so in voting - no matter what you think, other people won't change. They don't have anything conditioned on the outcome of your thought process, as in Newcomb's. No, not even if "people thinking like you" are a bloc. You still can't influence them. It's a coordination game, not Newcomb's.

Your reasoning resembles the "twins fallacy" in the Prisoner's Dilemma: the idea that just by choosing to cooperate you can magically force your identical partner to do the same. Come to think of it, PD sounds like a better model for voting to me.

Update: Eliezer seems to think PD and Newcomb's are related. Not sure why.

Comment author: grobstein 03 April 2009 09:10:59PM 2 points [-]

Please ... Newcomb is a toy non-mathematizable problem and not a valid argument for anything at all.

Why?

Comment author: taw 03 April 2009 10:32:42PM 5 points [-]

As far as I can tell, Newcomb problem exists only in English, and only because a completely aphysical causality loop is introduced. Every mathematization I've ever seen collapses it to either trivial one-boxing problem, or trivial two-boxing problem.

If anybody wants this problem to be treated seriously, maths first to show the problem is real! Otherwise, we're really not much better than if we were discussing quotes from the Bible.

Comment author: Douglas_Knight 05 April 2009 11:27:44PM 5 points [-]

If you've seen formalizations, then it is formalizable. What are the formalizations?

Since I think the answer is obviously one-box, it doesn't surprise me that there is a formalization in which that answer is obvious. I have never seen a formalization in which the answer is two-box. I have seen the argument that "causal decision theory" (?) chooses to two-box. People jump from that to the conclusion that the answer is two-box, but that is an idiotic conclusion. Given the premise, the correct conclusion is that this decision theory is inadequate. Anyhow, I don't believe the argument. I interpret it simply as the decision theory failing to believe the statement of the problem. There is a disconnect between the words and the formalization of that decision theory.

The issue is not about formalizing Newcomb's problem; the problem is creating a formal decision theory that can understand a class of scenarios including Newcomb's problem. (It should be possible to tweak the usual decision theory to make it capable of believing Newcomb's problem, but I don't think that would be adequate for a larger class of problems.)

Comment author: infotropism 03 April 2009 08:37:58PM 4 points [-]

The rationality that doesn't secure your wish isn't the true rationality.

Winning has no fixed form. You'll do whatever is needed to succeed, however original or far fetched it would sound. How it sounds is irrelevant, how it works is the crux.

And If at first what you tried didn't work, then you'll learn, adapt, and try again, making no pause for excuses, if you merely want to succeed, you'll be firm as a rock, relentless in your attempts to find the path to success.

And if your winning didn't go as smoothly or well as you wanted or thought it should, in general, then learn, adapt, and try again. Think outside of the box, self recurse on winning itself. Eventually, you should refine and precise your methods into a tree, from general to specialized.

That tree will have a trunk of general cases and methods used to solve those, and any case that lies ahead, upwards on the tree; and the higher you go, the more specialized the method, the rarer the case it solves. The tree isn't fixed either, it can and will grow and change.

Comment author: timtyler 04 April 2009 08:02:21AM 1 point [-]

Re: The rationality that doesn't secure your wish isn't the true rationality.

Again with the example of handicap chess. You start with no knight. You wish to win. Actually you lose. Does that mean you were behaving irrationally? No, of course not! It is not whether you win or lose, but how you play the game.

Comment deleted 04 April 2009 07:22:15AM [-]
Comment author: Eliezer_Yudkowsky 04 April 2009 12:02:42PM 3 points [-]

Say "Rationalists are" instead of "Rationality is" and I'll agree with that.

Comment author: JamesAndrix 03 April 2009 06:50:22PM 5 points [-]

It seems to me that the disagreement isn't so much about winning as the expectation.

In fact I don't really agree with this winning vs. belief modes of rationality.

Both approaches are trying to maximize their expected payout. Eliezer's approach has a wider horizon of what it considers when figuring out what the universe is like.

The standard approach is that since the content of the boxes is already determined at the time of the choice, so taking both will always put you $1000 ahead.

Eliezer looks (I think) out to the most likely final outcomes. (or looks back at how the chain of causality of one's decision is commingled with the chain of causality of Omega's decision. )

I think that flaw in the standard approach is not 'not winning' but a false belief about the relationship between the boxes and your choices. (the belief that there isn't any.) Once you have the right answer, making the choice that wins is obvious.

The way we would know that the standard approach is the wrong one is by looking at results. That a certain set of choices consistently wins isn't evidence that it is rational, it is evidence that it wins. Believing that it wins is rational.

So maybe: "Rationality is learning how to win"

Comment author: CronoDAS 03 April 2009 08:00:42PM 2 points [-]

"Rationality is learning how to win"

I like that.

Comment author: orthonormal 03 April 2009 06:39:02PM 6 points [-]

I always thought that the majority of exposition in your Newcomb example went towards, not "Rationalists should WIN", but a weaker claim which seems to be a smaller inferential distance from most would-be rationalists:

Rationalists should not systematically lose; whatever systematically loses is not rationality.

(Of course, one needs the logical caveat that we're not dealing with a pure irrationalist-rewarder; but such things don't seem to exist in this universe at the moment.)

Comment author: timtyler 04 April 2009 08:10:19AM 0 points [-]

Re: Rationalists should not systematically lose; whatever systematically loses is not rationality.

Even if you are playing go with a 9-stone handicap against a shodan?

Comment author: orthonormal 05 April 2009 04:43:18AM 2 points [-]

Well, I don't think I'd fare better by thinking less rationally; and if I really needed to find a way to win, rationality at least shouldn't hurt me me in the process.

I was hoping to be pithy by neglecting a few implicit assumptions. For one, I mean that (in the absence of direct rewards for different cognitive processes) good rationalists shouldn't systematically lose when they can see a strategy that systematically wins. Of course there are Kobayashi Maru scenarios where all the rationality in the world can't win, but that's not what we're talking about.

Comment author: Nick_Tarleton 04 April 2009 08:14:07AM 3 points [-]

"Lose" = "perform worse than another (usable) strategy, all preferences considered".

Comment author: timtyler 04 April 2009 08:17:46AM -1 points [-]

Nick, show me a dictionary with this in and we can talk. Otherwise, it seems as though you are redefining a perfectly common and ordinary english word to mean something esoteric and counter-intuitive.

Comment author: Emile 03 April 2009 06:11:33PM 1 point [-]

Like William, I think "winning" is the problem, though for different reasons : "winning" has extra connotations, and tends to call up the image of the guy who climbs the corporate ladder through dishonesty and betrayal rather than trying to lead a happy and fulfilling life. Or someone who tries to win all debates by humiliating his opponent till nobody wants to speak to him any more.

Winning often doesn't mean getting what you want, but winning at something defined externally, or competing with others - which may indeed not always the rational thing to do.

The thread on the Rationality Questionaire seemed to have this problem - some questions seemed more focused on "winning as understood by society" rather then getting what you want.

Comment author: timtyler 03 April 2009 05:18:56PM 12 points [-]

Wikipedia has this right:

"a rational agent is specifically defined as an agent which always chooses the action which maximises its expected performance, given all of the knowledge it currently possesses."

Expected performance. Not actual performance. Whether its actual performance is good or not depends on other factors - such as how malicious the environment is, whether the agent's priors are good - and so on.

Comment author: Eliezer_Yudkowsky 03 April 2009 05:55:44PM 4 points [-]

Problem with that in human practice is that it leads to people defending their ruined plans, saying, "But my expected performance was great!" Vide the failed trading companies saying it wasn't their fault, the market had just done something that it shouldn't have done once in the lifetime of the universe. Achieving a win is much harder than achieving an expectation of winning (i.e. something that it seems you could defend as a good try).

Comment author: anonym 04 April 2009 01:55:43AM 5 points [-]

It sounds like the objection you're giving here is that "some people will misinterpret expected performance in the technical sense as expected performance in the colloquial sense (i.e., my guess as to how things will turn out)." That doesn't seem like much of a criticism though, and it doesn't sound severe enough to throw out what is a pretty standard definition. People will also misinterpret your alternate definition, as we have seen.

Do you have other objections?

Comment author: Jonathan_Graehl 03 April 2009 09:34:26PM 5 points [-]

What you say is important: the vast majority of whining "rationalists" weren't done dirty by a universe that "nobody could have foreseen" (the sub-prime mortgage crisis/piloting jets into buildings). If you sample a random loser claiming such (my reasoning was flawless, my priors incorporated all feasibly available human knowledge), an impartial judge would in nearly all cases correctly call them to task.

But clearly it's not always the case that my reasoning (and/or priors) is at fault when I lose. My updates shouldn't overshoot based on empirical noise and false humility. I think what you want to say is that most likely even (especially?) the most proud rationalists probably shield themselves from attributing their loss to their own error ("eat less salt").

I'd like some quantifiable demonstration of an externalizing bias, some calibration of my own personal tendency to deny evidence of my own irrationality (or of my wrong priors).

Comment author: HughRistik 03 April 2009 07:46:08PM *  10 points [-]

Problem with that in human practice is that it leads to people defending their ruined plans, saying, "But my expected performance was great!"

It's true that people make this kind of response, but that doesn't make it valid, or mean that we have to throw away the notion of rationality as maximizing expected performance, rather than actual performance.

In the case of failed trading companies, can't we just say that despite their fantasies, their expected performance shouldn't have been so great as they thought? And the fact that their actual results differed from their expected results should cast suspicion on their expectations.

Perhaps we can say that expectations about performance be epistemically rational, and only then can an agent who maximizes their expected performance be instrumentally rational.

Achieving a win is much harder than achieving an expectation of winning (i.e. something that it seems you could defend as a good try).

Some expectations win. Some expectations lose. Yet not all expectations are created equal. Non-accidental winning starts with something that seems good to try (can accidental winning be rational?). At least, there is some link between expectations and rationality, such that we can call some expectations more rational than others, regardless of whether they actually win or lose.

An example SoullessAutomaton made was that we shouldn't consider lottery winners rational, even though they won, because they should not have expected to. Conversely, all sorts of inductive expectations can be rational, even though sometimes they will fail due to the problem of induction. For instance, it's rational to expect that the sun will rise tomorrow. If Omega decides to blow up the sun, my expectation will still have been rational, even though I turned out to be wrong.

Comment author: SoullessAutomaton 03 April 2009 09:35:06PM 2 points [-]

Yet not all expectations are created equal. Non-accidental winning starts with something that seems good to try (can accidental winning be rational?).

In the real world, of course, most things are some mixture of controllable and randomized. Depending on your definition of accidental, it can be rational to make low-cost steps to position yourself to take advantage of possible events you have no control over. I wouldn't call this accidental, however, because the average expected gain should be net positive, even if one expects (id est, with confidence greater than 50%) to lose.

I used the lottery as an example because it's generally a clear-cut case where the expected gain minus the cost of participating is net negative and the controllable factor (how many tickets you buy) has extremely small impact.

Comment author: HughRistik 03 April 2009 09:58:51PM 3 points [-]

Yes, and I liked your example for exactly this reason: the expected value of buying lottery tickets is negative.

I think that this shows that it is irrational to take an action where it's clear-cut that the expected value is negative, even though due to chance, one iteration of that action might produce a positive result. You are using accidental the same way I am: winning from an action with a negative expected value is what I would call accidental, and winning with a positive expected value is non-accidental.

Things are a bit more complicated when we don't know the expected value of an action. For example, in Eliezer's examples of failed trading companies, we don't know the correct expected value of their trading strategies, or whether they were positive or negative.

In cases where the expected value of an action is unknown, perhaps the instrumental rationality of the action is contingent on the epistemic rationality of our estimation of its expected value.

Comment author: SoullessAutomaton 03 April 2009 10:24:16PM 3 points [-]

I like your definition of an accidental win, it matches my intuitive definition and is stated more clearly than I would have been able to.

In cases where the expected value of an action is unknown, perhaps the instrumental rationality of the action is contingent on the epistemic rationality of our estimation of its expected value.

Yes. Actually, I think the "In cases where the expected value of an action is unknown" clause is likely unnecessary, because the accuracy of an expected value calculation is always at least slightly uncertain.

Furthermore, the second-order calculation of the expected value of expending resources to increase epistemological rationality should be possible; and in the case that acting on a proposition is irrational due to low certainty, and the second-order value of increasing certainty is negative, the rational thing to do is shrug and move on.

Comment author: Technologos 03 April 2009 07:15:34PM 3 points [-]

I'm not sure how you can implement an admonition to Win and not just to (truly, sincerely) try. What is the empirical difference?

I suppose you could use an expected regret measure (that is, the difference between the ideal result and the result of the decision summed across the distribution of probable futures) instead of an expected utility measure.

Expected regret tends to produce more robust strategies than expected utility. For instance, in Newcomb's problem, we could say that two-boxing comes from expected utility but one-boxing comes from regret-minimizing (since a "failed" two-box gives $1,000,000-$1,000=$999,000 of regret, if you believe Omega would have acted differently if you had been the type of person to one-box, where a "failed" one-box gives $1000-$0=$1,000 of regret).

Using more robust strategies may be a way to more consistently Win, though perhaps the true goal should be to know when to use expected utility and when to use expected regret (and therefore to take advantage both of potential bonanzas and of risk-limiting mechanisms).

Comment author: robzahra 03 April 2009 10:55:43PM *  5 points [-]

I'm quite confident there is only a language difference between eliezer's description and the point a number of you have just made. Winning versus trying to win are clearly two different things, and it's also clear that "genuinely trying to win" is the best one can do, based on the definition those in this thread are using. But Eli's point on ob was that telling oneself "I'm genuinely trying to win" often results in less than genuinely trying. It results in "trying to try"...which means being satisfied by a display of effort rather than utility maximizing. So instead, he arguesn why not say to oneself the imperative "Win!", where he bakes the "try" part into the implicit imperative. I agree eli's language usage here may be slightly non standard for most of us (me included) and therefore perhaps misleading to the uninitiated, but I'm doubtful we need to stress about it too much if the facts are as I've stated. Does anyone disagree? Perhaps one could argue eli should have to say, "Rational agents should win_eli" and link to an Explanation like this thread, if we are genuinely concerned about people getting confused.

Comment author: timtyler 04 April 2009 05:06:29AM 2 points [-]

Eliezer seems to be talking about actually winning - e.g.: "Achieving a win is much harder than achieving an expectation of winning".

He's been doing this pretty consistently for a while now - including on his administrator's page on the topic:

"Instrumental rationality: achieving your values."

That is why this discussion is still happening.

Comment author: grobstein 03 April 2009 08:45:48PM 5 points [-]

Here's a functional difference: Omega says that Box B is empty if you try to win what's inside it.

Comment author: byrnema 04 April 2009 01:58:56AM *  1 point [-]

Yes! This functional difference is very important!

In Logic, you begin with a set of non-contradicting assumptions and then build a consistent theory based on those assumptions. The deductions you make are analogous to being rational. If the assumptions are non-contradicting, then it is impossible to deduce something false in the system. (Analogously, it is impossible for rationality not to win.) However, you can get a paradox by having a self-referential statement. You can prove that every sufficiently complex theory is not closed -- there are things that are true that you can't prove from within the system. Along the same lines, you can build a paradox by forcing the system to try to talk about it itself.

What Grobstein has presented is a classic paradox and is the closest you can come to rationality not winning.

Comment author: Technologos 28 April 2009 07:46:05PM *  0 points [-]

I understand all that, but I still think it's impossible to operationalize an admonition to Win. If

Omega says that Box B is empty if you try to win what's inside it.

then you simply cannot implement a strategy that will give you the proceeds of Box B (unless you're using some definition of "try" that is inconsistent with "choose a strategy that has a particular expected result").

I think that falls under the "ritual of cognition" exception that Eliezer discussed for a while: when Winning depends directly on the ritual of cognition, then of course we can define a situation in which rationality doesn't Win. But that is perfectly meaningless in every other situation (which is to say, in the world), where the result of the ritual is what matters.

Comment author: timtyler 03 April 2009 08:10:50PM *  5 points [-]

Agents do try to win. The don't necessarily actually win. For example, if they face a superior opponent. Kasparov was behaving in a highly rational manner in his battle with Deep Blue. He didn't win. He did try to, though. Thus the distinction between trying to win and actually winning.

Comment author: thomblake 03 April 2009 08:00:05PM 2 points [-]

see http://www.overcomingbias.com/2008/10/trying-to-try.html

It's really easy to convince yourself that you've truly, sincerely tried - trying to try is not nearly as effective as trying to win.

Comment author: Technologos 28 April 2009 08:03:02PM 0 points [-]

I agree. I'm just noting that an admonition to Win is strictly an admonition to try, phrased more strongly. Winning is not an action--it is a result. All I can suggest are actions that get you to that result.

I can tell you "don't be satisfied with trying and failing," but that's not quite the same.

Comment author: timtyler 03 April 2009 08:23:17PM *  0 points [-]

As for the "Trying-to-try" page - an argument from Yoda and the Force? It reads like something out of a self-help manual!

Sure: if you are trying to inspire confidence in yourself in order to improve your performance, then you might under some circumstances want to think only of winning - and ignore the possibility of trying and failing. But let's not get our subjects in a muddle, here - the topic is the definition of instrumental rationality, not how some new-age self-help manual might be written.

Comment author: timtyler 03 April 2009 08:06:21PM 2 points [-]

The intended distinction was originally between trying to win and actually winning. You are comparing two kinds of trying.

Comment author: thomblake 03 April 2009 08:24:28PM 4 points [-]

You are comparing two kinds of trying.

I'm not sure how you can implement an admonition to Win and not just to (truly, sincerely) try. What is the empirical difference?

Based on the above, I believe the distinction was between two different kinds of admonitions. I was pointing out that an admonition to win will cause someone to try to win, and an admonition to try will cause someone to try to try.

Comment author: Eliezer_Yudkowsky 04 April 2009 11:37:28AM 3 points [-]

Thomblake's interpretation of my post matches my own.

Comment author: timtyler 03 April 2009 08:39:40PM 4 points [-]

Right, but again, the topic is the definition of instrumental rationality, and whether it refers to "trying to win" or "actually winning".

What do "admonitions" have to do with things? Are you arguing that because telling someone to "win" may some positive effect that telling someone to "try to win" lacks that we should define "instrumental rationality" to mean "winning" - and not "trying to win"?

Isn't that an idiosyncracy of human psychology - which surely ought to have nothing to do with the definition of "instrumental rationality".

Consider the example of handicap chess. You start with no knight. You try to win. Actually you lose. Were you behaving rationally? I say: you may well have been. Rationality is more about the trying, than it is about the winning.

Comment author: thomblake 03 April 2009 08:45:33PM 1 point [-]

The question was about admonitions. I commented based on that. I didn't mean anything further about instrumental rationality.

Comment author: timtyler 03 April 2009 09:08:16PM 4 points [-]

OK. I don't think we have a disagreement, then.

I consider it to be a probably-true fact about human psychology that if you tell someone to "try" rather than telling them to "win" then that introduces failure possibilites into their mind. That may have a positive effect, if they are naturally over-confident - or a negative one, if they are naturally wracked with self-doubt.

It's the latter group who buy self-help books: the former group doesn't think it needs them. So the self-help books tell you to "win" - and not to "try" ;-)

Comment author: timtyler 03 April 2009 06:48:19PM 13 points [-]

Expected performance is what rational agents are actually maximising.

Whether that corresponds to actual performance depends on what their expectations are. What their expectations are typically depends on their history - and the past is not necessarily a good guide to the future.

Highly rational agents can still lose. Rational actions (that follow the laws of induction and deduction applied to their sense data) are not necessarily the actions that win.

Rational agents try to win - and base their efforts on their expectations. Whether they actually win depends on whether their expectations are correct. In my view, attempts to link rationality directly to "winning" miss the distinction between actual and expected utility.

There are reasons for associations between expected performance and actual performance. Indeed, those associations are why agents have the expectations they do. However, the association is statistical in nature.

Dissect the brain of a rational agent, and it is its expected utility that is being maximised. Its actual utility is usually not something that is completely under its control.

It's important not to define the "rational action" as "the action that wins". Whether an action is rational or not should be a function of an agent's sense data up to that point - and should not vary depending on environmental factors which the agent knows nothing about. Otherwise, the rationality of an action is not properly defined from an agent's point of view.

I don't think that the excuses humans use for failures is an issue here.

Behaving rationally is not the only virtue needed for success. For example, you also need to enter situations with appropriate priors.

Only if you want rationality to be the sole virtue, should "but I was behaving rationally" be the ultimate defense against an inquisition.

Rationality is good, but to win, you also need effort, persistence, good priors, etc - and it would be very, very bad form to attempt to bundle all those into the notion of being "rational".

Comment author: Eliezer_Yudkowsky 04 April 2009 11:32:32AM 9 points [-]

Expected performance is what rational agents are actually maximising.

Does that mean that I should mechanically overwrite my beliefs about the chance of a lottery ticket winning, in order to maximize my expectation of the payout? As Nesov says, rationality is about utility; which is why a rational agent in fact maximizes their expectation of utility, while trying to maximize utility (not their expectation of utility!).

It may help to understand this and some of the conversations below if you realize that the word "try" behaves a lot like "quotation marks" and that having an extra "pair" of quotation "marks" can really make "your" sentences seem a bit odd.

Comment author: bentarm 04 April 2009 11:41:17PM *  3 points [-]

I'm not sure I get this at all.

I offer you a bet, I'll toss a coin, and give you £100 if it comes up heads, you give me £50 if it comes up tails. Presumably you take the bet right? Because your expected return is £50 - surely this is the sense in which rationalists maximise expected utility. We don't mean "the amount of utility they expect to win", but expectation in the technical sense - ie, the product of the likelihood of various events happening with their utility in the univserses in which those events happen (or probably more properly an integral...)

If you expect to lose £50 and you are wrong, that doesn't actually say anything about the expectation of your winnings.

Comment author: Nebu 14 April 2009 03:57:56PM 2 points [-]

If you expect to lose £50 and you are wrong, that doesn't actually say anything about the expectation of your winnings.

It does, however, say something about your expectation of your winnings. Expectation can be very knowledge dependent. Let's say someone rolls two six sided dice, and then offers you a bet where you win $100 if the sum of the dice is less than 5, but lose $10 if the sum is greater than 5. You might perform various calculations to determine your expected value of accepting the bet. But if I happen to peak and see one of the dice has landed on 6, then I will calculate a different expected value than you will.

So we have different expected values for calculating the bet, because we have different information.

So EY's point is that if a rational agent's only purpose was to maximize (their) expected utility, they could easily do this by selectively ignoring information, so that their calculations turn out a specific way.

But actually rational agents are not interested in maximizing (their) expected utility. They are interested in maximizing real utility. Except it's impossible to do this without perfect information, and so what agents end up doing is maximizing expected utility, although they are trying to maximize real utility.

It's like if I'm taking a history exam in school. I am trying to achieve 100% on the exam, but end up instead achieving only 60% because I have imperfect information. My goal wasn't 60%, it was 100%. But the actual actions I took (the answers I selected) led to to arrive at 60% instead of my true goal.

Rational agents are trying to maximize real utility, but end up maximizing expected utility (by definition), even though that's not their true goal.

Comment author: timtyler 04 April 2009 12:23:24PM *  2 points [-]

Re: Does that mean that I should mechanically overwrite my beliefs about the chance of a lottery ticket winning, in order to maximize my expectation of the payout?

No, it doesn't. It means that the process going on in the brains of intelligent agents can be well modelled as calculating expected utilities - and then selecting the action that corresponds to the largest one.

Intelligent agents are better modelled as Expected Utility Maximisers than Utility Maximisers. Whether they actually maximise utility depends on whether they are in an environment where their expectations pan out.

Comment author: loqi 05 April 2009 12:11:25AM 3 points [-]

Intelligent agents are better modelled as Expected Utility Maximisers than Utility Maximisers.

By definition, intelligent agents want to maximize total utility. In the absence of perfect knowledge, they act on expected utility calculations. Is this not a meaningful distinction?

Comment author: MichaelBishop 03 April 2009 07:28:57PM 2 points [-]

I am inclined to argue along exactly the same lines as Tim, though I worry there is something I am missing.

Comment author: timtyler 03 April 2009 05:35:30PM 0 points [-]

Of course, this isn't the first time I have pointed this out - see:

http://lesswrong.com/lw/33/comments_for_rationality/

Nobody seemed to have any coherent criticism the last time around - and yet now we have the same issue all over again.

Comment author: marc 03 April 2009 05:08:29PM 4 points [-]

What about cases where any rational course of action still leaves you on the losing side?

Although this may seem to be impossible according to your definition of rationality, I believe it's possible to construct such a scenario because of the fundamental limitations of a human brains ability to simulate.

In previous posts you've said that, at worst, the rationalist can simply simulate the 'irrational' behaviour that is currently the winning strategy. I would contend that humans can't simulate effectively enough for this to be an option. After all we know that several biases stem from our inability to effectively simulate our own future emotions, so to effectively simulate an entire other beings response to a complex situation would seem to be a task beyond the current human brain.

As a concrete example I might suggest the ability to lie. I believe it's fairly well established that humans are not hugely effective liars and therefore the most effective way to lie is to truly believe the lie. Does this not strongly suggest that limitations of simulation mean that a rational course of action can still be beaten by an irrational one?

I'm not sure that even if this is true it should effect a universal definition of rationality - but it would place bounds on the effectiveness of rationality in beings of limited simulation capacity.

Comment author: James_Miller 03 April 2009 04:29:14PM *  4 points [-]

If humans are imperfect actors then in situations (such as a game of chicken) in which it is better to (1) be irrational and seen as irrational then it is to (2) be rational and seen as rational

then the rational actor will lose.

Of course holding constant everyone else's beliefs about you, you always gain by being more rational.

Comment author: Eliezer_Yudkowsky 03 April 2009 06:00:23PM 3 points [-]

Given that I one-box on Newcomb's Problem and keep my word as Parfit's Hitchhiker, it would seem that the rational course of action is to not steer your car even if it crashes (if for some reason winning that game of chicken is the most important thing in the universe).

Comment author: Jonathan_Graehl 03 April 2009 09:24:31PM 3 points [-]

For Newcomb's Problem, is it fair to say that if you believe the given information, the crux is whether you believe it's possible (for Omega) to have a 99%+ correct prediction of your decision based on the givens? Refusal to accept that seems to me the only justification for two-boxing. Perhaps that's a sign that I'm less tied to a fixed set of "rationalist" procedures than a perfect rationalist would be, but I would feel like I were pretending to say otherwise.

I also wonder if the many public affirmations I've heard of "I would one-box Newcomb's Problem" are attempts at convincing Omega to believe us in the unlikely event of actually encountering the Problem. It does give a similar sort of thrill to "God will rapture me to heaven."

Comment author: James_Miller 03 April 2009 07:12:53PM 4 points [-]

You are playing chicken with your irrational twin. Both of you would rather survive than win. Your twin, however, doesn't understand that it's possible to die when playing chicken. In the game your twin both survives and wins whereas you survive but lose.

Comment author: Aurini 03 April 2009 08:06:39PM *  1 point [-]

Then you murder the twin prior to the game of chicken, and fake his suicide. Or you intimidate the twin, using your advanced rational skills to determine how exactly to best fill them with fear and doubt.

But before murdering or risking an uncertain intimidation feint, there's another question you need to ask yourself. How certain are you that the twin is irrational? The Cold War was (probably) a perceptual error; neither side realized that they were in a prisoners dilemma, they both assumed that the other side preferred "unbalanced armament" over "mutual armament" over "mutual disarmament;" in reality, the last two should have been switched.

Worst case scenario? You die playing chicken, because the stakes were worth it. The Rational path isn't always nice.

(There are some ethical premises implicit in this argument, premises which I plan to argue are natural derivatives from Game Theory... but I'm still working on that article.)

Comment author: rwallace 03 April 2009 07:21:48PM 0 points [-]

My answer to that one is that I don't play chicken in the first place unless the stake is something I'm prepared to die for.

Comment author: James_Miller 03 April 2009 07:27:07PM 5 points [-]

There are lots of chicken like games that don't involve death. For example, your boss wants some task done and either you or a co-worker can do it. The worst outcome for both you and the co-worker is for the task to not get done. The best is for the other person to do the task.

Comment author: rwallace 03 April 2009 07:30:37PM 2 points [-]

My answer still applies - I'm not going to make a song and dance about who does it, unless the other guy has been systematically not pulling his weight and it's got to the point where that matters more to me than this task getting done.

Comment author: rwallace 03 April 2009 06:39:43PM 1 point [-]

+1 for "Rationalists win". What is Parfit's Hitchhiker? I couldn't find an answer on Google.

Comment author: grobstein 03 April 2009 07:05:48PM 3 points [-]

It's a test case for rationality as pure self-interest (really it's like an altruistic version of the game of Chicken).

Suppose I'm purely selfish and stranded on a road at night. A motorist pulls over and offers to take me home for $100, which is a good deal for me. I only have money at home. I will be able to get home then IFF I can promise to pay $100 when I get home.

But when I get home, the marginal benefit to paying $100 is zero (under assumption of pure selfishness). Therefore if I behave rationally at the margin when I get home, I cannot keep my promise.

I am better off overall if I can commit in advance to keeping my promise. In other words, I am better off overall if I have a disposition which sometimes causes me to behave irrationally at the margin. Under the self-interest notion of rationality, then, it is rational, at the margin of choosing your disposition, to choose a disposition which is not rational under the self-interest notion of rationality. (This is what Parfit describes as an "indirectly self-defeating" result; note that being indirectly self-defeating is not a knockdown argument against a position.)

Comment author: ciphergoth 03 April 2009 08:02:46PM *  1 point [-]

Thank you, I too was curious.

We need names for these positions; I'd use hyper-rationalist but I think that's slightly different. Perhaps a consequentialist does whatever has the maximum expected utility at any given moment, and a meta-consequentialist is a machine built by a consequentialist which is expected to achieve the maximum overall utility at least in part through being trustworthy to keep commitments a pure consequentialist would not be able to keep.

I guess I'm not sure why people are so interested in this class of problems. If you substitute Clippy for my lift, and up the stakes to a billion lives lost later in return for two billion saved now, there you have a problem, but when it's human beings on a human scale there are good ordinary consequentialist reasons to honour such bargains, and those reasons are enough for the driver to trust my commitment. Does anyone really anticipate a version of this situation arising in which only a meta-consequentialist wins, and if so can you describe it?

Comment author: grobstein 03 April 2009 08:07:48PM 1 point [-]

I do think these problems are mostly useful for purposes of understanding and (moreso) defining rationality ("rationality"), which is perhaps a somewhat dubious use. But look how much time we're spending on it.

Comment author: grobstein 03 April 2009 08:05:42PM *  2 points [-]

I very much recommend Reasons and Persons, by the way. A friend stole my copy and I miss it all the time.

Comment author: ciphergoth 04 April 2009 08:38:38AM 3 points [-]

OK, thanks!

Your friend stole a book on moral philosophy? That's pretty special!

Comment author: MichaelHoward 05 April 2009 02:18:47PM 3 points [-]
Comment author: gjm 03 April 2009 11:35:44PM 1 point [-]

It's still in print and readily available. If you really miss it all the time, why haven't you bought another copy?

Comment author: grobstein 03 April 2009 11:37:27PM 0 points [-]

It's $45 from Amazon. At that price, I'm going to scheme to steal it back first.

OR MAYBE IT'S BECAUSE I'M CRAAAZY AND DON'T ACT FOR REASONS!

Comment author: gjm 04 April 2009 12:35:49AM 2 points [-]

Gosh. It's only £17 in the UK.

(I wasn't meaning to suggest that you're crazy, but I did wonder about ... hmm, not sure whether there's a standard name for it. Being less prepared to spend X to get Y on account of having done so before and then lost Y. A sort of converse to the endowment effect.)

Comment author: Nick_Tarleton 04 April 2009 06:51:48AM 2 points [-]

Mental accounting has that effect in the short run, but seems unlikely to apply here.

Comment author: rwallace 03 April 2009 07:19:33PM 2 points [-]

Ah, thanks. I'm of the school of thought that says it is rational both to promise to pay the $100, and to have a policy of keeping promises.

Comment author: GuySrinivasan 03 April 2009 08:22:58PM 1 point [-]

I think it is both right and expected-utility-maximizing to promise pay the $100, right to pay the $100, and not expected-utility-maximizing to pay the $100 under standard assumptions of you'll never see the driver again or whatnot.

Comment author: thomblake 03 April 2009 08:31:48PM 1 point [-]

You're assuming it does no damage to oneself to break one's own promises. Virtue theorists would disagree.

Breaking one's promises damages one's integrity - whether you consider that a trait of character or merely a valuable fact about yourself, you will lose something by breaking your promise even if you never see the fellow again.

Comment author: grobstein 03 April 2009 08:39:51PM 1 point [-]

Your argument is equivalent to, "But what if your utility function rates keeping promises higher than a million orgasms, what then?"

The hypo is meant to be a very simple model, because simple models are useful. It includes two goods: getting home, and having $100. Any other speculative values that a real person might or might not have are distractions.

Comment author: rwallace 03 April 2009 11:44:51PM 2 points [-]

Simple models are fine as long as we don't forget they are only approximations. Rationalists should win in the real world.

Comment author: thomblake 03 April 2009 08:43:00PM 2 points [-]

Except that you mention both persons and promises in the hypothetical example, so both things factor into the correct decision. If you said that it's not a person making the decision, or that there's no promising involved, then you could discount integrity.

Comment author: grobstein 03 April 2009 07:29:55PM 1 point [-]

Yes, this seems unimpeachable. The missing piece is, rational at what margin? Once you are home, it is not rational at the margin to pay the $100 you promised.

Comment author: randallsquared 03 April 2009 08:08:43PM 2 points [-]

This assumes no one can ever find out you didn't pay, as well. In general, though, it seems better to assume everything will eventually be found out by everyone. This seems like enough, by itself, to keep promises and avoid most lies.

Comment author: grobstein 03 April 2009 08:09:55PM 1 point [-]

Right. The question of course is, "better" for what purpose? Which model is better depends on what you're trying to figure out.

Comment author: grobstein 03 April 2009 06:14:43PM *  1 point [-]

Why don't you accept his distinction between acting rationally at a given moment and having the disposition which it is rational to have, integrated over all time?

EDIT: er, Parfit's, that is.

Comment author: abigailgem 03 April 2009 05:33:32PM 1 point [-]

James, when you say, "be rational", I think this shows a misunderstanding.

It may be really important to impress people with a certain kind of reckless courage. Then it is Rational to play chicken as bravely as you can. This Wins in the sense of being better than the alternative open to you.

Normally, I do not want to take the risk of being knocked down by a car. Only in this case is it not rational to play chicken: because not playing achieves what I want.

I do not see why a rationalist should be less courageous, less able to estimate distances and speeds, and so less likely to win at Chicken.

Comment author: grobstein 03 April 2009 06:33:21PM 3 points [-]

No. The point is that you actually want to survive more than you want to win, so if you are rational about Chicken you will sometimes lose (consult your model for details). Given your preferences, there will always be some distance \epsilon before the cliff where it is rational for you to give up.

Therefore, under these assumptions, the strategy "win or die trying" seemingly requires you to be irrational. However, if you can credibly commit to this strategy -- be the kind of person who will win or die trying -- you will beat a rational player every time.

This is a case where it is rational to have an irrational disposition, a disposition other than doing what is rational at every margin.

Comment author: Annoyance 03 April 2009 06:40:09PM -1 points [-]

But a person who truly cares more about winning than surviving can be utterly rational in choosing that strategy.

Comment author: Technologos 03 April 2009 07:04:16PM 1 point [-]

Agreed. In fact, the classic game-theoretic model of chicken requires that the players vastly prefer losing their pride to losing their lives. If winning/losing > losing/dying, then in a situation with imperfect information, we would assign a positive probability to playing aggressively.

And technically speaking, it is most rational, in the game-theoretic sense, to disable your steering ostentatiously before the other player does so as well. In that case, you've won the game before it begins, and there is no actual risk.

Comment author: James_Miller 03 April 2009 07:17:07PM 1 point [-]

No, if you are rational the best action is to convince your opponent that you have disabled your steering when in fact you have not done so.

Comment author: Technologos 04 April 2009 06:33:30PM 1 point [-]

Either a) your opponent truly does believe that you've disabled your steering, in which case the outcomes are identical and the actions are equally rational, or b) we account for the (small?) chance that your opponent can determine that you actually have not disabled your steering, in which case he ostentatiously disables his and wins. Only by setting up what is in effect a doomsday device can you ensure that he will not be tempted to information-gathering brinksmanship.

Comment author: grobstein 03 April 2009 04:43:11PM 3 points [-]

This is a classic point and clearer than the related argument I'm making above. In addition to being part of the accumulated game theory learning, it's one of the types of arguments that shows up frequently in Derek Parfit's discussion of what-is-rationality, in Ch. 1 of Reasons and Persons.

I feel like there are difficulties here that EY is not attempting to tackle.

Comment author: Vladimir_Nesov 03 April 2009 04:26:19PM *  25 points [-]

Rationality is about winning.

The about captures the expected systematic winning part, as you are considering the model of winning, not necessarily the accidental winning itself. It limits the scope to the winning only, leaving only the secondary roles for parry, hit, spring, strike or touch. Being a study about the real thing, rationality employs a set of tricks that allow to work it, in special cases and at coarse levels of detail. Being about the real thing, rationality aims to give the means for actually winning.

Comment author: Eliezer_Yudkowsky 03 April 2009 04:16:12PM 5 points [-]

I guess when I look over the comments, the problem with the phraseology is that people seem to inevitably begin debating over whether rationalists win and asking how much they win - the properties of a fixed sort of creature, the "rationalist" - rather than saying, "What wins systematically? Let us define rationality accordingly."

Not sure what sort of catchphrase would solve this.

Comment author: astray 03 April 2009 04:29:15PM 1 point [-]

It runs into problems elsewhere, but what about "Rationalism should win" ?

Comment author: Eliezer_Yudkowsky 03 April 2009 04:34:29PM 2 points [-]

Well, that's wrong, but thinking about why it's wrong leads me to realize that maybe "Rationality should win" would have been a better move.

But I did also want to convey the idea that aspiring to be a rationalist means aspiring to be stronger, something more formidable than a debating style... well, I guess "rationality should win" conveys a bit of that too.

Comment author: AlexU 03 April 2009 04:21:28PM *  3 points [-]

Yes. Rationalism shouldn't be see as a bag of discrete tricks, but rather, as the means for achieving any given end -- what it takes to do something you want to do. The particulars will vary, of course, depending on the end in question, but the rational individual should do better at figuring them out.

On a side note, I'm not sure coming up with better slogans, catchphrases, and neologisms is the right thing to be aiming for.

Comment author: gwern 03 April 2009 07:33:10PM 1 point [-]

'Whatever wins is rational'?

'Winners are rational'?

'Rationality is winning'?

Hm. Sloganeering is harder than it looks.

Comment author: Eliezer_Yudkowsky 03 April 2009 04:34:52PM 4 points [-]

Do not underestimate the power of poetry.

Comment author: Annoyance 03 April 2009 05:04:31PM -1 points [-]

Yes, but power to do what? Sometimes poetry conveys ideas and associations, sometimes it just annoys people.

Comment author: AlexU 03 April 2009 04:08:42PM *  2 points [-]

All else being equal, shouldn't rationalists, almost by definition, win? The only way this wouldn't happen would be in a contest of pure chance, in which rationality could confer no advantage. It seems like we're just talking semantics here.

Comment author: anonym 04 April 2009 10:58:45PM 3 points [-]

If human beings had perfect control over their minds and bodies -- e.g., could tweak System 1 without limit and perform any physically possible act/behavior -- your point would be stronger.

However, as others have mentioned elsewhere, there may be cases where we are just not capable of implementing a strategy that rationality suggests is optimal (e.g., convincingly pretending to be more confident than you are to the point that all relevant System 1 impulses/reactions are those of a person who is naturally overconfident).

It may be the case that an ubermensch rationalist can eventually learn to do anything that can be done via non-rational means, but that's not clear a priori, especially if we consider finite lifespans and opportunity costs.

Comment author: AlexU 04 April 2009 11:04:55PM 3 points [-]

Agreed. Particularly in hypothetical cases where one rationally concludes that it would be in their best interest to behave irrationally, e.g., over-confidence in oneself or belief in God. Even if one arrived at those conclusions, it's not clear to me how anyone could decide to become irrational in those ways. Pascal's notion of "bootstrapping" oneself into religious belief never struck me as very plausible. Interestingly though, "faking" confidence in oneself often does tend to lead to real confidence via some sort of feedback mechanism, e.g., interactions with women.

Comment author: billswift 03 April 2009 03:33:55PM 1 point [-]

"abandon reasonableness" is never necessary; though I think we may be using reasonable somewhat differently. I think "reasonable" includes the idea of "appropriate to the situation"

quoting myself : "There is a supposed "old Chinese saying": The wise man defends himself by never being attacked. Which is excellent, if incomplete, advice. I completed it myself with "But only an idiot counts on not being attacked." Don't use violence unless you really need to, but if you need to don't hold back." http://williambswift.blogspot.com/2009/03/violence.html

As to your overall point, I agree that rationalists should win. General randomness, unknowns, and opposition from other agents prevent consistent victories in the real world. But if you are not winning more than losing you definitely are not being rational.

Comment author: Eliezer_Yudkowsky 04 April 2009 12:06:51PM 0 points [-]

Standard proverb: "If you would have peace, prepare for war."

Comment author: SoullessAutomaton 03 April 2009 03:43:23PM 4 points [-]

Don't use violence unless you really need to, but if you need to don't hold back.

By corollary:

"Rule #6: If violence wasn't your last resort, you failed to resort to enough of it." -- The Seven Habits of Highly Effective Pirates

Comment author: abigailgem 03 April 2009 03:32:33PM *  8 points [-]

Suggestion: "Rationalists seek to Win, not to be rational".

Suggestion: "If what you think is rational appears less likely to Win than what you think is irrational, then you need to reassess probabilities and your understanding of what is rational and what is irrational".

Suggestion: "It is not rational to do anything other than the thing which has the best chance of winning".

If I have a choice between what I define as the "Rational" course of action, and a course of action which I describe as "irrational" but which I predict has a better chance of winning, I am either predicting badly or wrongly defining what is Rational.

I am not sure my suggestions are Better, but I am groping towards understanding and hope my gropings help.

EDIT: and the warning is that we may deceive ourselves into thinking that we are being rational, when we are missing something, using the wrong map, arguing fallaciously. So what about:

Suggestion: "If you are not Winning, consider whether you are really being rational".

"If you are not Winning more than people you believe to be irrational, this may be evidence that you are not really being rational".

On a different tack, "Rationalists win wherever rationality is an aid to winning". I am not going to win millions on the Lottery, because I do not play it.