# Rationality is Systematized Winning

48 03 April 2009 02:41PM

Followup toNewcomb's Problem and Regret of Rationality

"Rationalists should win," I said, and I may have to stop saying it, for it seems to convey something other than what I meant by it.

Where did the phrase come from originally?  From considering such cases as Newcomb's Problem:  The superbeing Omega sets forth before you two boxes, a transparent box A containing $1000 (or the equivalent in material wealth), and an opaque box B that contains either$1,000,000 or nothing.  Omega tells you that It has already put $1M in box B if and only if It predicts that you will take only box B, leaving box A behind. Omega has played this game many times before, and has been right 99 times out of 100. Do you take both boxes, or only box B? A common position - in fact, the mainstream/dominant position in modern philosophy and decision theory - is that the only reasonable course is to take both boxes; Omega has already made Its decision and gone, and so your action cannot affect the contents of the box in any way (they argue). Now, it so happens that certain types of unreasonable individuals are rewarded by Omega - who moves even before they make their decisions - but this in no way changes the conclusion that the only reasonable course is to take both boxes, since taking both boxes makes you$1000 richer regardless of the unchanging and unchangeable contents of box B.

And this is the sort of thinking that I intended to reject by saying, "Rationalists should win!"

Said Miyamoto Musashi:  "The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means.  Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement.  It is essential to attain this.  If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him."

Said I:  "If you fail to achieve a correct answer, it is futile to protest that you acted with propriety."

This is the distinction I had hoped to convey by saying, "Rationalists should win!"

There is a meme which says that a certain ritual of cognition is the paragon of reasonableness and so defines what the reasonable people do.  But alas, the reasonable people often get their butts handed to them by the unreasonable ones, because the universe isn't always reasonable.  Reason is just a way of doing things, not necessarily the most formidable; it is how professors talk to each other in debate halls, which sometimes works, and sometimes doesn't.  If a hoard of barbarians attacks the debate hall, the truly prudent and flexible agent will abandon reasonableness.

No.  If the "irrational" agent is outcompeting you on a systematic and predictable basis, then it is time to reconsider what you think is "rational".

For I do fear that a "rationalist" will clutch to themselves the ritual of cognition they have been taught, as loss after loss piles up, consoling themselves:  "I have behaved virtuously, I have been so reasonable, it's just this awful unfair universe that doesn't give me what I deserve.  The others are cheating by not doing it the rational way, that's how they got ahead of me."

It is this that I intended to guard against by saying:  "Rationalists should win!"  Not whine, win.  If you keep on losing, perhaps you are doing something wrong.  Do not console yourself about how you were so wonderfully rational in the course of losing.  That is not how things are supposed to go.  It is not the Art that fails, but you who fails to grasp the Art.

Likewise in the realm of epistemic rationality, if you find yourself thinking that the reasonable belief is X (because a majority of modern humans seem to believe X, or something that sounds similarly appealing) and yet the world itself is obviously Y.

But people do seem to be taking this in some other sense than I meant it - as though any person who declared themselves a rationalist would in that moment be invested with an invincible spirit that enabled them to obtain all things without effort and without overcoming disadvantages, or something, I don't know.

Maybe there is an alternative phrase to be found again in Musashi, who said:  "The Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size."

"Rationality is the spirit of winning"?  "Rationality is the Way of winning"?  "Rationality is systematized winning"?  If you have a better suggestion, post it in the comments.

Sort By: Controversial
Comment author: 03 April 2009 09:09:13PM -1 points [-]

Please ... Newcomb is a toy non-mathematizable problem and not a valid argument for anything at all. There must be a better example, or the entire problem is invalid.

Comment author: 03 April 2009 10:09:32PM 6 points [-]

There must be a better example, or the entire problem is invalid.

I've long thought that voting in general is largely isomorphic to Newcomb's. If you cop out and don't vote, then everyone like you will reason the same way and not vote, and your favored candidates/policies will fail; but if you vote then the reverse might happen; and if you then carry it one more step... If you could just decide to one-box/vote then maybe everyone else like you will.

Comment author: 04 April 2009 01:11:14AM *  1 point [-]

Sorry, in voting you don't play the singular boss role that you play in Newcomb's problem. But it's amusing how far democracy proponents will go to convince themselves that their vote matters. :-)

Comment author: 04 April 2009 01:34:09AM 3 points [-]

I haven't worked it out rigorously (else you would've seen a post on it by now!), but it seems to me in close elections (Florida 2000, say) that thought process could be valid. Considering how small the margins sometimes are, and how much of the electorate doesn't vote, it doesn't strike me as implausible that there are enough people thinking like me to make a difference.

And of course we could just specify as a condition that you and yours are a bloc powerful enough to affect the election. (Maybe you're numerous, maybe there're only a few electors, whatever.)

But it's amusing how far democracy proponents will go to convince themselves that their vote matters.

The problem with irrelevant ad hominems is that they're very often based on flimsy evidence and so often wrong. I didn't even vote last year because I figured my vote didn't matter. I was not surprised.

Comment author: 04 April 2009 10:56:07AM *  2 points [-]

In Newcomb's problem you're the boss, e.g. you can assign yourself a suitable utility function beforehand to keep the million and screw the thousand. Not so in voting - no matter what you think, other people won't change. They don't have anything conditioned on the outcome of your thought process, as in Newcomb's. No, not even if "people thinking like you" are a bloc. You still can't influence them. It's a coordination game, not Newcomb's.

Your reasoning resembles the "twins fallacy" in the Prisoner's Dilemma: the idea that just by choosing to cooperate you can magically force your identical partner to do the same. Come to think of it, PD sounds like a better model for voting to me.

Update: Eliezer seems to think PD and Newcomb's are related. Not sure why.

Comment author: 03 April 2009 09:10:59PM 2 points [-]

Please ... Newcomb is a toy non-mathematizable problem and not a valid argument for anything at all.

Why?

Comment author: 03 April 2009 10:32:42PM 5 points [-]

As far as I can tell, Newcomb problem exists only in English, and only because a completely aphysical causality loop is introduced. Every mathematization I've ever seen collapses it to either trivial one-boxing problem, or trivial two-boxing problem.

If anybody wants this problem to be treated seriously, maths first to show the problem is real! Otherwise, we're really not much better than if we were discussing quotes from the Bible.

Comment author: 05 April 2009 11:27:44PM 5 points [-]

If you've seen formalizations, then it is formalizable. What are the formalizations?

Since I think the answer is obviously one-box, it doesn't surprise me that there is a formalization in which that answer is obvious. I have never seen a formalization in which the answer is two-box. I have seen the argument that "causal decision theory" (?) chooses to two-box. People jump from that to the conclusion that the answer is two-box, but that is an idiotic conclusion. Given the premise, the correct conclusion is that this decision theory is inadequate. Anyhow, I don't believe the argument. I interpret it simply as the decision theory failing to believe the statement of the problem. There is a disconnect between the words and the formalization of that decision theory.

The issue is not about formalizing Newcomb's problem; the problem is creating a formal decision theory that can understand a class of scenarios including Newcomb's problem. (It should be possible to tweak the usual decision theory to make it capable of believing Newcomb's problem, but I don't think that would be adequate for a larger class of problems.)

Comment author: 03 April 2009 09:41:12PM 0 points [-]

I already understood what you meant by "rationalists should win", Eliezer, but I don't find Newcomb's problem very convincing as an example. The way I see it, if you one-box you've lost. You could have gotten an extra $1000 but you chose not to. Comment author: 03 April 2009 09:58:21PM * 2 points [-] And yet those who one-box get$999000 more than those who don't. What gives? If there is a systematic, predictable thing that offers one-boxers $1000000 and offers two-boxers$1000, and there is not a systematic, predictable thing that provides some sort of countering offer to two-boxers, by one-boxing you still get more money.

I can't think of something of equal power level (examining your decisions, not your method of arriving at those decisions) which would be able to provide the countering offer to two-boxers.

Comment author: 03 April 2009 10:16:12PM 1 point [-]

Simple: most situations in real life aren't like this. If you believe Omega and one-box, you'll lose when he's lying. If your decision theory works better in hypothetical situations and worse in real life, then it doesn't make you win.

Comment author: 04 April 2009 01:33:52AM *  8 points [-]

Also, I don't think Eliezer keeps harping on Newcomb's problem because he anticipates experiencing precisely that scenario. I see several important points that I don't think have been clearly made (not that I'm the one to do so):

1. We can choose whether and when to implement certain decision algorithms, including classical causal decision theory (CCDT). This choice may in fact be trivial, or it may be subtle, but it is a worthy question for a rationalist.

2. Although, for any fixed set of options, implementing CCDT maximizes your return, there are in fact cases where the options you have depend on the outcome of a model of your decision algorithm. I'm not talking about Omega, I'm talking about human social life. We base a large portion of our interactions with others on our anticipations of how they might respond. (This isn't often done rationally by anyone's standards, but it can be.)

3. It gets confusing (in particular, Hofstadterian) here, but a plausibly better outcome might be reached in the Prisoner's Dilemma by selfish non-strangers mutually modeling the other's likely decision process, and recognizing that only C-C and D-D are stable outcomes under mutual modeling.

Of course, I still feel a bit uncomfortable with this line of reasoning.

Comment author: 03 April 2009 10:11:26PM *  2 points [-]

Of course the one-boxers get more money: They were put in a situation in which they could either get $1 000 000 or$1 001 000, whereas the two-boxers were put in a situation in which they could get $0 or$1000.

It makes no sense to compare the two decisions the way you and Eliezer do. It's like organizing a swimming competition between an Olympic athlete who has to swim ten kilometers to win and an untrained fatass who only has to swim a hundred meters to win, and concluding that because the fatass wins more often than the athlete, therefore fatasses clearly make better swimmers than athletes.

Comment author: 03 April 2009 10:33:33PM 1 point [-]

But if you were put into said hypothetical competition, and could somehow decide just before the contest began whether to be an Olympic athlete or an untrained fatass, which would you choose?

I think you're getting overly distracted by the details of the problem construction and missing the point.

Comment author: 03 April 2009 10:36:55PM -1 points [-]

You're assuming that you can just choose how you go about making decisions every time you make a decision. If you're not granted that assumption, Furcas's analysis is spot on. Two-boxers succeed in other places and also on Newcomb; one-boxers fail in many situations that are similar to Newcomb but not as nice. So you need to decide what sort of decisions you'll make in general, and that will (arguably) determine how much money is in the boxes in this particular experiment.

Comment author: 04 April 2009 07:50:09AM 2 points [-]

one-boxers fail in many situations that are similar to Newcomb but not as nice.

Such as?

(Is this meant to refer to failures of evidential decision theory? There are other options.)

Comment author: 03 April 2009 10:44:11PM 1 point [-]

If my only goal were to win that particular competition (and not to be a good swimmer), of course I'd choose to turn into a fatass and lose all my training. Likewise, if I could choose to precommit to one-boxing in Newcomb-like problems, I would, because pre-commitment has an effect on what will be in box B (whereas the actual decision does not).

The details are what makes Newcomb's problem what it is, so I don't see how it's possible to get "overly distracted" by them. Correct me if I'm wrong, but pre-commitment isn't an option in Newcomb's problem, so the best, the most rational, the winning decision is two-boxing.

Comment author: 03 April 2009 11:11:02PM *  4 points [-]

Correct me if I'm wrong, but pre-commitment isn't an option in Newcomb's problem, so the best, the most rational, the winning decision is two-boxing.

By construction, Omega's predictions are known to be essentially infallible. Given that, whatever you choose, you can safely assume Omega will have correctly predicted that choice. To what extent, then, is pre-commitment distinguishable from deciding on the spot?

In a sense there is an implicit pre-commitment in the structure of the problem; while you have not pre-committed to a choice on this specific problem, you are essentially pre-committed to a decision-making algorithm.

Eliezer's argument, if I understand it, is that any decision-making algorithm that results in two-boxing is by definition irrational due to giving a predictably bad outcome.

Comment author: 03 April 2009 11:26:07PM -1 points [-]

Eliezer's argument, if I understand it, is that any decision-making algorithm that results in two-boxing is by definition irrational due to giving a predictably bad outcome.

So he's assuming the conclusion that you get a bad outcome? Golly.

Comment author: 03 April 2009 11:31:02PM 1 point [-]

The result of two-boxing is a thousand dollars. The result of one-boxing is a million dollars. By definition, a mind that always one-boxes receives a better payout than one that always two-boxes, and therefore one-boxing is more rational, by definition.

Comment author: 03 April 2009 11:41:32PM *  1 point [-]

The result of two-boxing is a thousand dollars more than you would have gotten otherwise. The result of one-boxing is a thousand dollars less than you would have gotten otherwise. Therefore two-boxing is more rational, by definition.

What determines whether you'll be in a 1M/1M+1K situation or in a 0/1K situation is the kind of mind you have, but in Newcomb's problem you're not given the opportunity to affect what kind of mind you have (by pre-commiting to one-boxing, for example), you can only decide whether to get X or X+1K, regardless of X's value.

Comment author: 04 April 2009 12:48:42AM 2 points [-]

Suppose for a moment that one-boxing is the Foo thing to do. Two-boxing is the expected-utility-maximizing thing to do. Omega decided to try to reward those minds which it predicts will choose to do the Foo thing with a decision between doing the Foo thing and gaining $1000000, and doing the unFoo thing and gaining$1001000, while giving those minds which will choose to do the unFoo thing a decision between doing the Foo thing and gaining $0 and doing the unFoo thing and gaining$1000.

The relevant question is whether there is a generalization of the computation Foo which we can implement that doesn't screw us over on all sorts of non-Newcomb problems. Drescher for instance claims that acting ethically implies, among other things, doing the Foo thing, even when it is obviously not the expected-utility-maximizing thing.

Comment author: 04 April 2009 01:36:58AM 1 point [-]

See Arguing "By Definition". It's particularly problematic when the definition of "rational" is precisely what's in dispute.

Comment author: 03 April 2009 11:35:45PM *  3 points [-]

True, we don't know the outcome. But we should still predict that it will be bad, due to Omega's 99% accuracy rate.

Don't mess with Omega.

Comment author: 04 April 2009 12:59:58AM 1 point [-]

In a sense there is an implicit pre-commitment in the structure of the problem; while you have not pre-committed to a choice on this specific problem, you are essentially pre-committed to a decision-making algorithm.

That's an interesting, and possibly fruitful, way of looking at the problem.

Comment author: 03 April 2009 11:30:41PM *  1 point [-]

Pre-commitment is different from deciding on the spot because once you're on the spot, there is nothing, absolutely nothing you can do to change what's in box B. It's over. It's a done deal. It's beyond your control.

My understanding of Eliezer's argument is the same as yours. My objection is that two-boxing doesn't actually give a bad outcome. It gives the best outcome you can get given the situation you're in. That you don't know what situation you're in until after you've opened box B doesn't change that fact. As Eliezer is so fond of saying, the map isn't the territory.

Comment author: 04 April 2009 12:59:58AM 1 point [-]

Pre-commitment is different from deciding on the spot because once you're on the spot, there is nothing, absolutely nothing you can do to change what's in box B.

If your decision on the spot is 100 percent predictable ahead of time, as is explicitly assumed in the problem construction, you are effectively pre-committed as far as Omega is concerned. You, apparently, have essentially pre-committed to opening two boxes.

My objection is that two-boxing doesn't actually give a bad outcome. It gives the best outcome you can get given the situation you're in.

And yet, everyone who opens one box does better than the people who open two boxes.

You seem to have a very peculiar definition of "best outcome".

Comment author: 04 April 2009 02:03:06AM *  0 points [-]

If your decision on the spot is 100 percent predictable ahead of time, as is explicitly assumed in the problem construction, you are effectively pre-committed as far as Omega is concerned. You, apparently, have essentially pre-committed to opening two boxes.

What I meant by 'pre-commitment' is a decision that we can make if and only if we know about Newcomb-like problems before being faced with one. In other words, it's a decision that can affect what Omega will put in box B. That Omega can deduce what my decision will be doesn't mean that the decision is already taken.

And yet, everyone who opens one box does better than the people who open two boxes.

And every fatass who competes against an Olympic athlete in the scenario I described above does 'better' than the athlete. So what? Unless the athlete knows about the competition's rules ahead of time and eats non-stop to turn himself into a fatass, there's not a damn thing he can do about it, except try his best once the competition starts.

You seem to have a very peculiar definition of "best outcome".

It seems too obvious to say, but I guess I have to say it. "The best outcome" in this context is "the best outcome that it is possible to achieve by making a decision". If box B contains nothing, then the best outcome that it is possible to achieve by making a decision is to win a thousand dollars. If box B contains a million dollars, then the best outcome that it is possible to achieve by making a decision is to win one million and one thousand dollars.

Well, I don't see how I can explain myself more clearly than this, so this will be my last comment on this subject. In this thread. This week. ;)

Comment author: 12 April 2009 04:59:59PM 2 points [-]

This exchange has finally imparted a better understanding of this problem for me.

If you pre-commit now to always one-box – and you believe that about yourself – then deciding to one-box when Omega asks you is the best decision.

If you are uncertain of your commitment then you probably haven't really pre-committed! I haven't tried to math it, but I think your actual decision when Omega arrives would depend on the strength of your belief about your own pre-commitment. [Though a more-inconvenient possible world is the one in which you've never heard of this, or similar, puzzles!]

Now I grok why rationality should be self-consistent under reflection.

Comment author: 04 April 2009 04:03:41AM 3 points [-]

Of course the one-boxers get more money: They were put in a situation in which they could either get $1 000 000 or$1 001 000, whereas the two-boxers were put in a situation in which they could get $0 or$1000.

When faced with this decision, you are either in the real world, in which case you can get an extra $1000 by two boxing, or you are in a simulation, in which case you can arrange so your self in the real world gets and extra$1,000,000 by one boxing. Given that you can't tell which of these is the case, and that you are deterministic, you will make the same decision in both situations. So your choice is to either one box and gain $1,000,000 or two box and gain$1000. If you like having more money, it seems clear which of those choices is more rational.

Comment author: 03 April 2009 10:17:09PM 1 point [-]

This premise is not accepted by the 1-box contingent. Occasionally they claim there's a reason.

Comment author: 04 April 2009 02:22:40AM 1 point [-]

Can you please elaborate? I'm trying to catch up!

Comment author: 03 April 2009 10:30:42PM 1 point [-]

You mean they don't accept that the decision doesn't affect what's in box B?

Comment author: 04 April 2009 01:49:34AM 0 points [-]

Has it been settled then, that in this Newcomb's Problem, rationality and winning are at odds? I think it is quite relevant to this discussion whether or not they ever can be at odds.

My last comment got voted down -- presumably because whether or not rationality and winning are ever in conflict has been discussed in the previous post. (I'm a quick study and would like feedback as to why I get voted down.) However, was there some kind of consensus in the previous post? Do we just assume here that it is possible that rationality is not always the winning strategy? I cannot!

Looking through the comments, it sounds like many people think it is most rational to pick both boxes because of some assumption about how physical reality can't be altered. In a hypothetical reality where that assumption doesn't hold, it would be irrational to insist on applying it.

Comment author: 18 April 2011 11:51:44PM 3 points [-]

"Rationalists should win." is what sold me on this site. Its a good phrase.

Comment author: 04 April 2009 02:36:15AM *  1 point [-]

Rationality is the art of the optimal.

Rationality is optimally systematized systematizing. (:-))

Comment author: 03 April 2009 06:11:33PM 1 point [-]

Like William, I think "winning" is the problem, though for different reasons : "winning" has extra connotations, and tends to call up the image of the guy who climbs the corporate ladder through dishonesty and betrayal rather than trying to lead a happy and fulfilling life. Or someone who tries to win all debates by humiliating his opponent till nobody wants to speak to him any more.

Winning often doesn't mean getting what you want, but winning at something defined externally, or competing with others - which may indeed not always the rational thing to do.

The thread on the Rationality Questionaire seemed to have this problem - some questions seemed more focused on "winning as understood by society" rather then getting what you want.

Comment author: 03 April 2009 03:09:53PM 1 point [-]

There are two possible interpretations of "Rationalists should win", and it's likely the confusion is coming about from the second.

One use of "should" is to indicate a general social obligation: "people should be nice to each other", and the other is to indicate a personal entitlement: "you should be nice to me." i.e., "should" = "I deserve it"

It appears that some people may be using the latter interpretation, i.e., "I'm rational so I should win" -- placing the obligation on the universe rather than on themselves.

Perhaps "Rationalists choose to win", or "Winning is better than being right"?

Comment author: 03 April 2009 03:39:57PM 2 points [-]

"Winning is better than being right"

I think Eliezer's point is closer to "Winning is the same as being right"; i.e., the evidence that you're right is that you won.

Comment author: 03 April 2009 05:24:40PM 3 points [-]

"Winning" and "being right" are different concepts. That is the point of distinguishing between epistemic and instrumental rationality.

Comment author: 03 April 2009 08:49:03PM 5 points [-]

Actually the problem is an ambiguity in "right" -- you can take the "right" course of action (instrumental rationality, or ethics), or you can have "right" belief (epistemic rationality).

Comment author: 14 April 2009 05:25:11PM 0 points [-]

One problem is that "Rationalists should win" has two obvious interpretations for me:

1. Rationalists should win, therefore if rationalists aren't winning, there's something wrong with the world.
2. Rationalists should win, therefore if you aren't winning, you're not a rationalist.

Compare with:

1. People should donate money to Africa, therefore if people aren't donating to Africa, there's something wrong with the world.
2. People should donate money to Africa, therefore if you don't donate to Africa, you're not a person.

and

1. Protons should have an electric charge of 1, therefore if protons don't have an electric charge of 1, there's something wrong with the world.
2. Protons should have an electric charge of 1, therefore if you don't have an electric charge of 1, you are not a proton.
Comment author: 14 April 2009 05:48:01PM *  2 points [-]

Shouldness in "Rationalists should win" is a much more detailed notion than correspondence to winning situations. It refers to a property of achieving goals, as seen under uncertainty, in our case implemented by cognitive algorithms that search the solution-space for the right plans. Rationalists should-win, have a good measure of win-should-ness.

Looking over it all again, I should add that "rationality is about winning" is also an immensely simpler sentiment, that still seems to retain the gist of the message for which the "rationalists should win" motto was devised.

Comment author: 14 April 2009 05:32:46PM 2 points [-]

What "Rationalists should WIN" needs is a stake through its heart; it is misinterpreted so much more often than it is correctly used that we may need to do without it altogether.

Comment author: 03 April 2009 11:42:21PM 0 points [-]

Am I missing something? I think this answer is very simple: rationality and winning are never at odds.

(The only exception is when a rational being has incomplete information. If information tells him that the blue box has $100 and the red box has$0, and it is the other way around, it is rational for him to pick the blue box even though he doesn't win.)

Comment author: 04 April 2009 12:03:12AM 1 point [-]

The only exception is when a rational being has incomplete information

Even rational beings usually don't have complete information.

Comment author: 04 April 2009 02:18:34AM *  1 point [-]

Yes, I agree. I think being rational is always being aware that everything you "know" is a house of cards based on your assumptions. A change in assumptions will require rebuilding the house, and a false room means you need to challenge an assumption.

I'm just arguing that a false room never means that rationality (deduction) itself was wrong (i.e., not winning).

All a rational being can do is base decisions on the information they have. A question: is a rational position based upon incomplete information that leads to not winning really an example of "rationality" not winning? I think that in this discussion we are talking about the relationship between rationality and winning in the context of "enough" information.

Comment author: 04 April 2009 12:44:07PM 5 points [-]

Re: "First, foremost, fundamentally, above all else: Rational agents should WIN."

In an attempt to summarise the objections, there seem to be two fairly-fundamental problems:

1. Rational agents try. They cannot necessarily win: winning is an outcome, not an action;

2. "Winning" is a poor synonym for "increasing utility": sometimes agents should minimise their losses.

"Rationalists maximise expected utility" would be a less controversial formulation.

Comment author: 05 May 2012 10:19:29AM *  0 points [-]

I much prefer "rationalists should win" because it's simple, accessible language. Makes this article more powerful than it would otherwise be. Everyone gets winning; how many people find terms like expected utility maximisation meaningful on a gut level?

Comment author: 05 May 2012 11:12:43AM -1 points [-]

Charlie Sheen rationality.

Comment author: 04 April 2009 01:13:32PM 3 points [-]

I agree with your two problems, but the problem with your alternative and so many others presented here is that it doesn't so strongly speak to the distinction which EY means to draw, between wanting to be seen to have followed the forms for maximising expected utility and actually seeking to maximise expected utility.

Also, of course, one who at each moment makes the decision that maximises expected future utility defects against Clippy in both Prisoner's Dilemma and Parfit's Hitchhiker scenarios, and arguably two-boxes against Omega, and by EY's definition that counts as "not winning" because of the negative consequences of Clippy/Omega knowing that that's what we do.

Comment author: 04 April 2009 01:42:00PM 1 point [-]

Re: "it doesn't so strongly speak to the distinction which EY means to draw"

I wasn't trying to do that. It seems like a non-trivial concept. Is it important to try and capture that distinction in a slogan?

Re: "one who at each moment makes the decision that maximises expected future utility defects"

Expected utility maximising agents don't have commitment mechanisms, and can't be trusted to make promises? I am sceptical. In my view, you can express practically any agent as an expected utility maximiser. It seems easy enough to imagine commitment mechanisms. I don't see where the problem is.

Comment author: 04 April 2009 01:06:06PM 2 points [-]

"Rationalists maximise expected utility" would be a less controversial formulation.

But, alas, less catchy.

Comment author: 04 April 2009 01:22:38PM 5 points [-]

As contagious memes go, "rationalists should win" seems to be rather pathogenic to me. A proposed rationalist slogan shouldn't need so many footnotes. For the sake of minds everywhere, I think it would be best to try to kill it off in its early stages.

Comment author: 03 April 2009 11:26:56PM *  5 points [-]

Duhhh!

(Cf.)

Comment author: 03 April 2009 04:16:12PM 5 points [-]

I guess when I look over the comments, the problem with the phraseology is that people seem to inevitably begin debating over whether rationalists win and asking how much they win - the properties of a fixed sort of creature, the "rationalist" - rather than saying, "What wins systematically? Let us define rationality accordingly."

Not sure what sort of catchphrase would solve this.

Comment author: 03 April 2009 04:29:15PM 1 point [-]

It runs into problems elsewhere, but what about "Rationalism should win" ?

Comment author: 03 April 2009 04:34:29PM 2 points [-]

Well, that's wrong, but thinking about why it's wrong leads me to realize that maybe "Rationality should win" would have been a better move.

But I did also want to convey the idea that aspiring to be a rationalist means aspiring to be stronger, something more formidable than a debating style... well, I guess "rationality should win" conveys a bit of that too.

Comment author: 03 April 2009 04:21:28PM *  3 points [-]

Yes. Rationalism shouldn't be see as a bag of discrete tricks, but rather, as the means for achieving any given end -- what it takes to do something you want to do. The particulars will vary, of course, depending on the end in question, but the rational individual should do better at figuring them out.

On a side note, I'm not sure coming up with better slogans, catchphrases, and neologisms is the right thing to be aiming for.

Comment author: 03 April 2009 07:33:10PM 1 point [-]

'Whatever wins is rational'?

'Winners are rational'?

'Rationality is winning'?

Hm. Sloganeering is harder than it looks.

Comment author: 03 April 2009 04:34:52PM 4 points [-]

Do not underestimate the power of poetry.

Comment author: 03 April 2009 05:04:31PM -1 points [-]

Yes, but power to do what? Sometimes poetry conveys ideas and associations, sometimes it just annoys people.

Comment author: 03 April 2009 02:46:50PM 5 points [-]

I don't think I buy this for Newcomb-like problems. Consider Omega who says, "There will be $1M in Box B IFF you are irrational." Rationality as winning is probably subject to a whole family of Russell's-Paradox-type problems like that. I suppose I'm not sure there's a better notion of rationality. Comment author: 03 April 2009 02:51:50PM * 6 points [-] What you give is far harder than a Newcomb-like problem. In Newcomb-like problems, Omega rewards your decisions, he isn't looking at how you reach them. This leaves you free to optimize those decisions. Comment author: 03 April 2009 03:57:10PM 2 points [-] What you give is far harder than a Newcomb-like problem. In Newcomb-like problems, Omega rewards your decisions, he isn't looking at how you reach them. You misunderstand. In my variant, Omega is also not looking at how you reach your decision. Rather, he is looking at you beforehand -- "scanning your brain", if you will -- and evaluating the kind of person you are (i.e., how you "would" behave). This, along with the choice you make, determines your later reward. In the classical problem, (unless you just assume backwards causation,) what Omega is doing is assessing the kind of person you are before you've physically indicated your choice. You're rewarded IFF you're the kind of person who would choose only box B. My variant is exactly symmetrical: he assesses whether you are the kind of person who is rational, and responds as I outlined. Comment author: 03 April 2009 04:08:37PM 4 points [-] We have such an Omega: we just refer to it differently. After all, we are used to treating our genes and our environments as definite influences on our ability to Win. Taller people tend to make more money; Omega says "there will be$1mil in box B if you have alleles for height."

If Omega makes decisions based on properties of the agent, and not on the decisions either made or predicted to be made by the agent, then Omega is no different from, well, a lot of the world.

Rationality, then, might be better redefined under these observations as "making the decisions that Win whenever such decisions actually affect one's probability of Winning," though I prefer Eliezer's more general rules plus the tacit understanding that we are only including situations where decisions make a difference.

Comment author: 03 April 2009 04:21:11PM 1 point [-]

Yes, all well and good (though I don't see how you identify any distinction between "properties of the agent" and "decisions . . . predicted to be made by the agent" or why you care about it). My point is that a concept of rationality-as-winning can't have a definite extension say across the domain of agents, because of the existence of Russell's-Paradox problems like the one I identified.

This is perfectly robust to the point that weird and seemingly arbitrary properties are rewarded by the game known as the universe. Your proposed redefinition may actually disagree with EY's theory of Newcomb's problem. After all, your decision can't empty box B, since the contents of box B are determinate by the time you make your decision.

Comment author: 03 April 2009 06:06:14PM 0 points [-]

After all, your decision can't empty box B, since the contents of box B are determinate by the time you make your decision.

Hello. My name is Omega. Until recently I went around claiming to be all-knowing/psychic/whatever, but now I understand lying is Wrong, so I'm turning over a new leaf. I'd like to offer you a game.

Here are two boxes. Box A contains $1,000, box B contains$1,000,000. Both boxes are covered by touch-sensitive layer. If you choose box B only (please signal that by touching box B), it will send out a radio signal to box A, which will promptly disintegrate. If you choose both boxes (please signal that by touching box A first), a radio signal will be sent out to box B, which will disintegrate it's content, so opening it will reveal an empty box.

(I got the disintegrating technology from the wreck of a UFO that crashed into my barn, but that's not relevant here.)

I'm afraid, if I or my gadgets detect any attempt to temper with the operation of my boxes, I will be forced to disqualify you.

In case there is doubt, this is the same game I used to offer back in my deceitful days. The difference is, now the player knows the rules are enforced by cold hard electronics, so there's no temptation to try and outsmart anybody.

So, what will it be?

Comment author: 03 April 2009 06:40:18PM 1 point [-]

Yes, you are changing the hypo. Your Omega dummy says that it is the same game as Newcomb's problem, but it's not. As VN notes, it may be equivalent to the version of Newcomb's problem that assumes time travel, but this is not the classical (or an interesting) statement of the problem.

Comment author: 03 April 2009 06:29:20PM 1 point [-]

What is your point? You seem to be giving a metaphor for solving the problem by imagining that your action has a direct consequence of changing the past (and as a result, contents of the box in the present). More about that in this comment.

Comment author: 03 April 2009 10:42:25PM 3 points [-]

Naive argument coming up.

How Omega decides what to predict or what makes it's stated condition for B (aka. result of "prediction") come true, is not relevant. Ignoring the data that says it's always/almost always correct, however, seems ... not right. Any decision must be made with the understanding that Omega is most likely to predict it. You can't outsmart it by failing to update it's expected state of mind in the last second. The moment you decide to two-box is the moment Omega predicted, when it chose to empty box B.

Consider this:

Andy: "Sure, one box seems like the good choice, because Omega would take the million away otherwise. OK. ... Now that the boxes are in front of me, I'm thinking I should take both. Because, you know, two is better than one. And it's already decided, so my choice won't change anything. Both boxes."

Barry: "Sure, one box seems like the good choice, because Omega would take the million away otherwise. OK. ... Now that the boxes are in front of me, I'm thinking I should take both. Because, you know, two is better than one. Of course the outcome still depends on what Omega predicted. Say I choose both boxes. So if Omega's prediction is correct this time, I will find an empty B. But maybe Omega was wrong THIS time. Sure, and maybe THIS time I will also win the lottery. How it would have known is not relevant. The fact that O already acted on it's prediction doesn't make it more likely to be wrong. Really, what is the dilemma here? One box."

Ok, I don't expect that I'm the first person to say all this. But then, I wouldn't have expected anybody to two-box, either.

Comment author: 03 April 2009 11:01:43PM *  1 point [-]

major said:

Ignoring the data that says it's always/almost always correct, however, seems ... not right.

You're not the only person to wonder this. Either I'm missing something, or two-boxers just fail at induction.

I have to wonder how two-boxers would do on the "Hot Stove Problem."

In case you guys haven't heard of such a major problem in philosophy, I will briefly explain the Hot Stove Problem:

You have touched a hot stove 100 times. 99 times you have been burned. Nothing has changed about the stove that you know about. Do you touch it again?

Comment author: 04 April 2009 12:02:58AM 1 point [-]

I can see the relation to Newcomb - this is also a weird counterfactual that will never happen. I haven't deliberately touched a hot stove in my adult life, and don't expect to. I certainly won't get to 99 times.

Comment author: 03 April 2009 04:22:57PM *  3 points [-]

Quoting myself:

(though I don't see how you identify any distinction between "properties of the agent" and "decisions . . . predicted to be made by the agent" or why you care about it).

I'll go further and say this distinction doesn't matter unless you assume that Newcomb's problem is a time paradox or some other kind of backwards causation.

This is all tangential, though, I think.

Comment author: 03 April 2009 07:01:10PM 2 points [-]

If one defines rationality in some way that isn't about winning, your example shows that rationalists-in-such-a-sense might not win.

If one defines rationality as actually winning, your example shows that there are things that even Omega cannot do because they involve logical contradiction.

If one defines rationality as something like "expected winning given one's model of the universe" (for quibbles, see below), your example shows that you can't coherently carry around a model of the universe that includes a superbeing who deliberately acts so as to invalidate that model.

I find all three of these things rather unsurprising.

The traditional form of Newcomb's problem doesn't involve a superbeing deliberately acting so as to invalidate your model of the universe. That seems like a big enough difference from your version to invalidate inferences of the form "there's no such thing as acting rationally in grobstein's version of Newcomb's problem; therefore it doesn't make sense to use any version of Newcomb's problem in forming one's ideas about what constitutes acting rationally".

Comment author: 03 April 2009 07:55:42PM 1 point [-]

What is it, pray tell, that Omega cannot do?

Can he not scan your brain and determine what strategy you are following? That would be odd, because this is no stronger than the original Newcomb problem and does not seem to contain any logical impossibilities.

Can he not compute the strategy, S, with the property "that at each moment, acting as S tells you to act -- given (1) your beliefs about the universe at that point and (2) your intention of following S at all times -- maximizes your net utility [over all time]?" That would be very odd, since you seem to believe a regular person can compute S. If you can do it, why not Omega? (NB, no, it doesn't help to define an approximation of S and use that. If it's rational, Omega will punish you for it. If it's not, why are you doing it?)

Can he not compare your strategy to S, given that he knows the value of each? That seems odd, because a pushdown automaton could make the comparison. Do you require Omega to be weaker than a pushdown automaton?

No?

Then is it possible, maybe, that the problem is in the definition of S?

Comment author: 03 April 2009 11:45:01PM 3 points [-]

What is it, pray tell, that Omega cannot do?

Well, for instance, he cannot make 1+1=3. And, if one defines rationality as actually winning then he cannot act in such a way that rational people lose. This is perfectly obvious; and, in case you have misunderstood what I wrote (as it looks like you have), that is the only thing I said that Omega cannot do.

In the discussion of strategy S, my claim was not about what Omega can do but about what you (a person attempting to implement such a strategy) can consistently include in your model of the universe. If you are an S-rational agent, then Omega may decide to screw you over, in which case you lose; that's OK (as far as the notion of rationality goes; it's too bad for you) because S doesn't purport to guarantee that you don't lose.

What S does purport to do is to arrange that, in so far as the universe obeys your (incomplete, probabilistic, ...) model of it, you win on average. Omega's malfeasance is only a problem for this if it's included in your model. Which it can't be. Hence:

what your example shows [...] is that you can't consistently expect Omega to act in a way that falsifies your beliefs and/or invalidates your strategies for acting.

(Actually, I think that's not quite right. You could probably consistently expect that, provided your expectations about how he's going to to it were vague enough.)

I did not claim, nor do I believe, that a regular person can compute a perfectly rational strategy in the sense I described. Nor do I believe that a regular person can play chess without making any mistakes. None the less, there is such a thing as playing chess well; and there is such a thing as being (imperfectly, but better than one might be) rational. Even with a definition of the sort Eliezer likes.

Comment author: 03 April 2009 05:18:56PM 12 points [-]

Wikipedia has this right:

"a rational agent is specifically defined as an agent which always chooses the action which maximises its expected performance, given all of the knowledge it currently possesses."

Expected performance. Not actual performance. Whether its actual performance is good or not depends on other factors - such as how malicious the environment is, whether the agent's priors are good - and so on.

Comment author: 03 April 2009 05:35:30PM 0 points [-]

Of course, this isn't the first time I have pointed this out - see:

Nobody seemed to have any coherent criticism the last time around - and yet now we have the same issue all over again.

Comment author: 03 April 2009 05:55:44PM 4 points [-]

Problem with that in human practice is that it leads to people defending their ruined plans, saying, "But my expected performance was great!" Vide the failed trading companies saying it wasn't their fault, the market had just done something that it shouldn't have done once in the lifetime of the universe. Achieving a win is much harder than achieving an expectation of winning (i.e. something that it seems you could defend as a good try).

Comment author: 04 April 2009 01:55:43AM 5 points [-]

It sounds like the objection you're giving here is that "some people will misinterpret expected performance in the technical sense as expected performance in the colloquial sense (i.e., my guess as to how things will turn out)." That doesn't seem like much of a criticism though, and it doesn't sound severe enough to throw out what is a pretty standard definition. People will also misinterpret your alternate definition, as we have seen.

Do you have other objections?

Comment author: 03 April 2009 09:34:26PM 5 points [-]

What you say is important: the vast majority of whining "rationalists" weren't done dirty by a universe that "nobody could have foreseen" (the sub-prime mortgage crisis/piloting jets into buildings). If you sample a random loser claiming such (my reasoning was flawless, my priors incorporated all feasibly available human knowledge), an impartial judge would in nearly all cases correctly call them to task.

But clearly it's not always the case that my reasoning (and/or priors) is at fault when I lose. My updates shouldn't overshoot based on empirical noise and false humility. I think what you want to say is that most likely even (especially?) the most proud rationalists probably shield themselves from attributing their loss to their own error ("eat less salt").

I'd like some quantifiable demonstration of an externalizing bias, some calibration of my own personal tendency to deny evidence of my own irrationality (or of my wrong priors).

Comment author: 03 April 2009 07:46:08PM *  10 points [-]

Problem with that in human practice is that it leads to people defending their ruined plans, saying, "But my expected performance was great!"

It's true that people make this kind of response, but that doesn't make it valid, or mean that we have to throw away the notion of rationality as maximizing expected performance, rather than actual performance.

In the case of failed trading companies, can't we just say that despite their fantasies, their expected performance shouldn't have been so great as they thought? And the fact that their actual results differed from their expected results should cast suspicion on their expectations.

Perhaps we can say that expectations about performance be epistemically rational, and only then can an agent who maximizes their expected performance be instrumentally rational.

Achieving a win is much harder than achieving an expectation of winning (i.e. something that it seems you could defend as a good try).

Some expectations win. Some expectations lose. Yet not all expectations are created equal. Non-accidental winning starts with something that seems good to try (can accidental winning be rational?). At least, there is some link between expectations and rationality, such that we can call some expectations more rational than others, regardless of whether they actually win or lose.

An example SoullessAutomaton made was that we shouldn't consider lottery winners rational, even though they won, because they should not have expected to. Conversely, all sorts of inductive expectations can be rational, even though sometimes they will fail due to the problem of induction. For instance, it's rational to expect that the sun will rise tomorrow. If Omega decides to blow up the sun, my expectation will still have been rational, even though I turned out to be wrong.

Comment author: 03 April 2009 09:35:06PM 2 points [-]

Yet not all expectations are created equal. Non-accidental winning starts with something that seems good to try (can accidental winning be rational?).

In the real world, of course, most things are some mixture of controllable and randomized. Depending on your definition of accidental, it can be rational to make low-cost steps to position yourself to take advantage of possible events you have no control over. I wouldn't call this accidental, however, because the average expected gain should be net positive, even if one expects (id est, with confidence greater than 50%) to lose.

I used the lottery as an example because it's generally a clear-cut case where the expected gain minus the cost of participating is net negative and the controllable factor (how many tickets you buy) has extremely small impact.

Comment author: 03 April 2009 09:58:51PM 3 points [-]

Yes, and I liked your example for exactly this reason: the expected value of buying lottery tickets is negative.

I think that this shows that it is irrational to take an action where it's clear-cut that the expected value is negative, even though due to chance, one iteration of that action might produce a positive result. You are using accidental the same way I am: winning from an action with a negative expected value is what I would call accidental, and winning with a positive expected value is non-accidental.

Things are a bit more complicated when we don't know the expected value of an action. For example, in Eliezer's examples of failed trading companies, we don't know the correct expected value of their trading strategies, or whether they were positive or negative.

In cases where the expected value of an action is unknown, perhaps the instrumental rationality of the action is contingent on the epistemic rationality of our estimation of its expected value.

Comment author: 03 April 2009 10:24:16PM 3 points [-]

I like your definition of an accidental win, it matches my intuitive definition and is stated more clearly than I would have been able to.

In cases where the expected value of an action is unknown, perhaps the instrumental rationality of the action is contingent on the epistemic rationality of our estimation of its expected value.

Yes. Actually, I think the "In cases where the expected value of an action is unknown" clause is likely unnecessary, because the accuracy of an expected value calculation is always at least slightly uncertain.

Furthermore, the second-order calculation of the expected value of expending resources to increase epistemological rationality should be possible; and in the case that acting on a proposition is irrational due to low certainty, and the second-order value of increasing certainty is negative, the rational thing to do is shrug and move on.

Comment author: 03 April 2009 07:15:34PM 3 points [-]

I'm not sure how you can implement an admonition to Win and not just to (truly, sincerely) try. What is the empirical difference?

I suppose you could use an expected regret measure (that is, the difference between the ideal result and the result of the decision summed across the distribution of probable futures) instead of an expected utility measure.

Expected regret tends to produce more robust strategies than expected utility. For instance, in Newcomb's problem, we could say that two-boxing comes from expected utility but one-boxing comes from regret-minimizing (since a "failed" two-box gives $1,000,000-$1,000=$999,000 of regret, if you believe Omega would have acted differently if you had been the type of person to one-box, where a "failed" one-box gives$1000-$0=$1,000 of regret).

Using more robust strategies may be a way to more consistently Win, though perhaps the true goal should be to know when to use expected utility and when to use expected regret (and therefore to take advantage both of potential bonanzas and of risk-limiting mechanisms).

Comment author: 03 April 2009 08:00:05PM 2 points [-]

It's really easy to convince yourself that you've truly, sincerely tried - trying to try is not nearly as effective as trying to win.

Comment author: 03 April 2009 08:23:17PM *  0 points [-]

As for the "Trying-to-try" page - an argument from Yoda and the Force? It reads like something out of a self-help manual!

Sure: if you are trying to inspire confidence in yourself in order to improve your performance, then you might under some circumstances want to think only of winning - and ignore the possibility of trying and failing. But let's not get our subjects in a muddle, here - the topic is the definition of instrumental rationality, not how some new-age self-help manual might be written.

Comment author: 03 April 2009 08:06:21PM 2 points [-]

The intended distinction was originally between trying to win and actually winning. You are comparing two kinds of trying.

Comment author: 03 April 2009 08:24:28PM 4 points [-]

You are comparing two kinds of trying.

I'm not sure how you can implement an admonition to Win and not just to (truly, sincerely) try. What is the empirical difference?

Based on the above, I believe the distinction was between two different kinds of admonitions. I was pointing out that an admonition to win will cause someone to try to win, and an admonition to try will cause someone to try to try.

Comment author: 04 April 2009 11:37:28AM 3 points [-]

Thomblake's interpretation of my post matches my own.

Comment author: 03 April 2009 08:39:40PM 4 points [-]

Right, but again, the topic is the definition of instrumental rationality, and whether it refers to "trying to win" or "actually winning".

What do "admonitions" have to do with things? Are you arguing that because telling someone to "win" may some positive effect that telling someone to "try to win" lacks that we should define "instrumental rationality" to mean "winning" - and not "trying to win"?

Isn't that an idiosyncracy of human psychology - which surely ought to have nothing to do with the definition of "instrumental rationality".

Consider the example of handicap chess. You start with no knight. You try to win. Actually you lose. Were you behaving rationally? I say: you may well have been. Rationality is more about the trying, than it is about the winning.

Comment author: 03 April 2009 08:45:33PM 1 point [-]

The question was about admonitions. I commented based on that. I didn't mean anything further about instrumental rationality.

Comment author: 03 April 2009 09:08:16PM 4 points [-]

OK. I don't think we have a disagreement, then.

I consider it to be a probably-true fact about human psychology that if you tell someone to "try" rather than telling them to "win" then that introduces failure possibilites into their mind. That may have a positive effect, if they are naturally over-confident - or a negative one, if they are naturally wracked with self-doubt.

It's the latter group who buy self-help books: the former group doesn't think it needs them. So the self-help books tell you to "win" - and not to "try" ;-)

Comment author: 03 April 2009 08:45:48PM 5 points [-]

Here's a functional difference: Omega says that Box B is empty if you try to win what's inside it.

Comment author: 04 April 2009 01:58:56AM *  1 point [-]

Yes! This functional difference is very important!

In Logic, you begin with a set of non-contradicting assumptions and then build a consistent theory based on those assumptions. The deductions you make are analogous to being rational. If the assumptions are non-contradicting, then it is impossible to deduce something false in the system. (Analogously, it is impossible for rationality not to win.) However, you can get a paradox by having a self-referential statement. You can prove that every sufficiently complex theory is not closed -- there are things that are true that you can't prove from within the system. Along the same lines, you can build a paradox by forcing the system to try to talk about it itself.

What Grobstein has presented is a classic paradox and is the closest you can come to rationality not winning.

Comment author: 03 April 2009 10:55:43PM *  5 points [-]

I'm quite confident there is only a language difference between eliezer's description and the point a number of you have just made. Winning versus trying to win are clearly two different things, and it's also clear that "genuinely trying to win" is the best one can do, based on the definition those in this thread are using. But Eli's point on ob was that telling oneself "I'm genuinely trying to win" often results in less than genuinely trying. It results in "trying to try"...which means being satisfied by a display of effort rather than utility maximizing. So instead, he arguesn why not say to oneself the imperative "Win!", where he bakes the "try" part into the implicit imperative. I agree eli's language usage here may be slightly non standard for most of us (me included) and therefore perhaps misleading to the uninitiated, but I'm doubtful we need to stress about it too much if the facts are as I've stated. Does anyone disagree? Perhaps one could argue eli should have to say, "Rational agents should win_eli" and link to an Explanation like this thread, if we are genuinely concerned about people getting confused.

Comment author: 04 April 2009 05:06:29AM 2 points [-]

Eliezer seems to be talking about actually winning - e.g.: "Achieving a win is much harder than achieving an expectation of winning".

He's been doing this pretty consistently for a while now - including on his administrator's page on the topic:

That is why this discussion is still happening.

Comment author: 03 April 2009 08:10:50PM *  5 points [-]

Agents do try to win. The don't necessarily actually win. For example, if they face a superior opponent. Kasparov was behaving in a highly rational manner in his battle with Deep Blue. He didn't win. He did try to, though. Thus the distinction between trying to win and actually winning.

Comment author: 03 April 2009 06:48:19PM 13 points [-]

Expected performance is what rational agents are actually maximising.

Whether that corresponds to actual performance depends on what their expectations are. What their expectations are typically depends on their history - and the past is not necessarily a good guide to the future.

Highly rational agents can still lose. Rational actions (that follow the laws of induction and deduction applied to their sense data) are not necessarily the actions that win.

Rational agents try to win - and base their efforts on their expectations. Whether they actually win depends on whether their expectations are correct. In my view, attempts to link rationality directly to "winning" miss the distinction between actual and expected utility.

There are reasons for associations between expected performance and actual performance. Indeed, those associations are why agents have the expectations they do. However, the association is statistical in nature.

Dissect the brain of a rational agent, and it is its expected utility that is being maximised. Its actual utility is usually not something that is completely under its control.

It's important not to define the "rational action" as "the action that wins". Whether an action is rational or not should be a function of an agent's sense data up to that point - and should not vary depending on environmental factors which the agent knows nothing about. Otherwise, the rationality of an action is not properly defined from an agent's point of view.

I don't think that the excuses humans use for failures is an issue here.

Behaving rationally is not the only virtue needed for success. For example, you also need to enter situations with appropriate priors.

Only if you want rationality to be the sole virtue, should "but I was behaving rationally" be the ultimate defense against an inquisition.

Rationality is good, but to win, you also need effort, persistence, good priors, etc - and it would be very, very bad form to attempt to bundle all those into the notion of being "rational".

Comment author: 03 April 2009 07:28:57PM 2 points [-]

I am inclined to argue along exactly the same lines as Tim, though I worry there is something I am missing.

Comment author: 04 April 2009 11:32:32AM 9 points [-]

Expected performance is what rational agents are actually maximising.

Does that mean that I should mechanically overwrite my beliefs about the chance of a lottery ticket winning, in order to maximize my expectation of the payout? As Nesov says, rationality is about utility; which is why a rational agent in fact maximizes their expectation of utility, while trying to maximize utility (not their expectation of utility!).

It may help to understand this and some of the conversations below if you realize that the word "try" behaves a lot like "quotation marks" and that having an extra "pair" of quotation "marks" can really make "your" sentences seem a bit odd.

Comment author: 04 April 2009 11:41:17PM *  3 points [-]

I'm not sure I get this at all.

I offer you a bet, I'll toss a coin, and give you £100 if it comes up heads, you give me £50 if it comes up tails. Presumably you take the bet right? Because your expected return is £50 - surely this is the sense in which rationalists maximise expected utility. We don't mean "the amount of utility they expect to win", but expectation in the technical sense - ie, the product of the likelihood of various events happening with their utility in the univserses in which those events happen (or probably more properly an integral...)

If you expect to lose £50 and you are wrong, that doesn't actually say anything about the expectation of your winnings.

Comment author: 14 April 2009 03:57:56PM 2 points [-]

If you expect to lose £50 and you are wrong, that doesn't actually say anything about the expectation of your winnings.

It does, however, say something about your expectation of your winnings. Expectation can be very knowledge dependent. Let's say someone rolls two six sided dice, and then offers you a bet where you win $100 if the sum of the dice is less than 5, but lose$10 if the sum is greater than 5. You might perform various calculations to determine your expected value of accepting the bet. But if I happen to peak and see one of the dice has landed on 6, then I will calculate a different expected value than you will.

So we have different expected values for calculating the bet, because we have different information.

So EY's point is that if a rational agent's only purpose was to maximize (their) expected utility, they could easily do this by selectively ignoring information, so that their calculations turn out a specific way.

But actually rational agents are not interested in maximizing (their) expected utility. They are interested in maximizing real utility. Except it's impossible to do this without perfect information, and so what agents end up doing is maximizing expected utility, although they are trying to maximize real utility.

It's like if I'm taking a history exam in school. I am trying to achieve 100% on the exam, but end up instead achieving only 60% because I have imperfect information. My goal wasn't 60%, it was 100%. But the actual actions I took (the answers I selected) led to to arrive at 60% instead of my true goal.

Rational agents are trying to maximize real utility, but end up maximizing expected utility (by definition), even though that's not their true goal.

Comment author: 04 April 2009 12:23:24PM *  2 points [-]

Re: Does that mean that I should mechanically overwrite my beliefs about the chance of a lottery ticket winning, in order to maximize my expectation of the payout?

No, it doesn't. It means that the process going on in the brains of intelligent agents can be well modelled as calculating expected utilities - and then selecting the action that corresponds to the largest one.

Intelligent agents are better modelled as Expected Utility Maximisers than Utility Maximisers. Whether they actually maximise utility depends on whether they are in an environment where their expectations pan out.

Comment author: 05 April 2009 12:11:25AM 3 points [-]

Intelligent agents are better modelled as Expected Utility Maximisers than Utility Maximisers.

By definition, intelligent agents want to maximize total utility. In the absence of perfect knowledge, they act on expected utility calculations. Is this not a meaningful distinction?

Comment author: 03 April 2009 02:58:48PM 7 points [-]

Rationalists are the ones who win when things are fair, or when things are unfair randomly over an extended period. Rationality is an advantage, but it is not the only advantage, not the supreme advantage, not an advantage at all in some conceivable situations, and cannot reasonably be expected to produce consistent winning when things are unfair non-randomly. However, it is a cultivable advantage, which is among the things that makes it interesting to talk about.

A rationalist might be unfortunate enough that (s)he does not do well, but ceteris paribus, (s)he will do better. Maybe that could be the slogan - "rationalists do better"? With the implied parenthetical "(than they would do if they were not rationalists, with the caveat that you can concoct unlikely situations in which rationality is an impediment to some values of "doing well")".

Comment author: 03 April 2009 03:50:43PM 4 points [-]

"You can't reliably do better than rationality in a non-pathological universe" is probably closer to the math.

Comment author: 03 April 2009 05:34:31PM 5 points [-]

It's impossible to add substance to "non-pathological universe." I suspect circularity: a non-pathological universe is one that rewards rationality; rationality is the disposition that lets you win in a nonpathological universe.

You need to attempt to define terms to avoid these traps.

Comment author: 03 April 2009 05:52:13PM 0 points [-]

(likewise the fairness language of the parent post)

Comment author: 03 April 2009 07:32:08PM *  4 points [-]

Pathological universes are ones like: where there is no order and the right answer is randomly placed. Or where the facts are maliciously arranged to entrap in a recursive red herring where the simplest well-supported answer is always wrong, even after trying to out-think the malice. Or where the whole universe is one flawless red herring ("God put the fossils there to test your faith").

"No free lunch" demands they be mathematically conceivable. But to assert that the real universe behaves like this is to go mad.

Comment author: 03 April 2009 08:04:30PM 2 points [-]

Since we learn reason from the universe we're in, if we were in a universe you're referring to as "pathological", we (well, sentients, if any) would have learned a method of arriving at conclusions which matched that. Likewise, since the universe produced math, I don't think it has any meaning to talk of whether universes with different fundamental rules are "mathematically conceivable".

Comment author: 03 April 2009 09:17:06PM *  5 points [-]

http://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization

No search algorithm beats random picking in the totally general case. This implies the totally general case must include an equal balance of pathology and sanity. Intuitively, a problem could be structured so every good decision gives a bad result.

Edit: this post gives a perfect example of a pathological problem: there is only one decision to be made, a Bayesian loses, a random picker gets it right half the time and an anti-Bayesian wins.

However we seem to be living in a sane universe.

Comment author: 03 April 2009 03:09:31PM *  13 points [-]

Personally, I think the word "win" might be the problem. Winning is very binary, which isn't how rationality is defined. Perhaps "Rationalists maximize"?

Comment author: 03 April 2009 04:26:19PM *  25 points [-]

The about captures the expected systematic winning part, as you are considering the model of winning, not necessarily the accidental winning itself. It limits the scope to the winning only, leaving only the secondary roles for parry, hit, spring, strike or touch. Being a study about the real thing, rationality employs a set of tricks that allow to work it, in special cases and at coarse levels of detail. Being about the real thing, rationality aims to give the means for actually winning.

Comment author: 13 April 2013 06:57:37PM 1 point [-]

I'm not sure if it's better, but here's one that works well. Similar to the phrase, "Physician, heal thyself!" another way to say rationalists should win is to say, "Rationalist, improve thyself!"

If you aren't actually improving yourself and the world around you, then you aren't using the tools of rationality correctly. And it follows that to improve the world around you, you first have to be in a position to do so by doing the same to yourself.

Comment author: 21 December 2009 05:59:59AM 2 points [-]

"Rationality is whatever wins."

If it's not a winning strategy, you're not doing it right. If it is a winning strategy, overall in as long of terms as you can plan, then it's rationality. It doesn't matter what the person thinks: whether they'd call themselves rationalists or not.

Comment author: 16 April 2009 08:19:05PM 1 point [-]

I one box newcombs problem because the payoffs are too disproportionate to make it interesting. how about this? if omega predicted you would two box they are both empty if omega predicted you would one box both boxes have $1000 Comment author: 05 April 2009 03:36:56AM 3 points [-] Alleged rationalists should not find themselves envying the mere decisions of alleged nonrationalists, because your decision can be whatever you like. Eliezer said this in the Newcomb's Problem post which introduced "Rationalists should win". Perhaps for a slogan, shorten it to: "Rationalists should not envy the mere decisions of nonrationalists." This emphasizes that rationality contributes to winning through good decisions. A potential problem is that, in some circumstances, an alleged rationalist could find a factor that seems unrelated to their decisions to blame for losing, and therefore argue that their being rational is consistent with the slogan. For example, a someone who blames losing on luck might need to reconsider their probability theory that is informing their decisions. Though this should not be a fully general counterargument, someone who wins more often than others in the same situation is likely doing something right, even if they do not win with probability 1. Comment author: 04 April 2009 08:25:54AM * 1 point [-] Let's use one of Polya's 'how to solve it' strategies and see if the inverse helps: Irrationalists should lose. Irrationality is systematized losing. On another note, rationality can refer to either beliefs or behaviors. Does being a rationalist mean your beliefs are rational, your behaviors are rational, or both? I think behaving rationally, even with high probability priors, is still very hard for us humans in a lot of circumstances. Until we have full control of our subconscious minds and can reprogram our cognitive systems, it is a struggle to will ourselves to act in completely rational ways. To spread rationality, amongst humans at least, we might want to consider a divide and conquer approach focusing on people maintaining rational beliefs first, and maximizing rational behavior second. Comment author: 04 April 2009 08:22:55AM 3 points [-] Both boxes might be transparent. In this case, you would see the money in both boxes only if you are rational enough to understand, that you have to pick just B. Wouldn't that be an irrational move? Not all! You have to understand that to be rational. Comment author: 04 April 2009 06:54:32PM 2 points [-] That's brilliant! (I'm not sure what you mean by understand though.) In other words, Omega does one of the two things: it either offers you$1000 + $1, or only$10. It offers you $1000 +$1 only if it predicts that you won't take the $1, otherwise it only gives you$10.

This is a variant of counterfactual mugging, except that there is no chance involved. Your past self prefers to precommit to not taking the $1, while your present self faced with that situation prefers to take the 1$.

Comment author: 04 April 2009 07:53:39PM *  0 points [-]

Hmmm... It looks like the decision to take the $1 determines the situation where you make that decision out of reality. Effects of precommitment being restricted to the counterfactual branches are a usual thing, but in this problem they stare you right in the face, which is rather daring. Comment author: 04 April 2009 08:05:32PM 1 point [-] If you want your source code to be self-consistent under reflection, you know what you have to do. Comment author: 04 April 2009 08:21:07PM * 5 points [-] Another variation, playing only on real/counterfactual, without motivating the real decision. Omega comes to you and offers$1, if and only if it predicts that you won't take it. What do you do? It looks neutral, since expected gain in both cases is zero. But the decision to take the $1 sounds rather bizarre: if you take the$1, then you don't exist!

Agents self-consistent under reflection are counterfactual zombies, indifferent to whether they are real or not.

Comment author: 02 July 2011 07:53:01PM -1 points [-]

Seems roughly as disturbing as Wikipedia's article on Gaussian adaptation:

Gaussian adaptation as an evolutionary model of the brain obeying the Hebbian theory of associative learning offers an alternative view of free will due to the ability of the process to maximize the mean fitness of signal patterns in the brain by climbing a mental landscape in analogy with phenotypic evolution.

Such a random process gives us lots of freedom of choice, but hardly any will. An illusion of will may, however, emanate from the ability of the process to maximize mean fitness, making the process goal seeking. I. e., it prefers higher peaks in the landscape prior to lower, or better alternatives prior to worse. In this way an illusive will may appear. A similar view has been given by Zohar 1990. See also Kjellström 1999.

Comment author: 04 April 2009 01:46:35AM 6 points [-]

Rationality seems like a good name for the obvious ideal that you should believe things that are true and use this true knowledge to achieve your goals. Because social organisms are weird in ways whose details are beyond the scope of this comment, striving to be more rational might not pay off for a human seeking to move up in a human world---but aside from this minor detail relating to an extremely pathological case, it's still probably a good idea.

Comment author: 04 April 2009 05:33:55AM *  4 points [-]

Hmmm. Unless you are suggesting a different definition for rationality, I think I disagree. If an atheist has the goal of gaining business contacts (or something) and he can further this goal by joining a church, and impersonating the irrational behaviors he sees, he isn't being irrational. While behaviors that tend to have their origins in irrational thought are sometimes rewarded by human society, the irrationality itself never is. I think becoming more rational will help a person move up in a human status hierarchy, if that is the rationalist's goal. I think we have this stereotyped idea of rationalists as Asperger's-afflicted know-it-alls who are unable to deal with irrational humans. It simply doesn't have to be that way.

Comment author: 05 April 2009 02:14:36AM 2 points [-]

I denotatively agree with your conclusion, but I think that many if not most aspiring rationalists are incapable of that level of Machiavellianism. Suppose that your typical human cares about both social status and being forthright, and that there are social penalties for making certain true but unpopular statements. Striving for rationality in this situation, could very well mean having to choose between popularity and honesty, whereas the irrationalist can have her cake and eat it, too. So yes, some may choose popularity---but you see, it is a choice.

Comment author: 03 April 2009 09:20:15PM 3 points [-]

Rationality is winning that doesn’t generate a surprise; randomly winning the lottery generates a surprise. A good measure of rationality is the amount of complexity involved in order to win, and the surprise generated by that win. If to win at a certain task requires that your method have many complex steps, and you win, non-surprisingly, then the method used was a very rational one.

Comment author: 03 April 2009 08:37:58PM 4 points [-]

The rationality that doesn't secure your wish isn't the true rationality.

Winning has no fixed form. You'll do whatever is needed to succeed, however original or far fetched it would sound. How it sounds is irrelevant, how it works is the crux.

And If at first what you tried didn't work, then you'll learn, adapt, and try again, making no pause for excuses, if you merely want to succeed, you'll be firm as a rock, relentless in your attempts to find the path to success.

And if your winning didn't go as smoothly or well as you wanted or thought it should, in general, then learn, adapt, and try again. Think outside of the box, self recurse on winning itself. Eventually, you should refine and precise your methods into a tree, from general to specialized.

That tree will have a trunk of general cases and methods used to solve those, and any case that lies ahead, upwards on the tree; and the higher you go, the more specialized the method, the rarer the case it solves. The tree isn't fixed either, it can and will grow and change.

Comment author: 04 April 2009 08:02:21AM 1 point [-]

Re: The rationality that doesn't secure your wish isn't the true rationality.

Again with the example of handicap chess. You start with no knight. You wish to win. Actually you lose. Does that mean you were behaving irrationally? No, of course not! It is not whether you win or lose, but how you play the game.

Comment deleted 04 April 2009 07:22:15AM [-]
Comment author: 04 April 2009 12:02:42PM 3 points [-]

Say "Rationalists are" instead of "Rationality is" and I'll agree with that.

Comment author: 03 April 2009 06:50:22PM 5 points [-]

It seems to me that the disagreement isn't so much about winning as the expectation.

In fact I don't really agree with this winning vs. belief modes of rationality.

Both approaches are trying to maximize their expected payout. Eliezer's approach has a wider horizon of what it considers when figuring out what the universe is like.

The standard approach is that since the content of the boxes is already determined at the time of the choice, so taking both will always put you $1000 ahead. Eliezer looks (I think) out to the most likely final outcomes. (or looks back at how the chain of causality of one's decision is commingled with the chain of causality of Omega's decision. ) I think that flaw in the standard approach is not 'not winning' but a false belief about the relationship between the boxes and your choices. (the belief that there isn't any.) Once you have the right answer, making the choice that wins is obvious. The way we would know that the standard approach is the wrong one is by looking at results. That a certain set of choices consistently wins isn't evidence that it is rational, it is evidence that it wins. Believing that it wins is rational. So maybe: "Rationality is learning how to win" Comment author: 03 April 2009 08:00:42PM 2 points [-] "Rationality is learning how to win" I like that. Comment author: 03 April 2009 06:39:02PM 6 points [-] I always thought that the majority of exposition in your Newcomb example went towards, not "Rationalists should WIN", but a weaker claim which seems to be a smaller inferential distance from most would-be rationalists: Rationalists should not systematically lose; whatever systematically loses is not rationality. (Of course, one needs the logical caveat that we're not dealing with a pure irrationalist-rewarder; but such things don't seem to exist in this universe at the moment.) Comment author: 04 April 2009 08:10:19AM 0 points [-] Re: Rationalists should not systematically lose; whatever systematically loses is not rationality. Even if you are playing go with a 9-stone handicap against a shodan? Comment author: 04 April 2009 08:14:07AM 3 points [-] "Lose" = "perform worse than another (usable) strategy, all preferences considered". Comment author: 04 April 2009 08:17:46AM -1 points [-] Nick, show me a dictionary with this in and we can talk. Otherwise, it seems as though you are redefining a perfectly common and ordinary english word to mean something esoteric and counter-intuitive. Comment author: 05 April 2009 04:43:18AM 2 points [-] Well, I don't think I'd fare better by thinking less rationally; and if I really needed to find a way to win, rationality at least shouldn't hurt me me in the process. I was hoping to be pithy by neglecting a few implicit assumptions. For one, I mean that (in the absence of direct rewards for different cognitive processes) good rationalists shouldn't systematically lose when they can see a strategy that systematically wins. Of course there are Kobayashi Maru scenarios where all the rationality in the world can't win, but that's not what we're talking about. Comment author: 03 April 2009 05:08:29PM 4 points [-] What about cases where any rational course of action still leaves you on the losing side? Although this may seem to be impossible according to your definition of rationality, I believe it's possible to construct such a scenario because of the fundamental limitations of a human brains ability to simulate. In previous posts you've said that, at worst, the rationalist can simply simulate the 'irrational' behaviour that is currently the winning strategy. I would contend that humans can't simulate effectively enough for this to be an option. After all we know that several biases stem from our inability to effectively simulate our own future emotions, so to effectively simulate an entire other beings response to a complex situation would seem to be a task beyond the current human brain. As a concrete example I might suggest the ability to lie. I believe it's fairly well established that humans are not hugely effective liars and therefore the most effective way to lie is to truly believe the lie. Does this not strongly suggest that limitations of simulation mean that a rational course of action can still be beaten by an irrational one? I'm not sure that even if this is true it should effect a universal definition of rationality - but it would place bounds on the effectiveness of rationality in beings of limited simulation capacity. Comment author: 03 April 2009 04:29:14PM * 4 points [-] If humans are imperfect actors then in situations (such as a game of chicken) in which it is better to (1) be irrational and seen as irrational then it is to (2) be rational and seen as rational then the rational actor will lose. Of course holding constant everyone else's beliefs about you, you always gain by being more rational. Comment author: 03 April 2009 05:33:32PM 1 point [-] James, when you say, "be rational", I think this shows a misunderstanding. It may be really important to impress people with a certain kind of reckless courage. Then it is Rational to play chicken as bravely as you can. This Wins in the sense of being better than the alternative open to you. Normally, I do not want to take the risk of being knocked down by a car. Only in this case is it not rational to play chicken: because not playing achieves what I want. I do not see why a rationalist should be less courageous, less able to estimate distances and speeds, and so less likely to win at Chicken. Comment author: 03 April 2009 06:33:21PM 3 points [-] No. The point is that you actually want to survive more than you want to win, so if you are rational about Chicken you will sometimes lose (consult your model for details). Given your preferences, there will always be some distance \epsilon before the cliff where it is rational for you to give up. Therefore, under these assumptions, the strategy "win or die trying" seemingly requires you to be irrational. However, if you can credibly commit to this strategy -- be the kind of person who will win or die trying -- you will beat a rational player every time. This is a case where it is rational to have an irrational disposition, a disposition other than doing what is rational at every margin. Comment author: 03 April 2009 06:40:09PM -1 points [-] But a person who truly cares more about winning than surviving can be utterly rational in choosing that strategy. Comment author: 03 April 2009 07:04:16PM 1 point [-] Agreed. In fact, the classic game-theoretic model of chicken requires that the players vastly prefer losing their pride to losing their lives. If winning/losing > losing/dying, then in a situation with imperfect information, we would assign a positive probability to playing aggressively. And technically speaking, it is most rational, in the game-theoretic sense, to disable your steering ostentatiously before the other player does so as well. In that case, you've won the game before it begins, and there is no actual risk. Comment author: 03 April 2009 07:17:07PM 1 point [-] No, if you are rational the best action is to convince your opponent that you have disabled your steering when in fact you have not done so. Comment author: 04 April 2009 06:33:30PM 1 point [-] Either a) your opponent truly does believe that you've disabled your steering, in which case the outcomes are identical and the actions are equally rational, or b) we account for the (small?) chance that your opponent can determine that you actually have not disabled your steering, in which case he ostentatiously disables his and wins. Only by setting up what is in effect a doomsday device can you ensure that he will not be tempted to information-gathering brinksmanship. Comment author: 03 April 2009 06:00:23PM 3 points [-] Given that I one-box on Newcomb's Problem and keep my word as Parfit's Hitchhiker, it would seem that the rational course of action is to not steer your car even if it crashes (if for some reason winning that game of chicken is the most important thing in the universe). Comment author: 03 April 2009 09:24:31PM 3 points [-] For Newcomb's Problem, is it fair to say that if you believe the given information, the crux is whether you believe it's possible (for Omega) to have a 99%+ correct prediction of your decision based on the givens? Refusal to accept that seems to me the only justification for two-boxing. Perhaps that's a sign that I'm less tied to a fixed set of "rationalist" procedures than a perfect rationalist would be, but I would feel like I were pretending to say otherwise. I also wonder if the many public affirmations I've heard of "I would one-box Newcomb's Problem" are attempts at convincing Omega to believe us in the unlikely event of actually encountering the Problem. It does give a similar sort of thrill to "God will rapture me to heaven." Comment author: 03 April 2009 07:12:53PM 4 points [-] You are playing chicken with your irrational twin. Both of you would rather survive than win. Your twin, however, doesn't understand that it's possible to die when playing chicken. In the game your twin both survives and wins whereas you survive but lose. Comment author: 03 April 2009 08:06:39PM * 1 point [-] Then you murder the twin prior to the game of chicken, and fake his suicide. Or you intimidate the twin, using your advanced rational skills to determine how exactly to best fill them with fear and doubt. But before murdering or risking an uncertain intimidation feint, there's another question you need to ask yourself. How certain are you that the twin is irrational? The Cold War was (probably) a perceptual error; neither side realized that they were in a prisoners dilemma, they both assumed that the other side preferred "unbalanced armament" over "mutual armament" over "mutual disarmament;" in reality, the last two should have been switched. Worst case scenario? You die playing chicken, because the stakes were worth it. The Rational path isn't always nice. (There are some ethical premises implicit in this argument, premises which I plan to argue are natural derivatives from Game Theory... but I'm still working on that article.) Comment author: 03 April 2009 07:21:48PM 0 points [-] My answer to that one is that I don't play chicken in the first place unless the stake is something I'm prepared to die for. Comment author: 03 April 2009 07:27:07PM 5 points [-] There are lots of chicken like games that don't involve death. For example, your boss wants some task done and either you or a co-worker can do it. The worst outcome for both you and the co-worker is for the task to not get done. The best is for the other person to do the task. Comment author: 03 April 2009 07:30:37PM 2 points [-] My answer still applies - I'm not going to make a song and dance about who does it, unless the other guy has been systematically not pulling his weight and it's got to the point where that matters more to me than this task getting done. Comment author: 03 April 2009 06:39:43PM 1 point [-] +1 for "Rationalists win". What is Parfit's Hitchhiker? I couldn't find an answer on Google. Comment author: 03 April 2009 07:05:48PM 3 points [-] It's a test case for rationality as pure self-interest (really it's like an altruistic version of the game of Chicken). Suppose I'm purely selfish and stranded on a road at night. A motorist pulls over and offers to take me home for$100, which is a good deal for me. I only have money at home. I will be able to get home then IFF I can promise to pay $100 when I get home. But when I get home, the marginal benefit to paying$100 is zero (under assumption of pure selfishness). Therefore if I behave rationally at the margin when I get home, I cannot keep my promise.

I am better off overall if I can commit in advance to keeping my promise. In other words, I am better off overall if I have a disposition which sometimes causes me to behave irrationally at the margin. Under the self-interest notion of rationality, then, it is rational, at the margin of choosing your disposition, to choose a disposition which is not rational under the self-interest notion of rationality. (This is what Parfit describes as an "indirectly self-defeating" result; note that being indirectly self-defeating is not a knockdown argument against a position.)

Comment author: 03 April 2009 08:02:46PM *  1 point [-]

Thank you, I too was curious.

We need names for these positions; I'd use hyper-rationalist but I think that's slightly different. Perhaps a consequentialist does whatever has the maximum expected utility at any given moment, and a meta-consequentialist is a machine built by a consequentialist which is expected to achieve the maximum overall utility at least in part through being trustworthy to keep commitments a pure consequentialist would not be able to keep.

I guess I'm not sure why people are so interested in this class of problems. If you substitute Clippy for my lift, and up the stakes to a billion lives lost later in return for two billion saved now, there you have a problem, but when it's human beings on a human scale there are good ordinary consequentialist reasons to honour such bargains, and those reasons are enough for the driver to trust my commitment. Does anyone really anticipate a version of this situation arising in which only a meta-consequentialist wins, and if so can you describe it?

Comment author: 03 April 2009 08:07:48PM 1 point [-]

I do think these problems are mostly useful for purposes of understanding and (moreso) defining rationality ("rationality"), which is perhaps a somewhat dubious use. But look how much time we're spending on it.

Comment author: 03 April 2009 08:05:42PM *  2 points [-]

I very much recommend Reasons and Persons, by the way. A friend stole my copy and I miss it all the time.

Comment author: 03 April 2009 11:35:44PM 1 point [-]

It's still in print and readily available. If you really miss it all the time, why haven't you bought another copy?

Comment author: 03 April 2009 11:37:27PM 0 points [-]

It's $45 from Amazon. At that price, I'm going to scheme to steal it back first. OR MAYBE IT'S BECAUSE I'M CRAAAZY AND DON'T ACT FOR REASONS! Comment author: 04 April 2009 12:35:49AM 2 points [-] Gosh. It's only £17 in the UK. (I wasn't meaning to suggest that you're crazy, but I did wonder about ... hmm, not sure whether there's a standard name for it. Being less prepared to spend X to get Y on account of having done so before and then lost Y. A sort of converse to the endowment effect.) Comment author: 04 April 2009 06:51:48AM 2 points [-] Mental accounting has that effect in the short run, but seems unlikely to apply here. Comment author: 04 April 2009 08:38:38AM 3 points [-] OK, thanks! Your friend stole a book on moral philosophy? That's pretty special! Comment author: 05 April 2009 02:18:47PM 3 points [-] Comment author: 03 April 2009 07:19:33PM 2 points [-] Ah, thanks. I'm of the school of thought that says it is rational both to promise to pay the$100, and to have a policy of keeping promises.

Comment author: 03 April 2009 08:22:58PM 1 point [-]

I think it is both right and expected-utility-maximizing to promise pay the $100, right to pay the$100, and not expected-utility-maximizing to pay the $100 under standard assumptions of you'll never see the driver again or whatnot. Comment author: 03 April 2009 08:31:48PM 1 point [-] You're assuming it does no damage to oneself to break one's own promises. Virtue theorists would disagree. Breaking one's promises damages one's integrity - whether you consider that a trait of character or merely a valuable fact about yourself, you will lose something by breaking your promise even if you never see the fellow again. Comment author: 03 April 2009 08:39:51PM 1 point [-] Your argument is equivalent to, "But what if your utility function rates keeping promises higher than a million orgasms, what then?" The hypo is meant to be a very simple model, because simple models are useful. It includes two goods: getting home, and having$100. Any other speculative values that a real person might or might not have are distractions.

Comment author: 03 April 2009 11:44:51PM 2 points [-]

Simple models are fine as long as we don't forget they are only approximations. Rationalists should win in the real world.

Comment author: 03 April 2009 08:43:00PM 2 points [-]

Except that you mention both persons and promises in the hypothetical example, so both things factor into the correct decision. If you said that it's not a person making the decision, or that there's no promising involved, then you could discount integrity.

Comment author: 03 April 2009 07:29:55PM 1 point [-]

Yes, this seems unimpeachable. The missing piece is, rational at what margin? Once you are home, it is not rational at the margin to pay the \$100 you promised.

Comment author: 03 April 2009 08:08:43PM 2 points [-]

This assumes no one can ever find out you didn't pay, as well. In general, though, it seems better to assume everything will eventually be found out by everyone. This seems like enough, by itself, to keep promises and avoid most lies.

Comment author: 03 April 2009 08:09:55PM 1 point [-]

Right. The question of course is, "better" for what purpose? Which model is better depends on what you're trying to figure out.

Comment author: 03 April 2009 06:14:43PM *  1 point [-]

Why don't you accept his distinction between acting rationally at a given moment and having the disposition which it is rational to have, integrated over all time?

EDIT: er, Parfit's, that is.

Comment author: 03 April 2009 04:43:11PM 3 points [-]

This is a classic point and clearer than the related argument I'm making above. In addition to being part of the accumulated game theory learning, it's one of the types of arguments that shows up frequently in Derek Parfit's discussion of what-is-rationality, in Ch. 1 of Reasons and Persons.

I feel like there are difficulties here that EY is not attempting to tackle.

Comment author: 03 April 2009 04:08:42PM *  2 points [-]

All else being equal, shouldn't rationalists, almost by definition, win? The only way this wouldn't happen would be in a contest of pure chance, in which rationality could confer no advantage. It seems like we're just talking semantics here.

Comment author: 04 April 2009 10:58:45PM 3 points [-]

If human beings had perfect control over their minds and bodies -- e.g., could tweak System 1 without limit and perform any physically possible act/behavior -- your point would be stronger.

However, as others have mentioned elsewhere, there may be cases where we are just not capable of implementing a strategy that rationality suggests is optimal (e.g., convincingly pretending to be more confident than you are to the point that all relevant System 1 impulses/reactions are those of a person who is naturally overconfident).

It may be the case that an ubermensch rationalist can eventually learn to do anything that can be done via non-rational means, but that's not clear a priori, especially if we consider finite lifespans and opportunity costs.

Comment author: 04 April 2009 11:04:55PM 3 points [-]

Agreed. Particularly in hypothetical cases where one rationally concludes that it would be in their best interest to behave irrationally, e.g., over-confidence in oneself or belief in God. Even if one arrived at those conclusions, it's not clear to me how anyone could decide to become irrational in those ways. Pascal's notion of "bootstrapping" oneself into religious belief never struck me as very plausible. Interestingly though, "faking" confidence in oneself often does tend to lead to real confidence via some sort of feedback mechanism, e.g., interactions with women.

Comment author: 03 April 2009 03:33:55PM 1 point [-]

"abandon reasonableness" is never necessary; though I think we may be using reasonable somewhat differently. I think "reasonable" includes the idea of "appropriate to the situation"

quoting myself : "There is a supposed "old Chinese saying": The wise man defends himself by never being attacked. Which is excellent, if incomplete, advice. I completed it myself with "But only an idiot counts on not being attacked." Don't use violence unless you really need to, but if you need to don't hold back." http://williambswift.blogspot.com/2009/03/violence.html

As to your overall point, I agree that rationalists should win. General randomness, unknowns, and opposition from other agents prevent consistent victories in the real world. But if you are not winning more than losing you definitely are not being rational.

Comment author: 04 April 2009 12:06:51PM 0 points [-]

Standard proverb: "If you would have peace, prepare for war."

Comment author: 03 April 2009 03:43:23PM 4 points [-]

Don't use violence unless you really need to, but if you need to don't hold back.

By corollary:

"Rule #6: If violence wasn't your last resort, you failed to resort to enough of it." -- The Seven Habits of Highly Effective Pirates

Comment author: 03 April 2009 03:32:33PM *  8 points [-]

Suggestion: "Rationalists seek to Win, not to be rational".

Suggestion: "If what you think is rational appears less likely to Win than what you think is irrational, then you need to reassess probabilities and your understanding of what is rational and what is irrational".

Suggestion: "It is not rational to do anything other than the thing which has the best chance of winning".

If I have a choice between what I define as the "Rational" course of action, and a course of action which I describe as "irrational" but which I predict has a better chance of winning, I am either predicting badly or wrongly defining what is Rational.

I am not sure my suggestions are Better, but I am groping towards understanding and hope my gropings help.

EDIT: and the warning is that we may deceive ourselves into thinking that we are being rational, when we are missing something, using the wrong map, arguing fallaciously. So what about:

Suggestion: "If you are not Winning, consider whether you are really being rational".

"If you are not Winning more than people you believe to be irrational, this may be evidence that you are not really being rational".

On a different tack, "Rationalists win wherever rationality is an aid to winning". I am not going to win millions on the Lottery, because I do not play it.

Comment author: 03 April 2009 03:51:47PM 3 points [-]

Suggestion: "If you are not Winning, consider whether you are really being rational".

The problem with this is that winning as a metric is swamped with random noise and different starting points.

Someone winning the lottery when you don't is not evidence that you are not being rational.

Someone whose parents were both high-paid lawyers making a fortune in business when you don't is not evidence that you are not being rational.

Comment author: 03 April 2009 08:11:26PM 1 point [-]

I disagree. Someone winning the lottery when you don't is evidence that you are not being rational, if getting a large sum of money for little effort is a goal you'd shoot for. But on evaluation, it should be seen as evidence that counts for little or nothing. Most of us have already done that evaluation.