In the comments to this post, several people independently stated that being risk-averse is the same as having a concave utility function. There is, however, a subtle difference here. Consider the example proposed by one of the commenters: an agent with a utility function

u = sqrt(p) utilons for p paperclips.

The agent is being offered a choice between making a bet with a 50/50 chance of receiving a payoff of 9 or 25 paperclips, or simply receiving 16.5 paperclips. The expected payoff of the bet is a full 9/2 + 25/2 = 17 paperclips, yet its expected utility is only 3/2 + 5/2 = 4 = sqrt(16) utilons which is less than the sqrt(16.5) utilons for the guaranteed deal, so our agent goes for the latter, losing 0.5 expected paperclips in the process. Thus, it is claimed that our agent is risk averse in that it sacrifices 0.5 expected paperclips to get a guaranteed payoff.

Is this a good model for the cognitive bias of risk aversion? I would argue that it's not. Our agent ultimately cares about utilons, not paperclips, and in the current case it does perfectly fine at rationally maximizing expected utilons. A cognitive bias should be, instead, some irrational behavior pattern that can be exploited to take utility (rather than paperclips) away from the agent. Consider now another agent, with the same utility function as before, but who just has this small additional trait that it would strictly prefer a sure payoff of 16 paperclips to the above bet. Given our agent's utility function, 16 is the point of indifference, so could there be any problem with his behavior? Turns out there is. For example, we could follow the post on Savage's theorem (see Postulate #4). If the sure payoff of

16 paperclips = 4 utilons

is strictly preferred to the bet

{P(9 paperclips) = 0.5; P(25 paperclips) = 0.5} = 4 utilons,

then there must also exist some finite δ > 0 such that the agent must strictly prefer a guaranteed 4 utilons to betting on

{P(9) = 0.5 - δ; P(25) = 0.5 + δ) = 4 + 2δ utilons

- all at the loss of 2δ expected utilons! This is also equivalent to our agent being willing to pay a finite amount of paperclips to substitute the bet with the sure deal of the same expected utility.

What we have just seen falls pretty nicely within the concept of a bias. Our agent has a perfectly fine utility function, but it also has this other thing - let's name it "risk aversion" - that makes the agent's behavior fall short of being perfectly rational, and is independent of its concave utility function for paperclips. (Note that our agent has linear utility for utilons, but is still willing to pay some amount of those to achieve certainty) Can we somehow fix our agent? Let's see if we can redefine our utility function u'(p) in some way so that it gives us a consistent preference of

guaranteed 16 paperclips

over the

 {P(9) = 0.5; P(25) = 0.5}

bet, but we would also like to request that the agent would still strictly prefer the bet

{P(9 + δ) = 0.5; P(25 + δ) = 0.5}

to {P(16) = 1} for some finite δ > 0, so that our agent is not infinitely risk-averse. Can we say anything about this situation? Well, if u'(p) is continuous, there must also exist some number δ' such that 0 < δ' < δ and our agent will be indifferent between {P(16) = 1} and

{P(9 + δ') = 0.5; P(25 + δ') = 0.5}.

And, of course, being risk-averse (in the above-defined sense), our supposedly rational agent will prefer - no harm done - the guaranteed payoff to the bet of the same expected utility u'... Sounds familiar, doesn't it?

I would like to stress again that, although our first agent does have a concave utility function for paperclips, which causes it to reject bets with some expected payoff of paperclips to guaranteed payoffs of less paperclips, it still maximizes its expected utilons, for which it has linear utility. Our second agent, however, has this extra property that causes it to sacrifice expected utilons to achieve certainty. And it turns out that with this property it is impossible to define a well-behaved utility function! Therefore it seems natural to distinguish being rational with a concave utility function, on the one hand, from, on the other hand, being risk-averse and not being able to have a well-behaved utility function at all. The latter case seems much more subtle at the first sight, but causes a more fundamental kind of problem. Which is why I feel that a clear, even if minor, distinction between the two situations is still worth making explicit.

A rational agent can have a concave utility function. A risk-averse agent can not be rational.

(Of course, even in the first case the question of whether we want a concave utility function is still open.)

I've seen of people on Lesswrong taking cognitive structures that I consider to be biases as terminal values. Take risk aversion for example:

Risk Aversion

For a rational agent with goals that don't include "being averse to risk", risk aversion is a bias. The correct decision theory acts on expected utility, with utility of outcomes and probability of outcomes factored apart and calculated separately. Risk aversion does not factor them.

EDIT: There is some contention on this. Just substitute "that thing minimax algorithms do" for "risk aversion" in my writing. /EDIT

A while ago, I was working through the derivation of A* and minimax planning algorithms from a Bayesian and decision-theoretic base. When I was trying to understand the relationship between them, I realized that strong risk aversion, aka minimax, saves huge amounts of computation compared to the correct decision theory, and actually becomes more optimal as the environment becomes more influenced by rational opponents. The best way win is to deny the opponents any opportunity to weaken you. That's why minimax is a good algorithm for chess.

Current theories about the origin of our intelligence say that we became smart to outsmart our opponents in complex social games. If our intelligence was built for adversarial games, I am not surprised at risk aversion.

A better theoretical replacement, and a plausible causal history for why we have the bias instead of the correct algorithm are convincing to me as an argument against risk aversion as a value the way a rectangular 13x7 pebble heap is convincing to a pebble sorter as an argument against the correctness of a heap of 91 pebbles; it seems undeniable, but I don't have access to the hidden values that would say for sure.

And yet I've seen people on LW state that their "utility function" includes risk aversion. Because I don't understand the values involved, all I can do is state the argument above and see if other people are as convinced as me.

It may seem silly to take a bias as terminal, but there are examples with similar arguments that are less clear-cut, and some that we take as uncontroversially terminal:

Responsibility and Identity

The feeling that you are responsible for some things and not others, like say, the safety of your family, but not people being tortured in Syria, seems noble and practical. But I take it to be a bias.

I'm no evolutionary psychologist, but it seems to me that feelings of responsibility are a quick hack to kick you into motion where you can affect the outcome and the utility at stake is large. For the most part, this aligns well with utilitarianism; you usually don't feel responsible for things you can't really affect, like people being tortured in Syria, or the color of the sky. You do feel responsible to pull a passed out kid off the train tracks, but maybe you don't feel responsible to give them some fashion advice.

Responsibility seems to be built on identity, so it starts to go weird when you identify or don't identify in ways that didn't happen in the ancestral environment. Maybe you identify as a citizen of the USA, but not of Syria, so you feel shame and responsibility about the US torturing people, but the people being tortured in Syria are not your responsibility, even though both cases are terrible, and there is very little you can do about either. A proper utilitarian would feel approximately the same desire to do something about each, but our responsibility hack emphasizes responsibility for the actions of the tribe you identify with.

You might feel great responsibility to defend your past actions but not those other people, even tho neither is worth "defending". A rational agent would learn from both the actions of their own past selves and those of other people without seeking to justify or condemn; they would update and move on. There is no tribal council that will exile you if you change your tune or don't defend yourself.

You might be appalled that someone wishes to stop feeling responsibility for their past selves; "but if they don't feel responsibility for their actions, what will prevent them from murdering people, or encourage them to do good?". A rational utilitarian would do good and not do evil because they wish good and non-evil to be done, instead of because of feelings of responsibility that they don't understand.

This argument is a little harder to see and possibly a little less convincing, but again I am convinced that identity and responsibility are inferior to utilitarianism, tho they may have seemed almost terminal.

Justice

Surely justice is a terminal value; it feels so noble to desire it. Again I consider the desire for justice to be a biased heuristic.

in game theory the best solution for iterated prisoners dilemma is tit-for-tat: cooperate and be nice, but punish defectors. Tit-for-tat looks a lot like our instincts for justice, and I've heard that the prisoners dilemma is a simplified analog of many of the situations that came up in the ancestral environment, so I am not surprised that we have an instinct for it.

It's nice that we have a hardware implementation of tit-for-tat, but to the extent that we take it as terminal instead of instrumental-in-some-cases, it will make mistakes. It will work well when individuals might choose to defect from the group for greater personal gain, but what if we discover, for example, that some murders are not calculated defections, but failures of self control caused by a bad upbringing and lack of education. What if we then further discover that there is a two-month training course that has a high success rate of turning murderers into productive members of society. When Dan the Deadbeat kills his girlfriend, and the psychologists tell us he is a candidate for the rehab program, we can demand justice like we feel we ought to at a cost of hundreds of thousands of dollars and a good chunk of Dan's life, or we can run Dan thru the two month training course for a few thousand dollars, transforming him into a good, normal person. People who take punishment of criminals as a terminal value will choose prison for Dan, but people with other interests would say rehab.

One problem with this story is that the two-month murder rehab seems wildly impossible, but so do all of Omega's tricks. I think it's good to stress our theories at the limits, they seem to come out stronger, even for normal cases.

I was feeling skeptical about some people's approach to justice theory when I came up with this one, so I was open to changing my understanding of justice. I am now convinced that justice and punishment instincts are instrumental, and only approximations of the correct game theory and utilitarianism. The problem is, while I was convinced, someone who takes justice as terminal, and is not open to the idea that it might be wrong, is absolutely not convinced. They will say "I don't care if it is more expensive, or that you have come up with something that 'works better', it is our responsibility to the criminal to punish them for their misdeeds.". Part of the reason for this post is that I don't know what to say to this. All I can do is state the argument that convinced me, ask if they have something to protect, and feel like I'm arguing with a rock.

Before anyone who is still with me gets enthusiastic about the idea that knowing a causal history and an instrumentally better way is enough to turn a value into a bias, consider the following:

Love, Friendship, and Flowers

See the gift we give to tomorrow. That post contains plausible histories for why we ended up with nice things like love, friendship, and beauty; and hints that could lead you to 'better' replacements made out of game theory and decision theory.

Unlike the other examples, where I felt a great "Aha!" and decided to use the superior replacements when appropriate, this time I feel scared. I thought I had it all locked out, but I've found some existential angst lurking in the basement.

Love and such seem like something to protect, like I don't care if there are better solutions to the problem they were built to solve; I don't care if game theory and decision theory leads to more optimal replication. If I'm worried that love will go away, then there's no reason I ought to let it, but these are the same arguments as the people who think justice is terminal. What is the difference that makes it right this time?

Worrying and Conclusion

One answer to this riddle is that everyone is right with respect to themselves, and there's nothing we can do about disagreements. There's nothing someone who has one interpretation can say to another to justify their values against some objective standard. By the full power of my current understanding, I'm right, but so is someone who disagrees.

On the other hand, maybe we can do some big million-variable optimization on the contradictory values and heuristics that make up ourselves and come to a reflectively coherent understanding of which are values and which are biases. Maybe none of them have to be biases; it makes sense and seems acceptable that sometimes we will have to go against one of our values for greater gain in another. Maybe I'm asking the wrong question.

I'm confused, what does LW think?

Solution

I was confused about this for a while; is it just something that we have to (Gasp!) agree to disagree about? Do we have to do a big analysis to decide once and for all which are "biases" and which are "values"? My favored solution is to dissolve the distinction between biases and values:

All our neat little mechanisms and heuristics make up our values, but they come on a continuum of importance, and some of them sabotage the rest more than others.

For example, all those nice things like love and beauty seem very important, and usually don't conflict, so they are closer to values.

Things like risk aversion and hindsight bias and such aren't terribly important, but because they prescribe otherwise stupid behavior in the decision theory/epistemological realm, they sabotage the achievement of other bias/values, and are therefore a net negative.

This can work for the high-value things like love and beauty and freedom as well: Say you are designing a machine that will achieve many of your values, being biased towards making it beautiful over functional could sabotage achievement of other values. Being biased against having powerful agents interfering with freedom can prevent you from accepting law or safety.

So debiasing is knowing how and when to override less important "values" for the sake of more important ones, like overriding your aversion to cold calculation to maximize lives saved in a shut up and multiply situation.

A friend asked me to get her Bill O'Reilly's new book Killing Lincoln for Christmas.  I read its reviews on Amazon, and found several that said it wasn't as good as another book about the assassination, Blood on the Moon.  This seemed like a believable conclusion to me.  Killing Lincoln has no footnotes to document any of its claims, and is not in the Ford's Theatre national park service bookstore because the NPS decided it was too historically inaccurate to sell.  Nearly 200 books have been written about the Lincoln assassination, including some by professional Lincoln scholars.  So the odds seemed good that at least one of these was better than a book written by a TV talk show host.

But I was wrong.  To many people, this was not a believable conclusion.

(This is not about the irrationality of Fox network fans.  They are just a useful case study.)

1. Biases, those traits which affect everyone but me

I recently had the opportunity to run an exercise on bias and rationality with a group of (fellow) university students. I wasn't sure it was going to go down well. There's one response that always haunts me when it comes to introducing bias: That's an interesting description of other people but it doesn't describe me.

I can't remember the details (and haven't been able to track them down), but I once read about an experiment on some bias, let's say it was hindsight bias. The research team carried out a standard experiments which showed that the participants were biased as expected. After, they told these participants about hindsight bias. Most of the participants thought this was interesting and probably explained the actions of other people in the experiment but they didn't think it explained their own actions.

So going into the presentation, this is what I was worried about: People thinking these biases were just abstract and didn't affect them.

Then at the end, everyone's comments made it clear that this wasn't the case. They really had realised that these were biases which affected them. The question then is, what led them to reach this conclusion?

2. Living history, living bias

All of the other planets (and the Earth) orbit the Sun. Once upon a time, we didn't believe this: We thought that the these planets (and the Sun) orbited the Earth.

Imagine that you're alive all that time ago, when the balance of evidence has just swung so that it favours the theory that the planets orbit the Sun. However, at the time, you steadfastly insist that they orbit the Earth. Why? Because your father told you it did when you were a child and you always believe things your father told you. Then a friend explains all of the evidence in favour of the theory that the planets orbit the Sun. Eventually, you realise that you were mistaken all along and, at the same time, you realise something else: You realise that it was a mistake not to question a belief just because your father endorsed it.

If you think about history, you learn what beliefs were wrong. If you live history, you learn this and then you also learn what it feels like to mistakenly endorse an incorrect belief. Maybe next time it occurs then, you can avoid making the same error.

In teaching people about biases, I think its best to help students to live biases and not just think about them. That way, they'll know what it feels like to be biased and they'll know that they are biased.

3. Rationality puzzles

One of the best ways to do this, and the technique I used in my presentation, seems to be to use of rationality puzzles. Basically, these are puzzles where the majority of respondents tend to reason in a biased or fallacious way. Run a few of these puzzles and most students will reason incorrectly in at least one of them. This means  them a chance to experience being biased. If lessons focused on an abstract presentation biases instead, the student would think about the bias but not live it in the same way.

So on example rationality puzzle is the 2, 4, 6 task. When I ran this exercise for my presentation, I broke the group up into pairs and made one member of each pair the questioner and the other the respondent.

The respondent was given a slip of paper containing a number rule written upon it. This was a rule that a sequence of three numbers could either meet or fail to meet. I won't mention what the rule was yet, to give those who haven't come across the puzzle a chance to think about how they would proceed.

The questioner's job was to guess this rule. They were given one clue: The sequence 2, 4, 6 met the rule. The questioner was then allowed to ask whether other three number sequences met the rule and the respondent would let them know if it did. The questioner could ask about as many sequences as they wanted to and when they were confident they were to write their guess down (I limited the exercise to five minutes for practical purposes and everyone had written down an answer by then).

The answer was: Any three numbers in ascending order. No students in the group got the right answer.

I then used the exercise to explain a bias called positive bias. First, I noted that only 21% of respondents reached the right answer to this scenario. Then I pointed out that the interesting point isn't this figure but rather why so few people reach the right answer. Specifically, people think to test positive, rather than negative, cases. In other words, they're more likely to test cases that their theory predicts will occur (in this case, those that get a yes answer) then cases that their theory predicts won't. So if someone's initial theory was that the rule was, "three numbers, each two higher than the previous one" then they might test "10, 12, 14" as this is a positive case for their theory. On the other hand, they probably wouldn't test "10, 14, 12" or "10, 13, 14" as these are negative cases for their prediction of the rule.

This demonstrates positive bias - the bias toward thinking to test positive, rather then negative, cases for their theory (see here for previous discussion of the 2, 4, 6 task on Less Wrong).

Puzzles like this allow the student to live the bias and not just consider it on an abstract level.

4. Conclusion

In teaching people about biases we should be trying to make them live biases, rather than just thinking about them. Rationality puzzles offer one of the best ways to achieve this.

Of course, for any individual puzzle, some people will get the right answer. With the 2, 4, 6 puzzle in particular, a number of us have found people perform better on this task in casual, rather than formal, settings. The best way to deal with this is to present a series of puzzles that reveal a variety of different biases. Most people will reach the wrong answer in at least one puzzle.

5. More rationality puzzles

Bill the accountant and the conjunction fallacy

Wason Selection Task

World War II and Selection Effects (not quite a puzzle yet, but it feels like it could be made into one)

During the sessions at the 2011 rationality minicamp, we learned that some of our biases can be used constructively, rather than just tolerated and avoided.

For example, in an excellent article discussing intuitions and the way they are formed, psychologist Robin Hogarth recommends that "if people want to shape their intuitions, [they should] make conscious efforts to inhabit environments that expose them to the experiences and information that form the intuitions that they want."

Another example: Carl Shulman remarked that due to the availability heuristic we anticipate car crashes with frequencies determined by how many people we know of or have heard about who have gotten into one. So if you don't fear car crashes but you want to acquire a more accurate level of concern about driving, you could seek out news or footage of car crashes. Video footage may work best, because experiential data unconsciously inform our intuitions more effectively than, say, written data.

This fact may lie behind many effective strategies for getting your brain to do what you want it to do:

• Establishing 'pull' motivation' works best with strong visualization, and is reinforced upon experiencing the completion of the task.
• Rejection therapy, which many of us minicampers found helpful and effective. The point is to ask people for things they will probably deny you, which trains your body to realize that nothing bad happens when you are rejected. After a time, this improves social confidence.
• As looking glass self theory states,¹ we are shaped by how others see us. This is largely due to the experience of having people react to us in certain ways.

In The Mystery of the Haunted Rationalist we see a someone whose stated beliefs don't match their anticipations. Now we can actually use the brain's machinery to get it to do what we want it to: alieve that ghosts aren't real or dangerous. One method would be for our ghost stricken friend to get people to tell her detailed stories about pleasant nights they spent in haunted houses (complete with spooky details) where nothing bad happened. Alternatively, she could read some books or watch some videos with similar content. Best of all would be if she spent a month living in a 'haunted' house, perhaps after doing some of the other things to soothe her nerves. There are many who will attest that eventually one 'gets used to' the scary noises and frightening atmosphere of an old house, and ceases to be scared when sleeping in similar houses.

I attribute the effectiveness of these tactics mostly to successful persuasion of the non-conscious brain using experiential data.

So, it seems we have a (potentially very powerful) new technique to add to our rationalist arsenal. To summarize:

1. Find something you want to alieve.
2. Determine what experiences that alief should cause you to anticipate.
3. Have those experiences, by proxy if necessary, artificial or not.
4. Test whether you now anticipate what you want to.
5. If the test reveals progress, but not enough, repeat.

Examples:

• Want to alieve that boxing is dangerous²? Watch some footage of boxers being punched painfully in the face, and ask a good boxer to win a fight against you in a painful but non-damaging manner. Now are you reluctant to box someone you have a good chance of beating?
• Want to alieve that driving is dangerous? Watch footage of lots of car crashes, see Red Asphalt, and take a class from professional stunt drivers on how to crash safely. Now are you more reluctant to drive?
• Want to alieve that flying is not very dangerous? Get a pilot's view of a flight, and pay attention to how boring it is. Sit next to a pilot while they undergo a very realistic flight simulation that covers many possible accidents, and watch them successfully navigate each scenario. Now are you more willing to fly?
• Want to alieve snakes are generally not dangerous? Watch videos of safe snake interactions. Watch a pet store employee deal with a snake safely. Play with a snake under supervision without incident. Now do you exhibit less fear when encountering a snake?
• Want to alieve you are part of the Less Wrong community? Interact with other community members as though you are one, attend meetups, make friends in the community. Now do you empathize more strongly with contributors on Less Wrong than with those elsewhere on the internet?

It can be annoying when our unconsciously moderated aliefs don't match our rationality-influenced beliefs, but luckily our aliefs can be trained.

¹ Thanks to Hugh Ristik for talking about this at minicamp.

² Credit for this example goes to Brandon Reinhart.

Special thanks to Luke for all the help

This is a story about an odd fact about capital project decision making in engineering I noticed and how it might be related to cognitive biases

Background

Oddly high discount rates

Overestimating benefits, underestimating costs

To understand why MARRs are so high in chemical engineering, I tried to find research on the question and talked to experienced engineers and managers. Unfortunately, I never did find research on this topic (let me know if you know of relevant research). I talked to several managers and engineers and the most common answer I got was that investors are short sighted and primarily interested in short run profits. I didn't find this answer very plausible. Later, I met an engineer in charge of reviewing project evaluations made by other engineers in order to decide which projects would be approved. His explanation was that engineers usually overestimate the benefits of a project under consideration and underestimate the costs and that they gave engineers high MARRs in order to counterbalance this. I asked him why they didn't just apply a scaling factor to the costs and benefits and he explained that they did this a little bit, but engineers respond to this by inflating benefits and deflating costs even more! I later met another engineer who talked about doing exactly that; adjusting estimated costs down and estimated benefits up because the process evaluating projects did the reverse. 

One thing to note is that if engineers overestimate benefits, underestimate costs uniformly over time, then a high MARR will make projects which pay off in the short term artificially attractive (which is why I asked about using a scaling factor instead of a large interest rate). On the other hand, if engineers make more biased predictions about costs and benefits the further out they are in time (for example, if they tend to overestimate the productive life of equipment), then a high MARR is a more appropriate remedy.

Possible explanations

There are a couple of reasons why engineers might end to overestimate the benefits and underestimate the costs of projects. Any number of these may contribute. I suspect cognitive bias is a significant contributor. 

1. Confirmation bias suggests engineers will tend to overestimate the benefits and underestimate the costs of projects they initially think are good ideas. The head project engineer I spoke with described a common mindset thus, 
   'And this is why we tend to focus on the goodness and diminish the badness of projects.  We know they are good so all we need to do is prove it to get the approvers bought in. Then the project approvers over time notice that these projects returns are lower than expected so they say, “Let’s raise the bar.”  But guess what?  The bar never rises.  Why?  Because we still "know" what the good projects are and all we need to do is prove they are good.'
2. The planning fallacy suggests engineers will underestimate completion times and costs. 
3. Overconfidence suggests engineers will underestimate costs even when explicitly accounting for uncertainty.
4. Bad incentives: engineers may often be rewarded for spearheading projects and not punished commensurately if the project is not beneficial so that they often expect to be rewarded for spearheading a project even if they don't expect it to be a success.

Anyway I started asking our cost coordinator about predicted schedule and she is by far more accurate than the engineers with how long it takes to do a project. That has led me to think that an independent review would be a good step in project returns. Unfortunately, I have not noticed her to be any better on predicting project performance than the engineers. 

On adjusting predictions based on a track record

The problem with predicting a project will take longer than expected based on experience does not help because managers (usually engineers) want to know "why" so the can "fix it."

You are not a Bayesian homunculus whose reasoning is 'corrupted' by cognitive biases.

You just are cognitive biases.

You just are attribution substitution heuristics, evolved intuitions, and unconscious learning. These make up the 'elephant' of your mind, and atop them rides a tiny 'deliberative thinking' module that only rarely exerts itself, and almost never according to normatively correct reasoning.

You do not have the robust character you think you have, but instead are blown about by the winds of circumstance.

You do not have much cognitive access to your motivations. You are not Aristotle's 'rational animal.' You are Gazzaniga's rationalizing animal. Most of the time, your unconscious makes a decision, and then you become consciously aware of an intention to act, and then your brain invents a rationalization for the motivations behind your actions.

If an 'agent' is something that makes choices so as to maximize the fulfillment of explicit desires, given explicit beliefs, then few humans are very 'agenty' at all. You may be agenty when you guide a piece of chocolate into your mouth, but you are not very agenty when you navigate the world on a broader scale. On the scale of days or weeks, your actions result from a kludge of evolved mechanisms that are often function-specific and maladapted to your current environment. You are an adaptation-executor, not a fitness-maximizer.

Agency is rare but powerful. Homo economicus is a myth, but imagine what one of them could do if such a thing existed: a real agent with the power to reliably do things it believed would fulfill its desires. It could change its diet, work out each morning, and maximize its health and physical attractiveness. It could learn and practice body language, fashion, salesmanship, seduction, the laws of money, and domain-specific skills and win in every sphere of life without constant defeat by human hangups. It could learn networking and influence and persuasion and have large-scale effects on societies, cultures, and nations.

Even a little bit of agenty-ness will have some lasting historical impact. Think of Benjamin Franklin, Teddy Roosevelt, Bill Clinton, or Tim Ferris. Imagine what you could do if you were just a bit more agenty. That's what training in instrumental rationality is all about: transcending your kludginess to attain a bit more agenty-ness.

And, imagine what an agent could do without the limits of human hardware or software. Now that would really be something.

(This post was inspired by some conversations with Michael Vassar.)