One thing that tends to bug me about discourse round here is the notion that maximising one's expectation is the be all and end all of decisions one should make. Given any problem, one should look at it, and pick the course that maximising one's expectation. This usually ignores two major problems: what if my utility is non-linear, and what about risk aversion?

Let’s take an example: I bump into Omega, who offers me a choice: I can take a certain 1 unit of utility, or have a 1 in 10 million chance of getting 1 billion utility. The naive expectation maximiser will take that chance: after all, their expectation will be 100 units of utility, which is much better than a measly one! In all likelihood, our maximiser will walk away with nothing. It's certainly true that if this is repeated enough then we would expect our maximiser to be doing better... but a simple calculation reveals that it will have to occur around 7 million times for our maximiser to have a greater than 0.5 chance of having actually won (once or more times).

This is a problem with tiny probabilities and large utilities: some justifications of cryonics have run along the lines of a Pascal’s wager, where a small monetary investment gives one a massive utility, so large, in fact, that no matter how small the probability of cryonics it makes sense to invest. But if the probability becomes small enough, then I've probably just wasted a lot of money. After all, we only get to live once (Note that I am aware that some people have much higher probability estimates for cryonics, which is fine: I'm addressing those who do not).

Without multiple repetition, risk aversion is, I would argue, an extremely sensible strategy for utility maximisation. Of course if one believes that one will be faced with a similar choice multiple times, then one can revert back to utility maximisation. As to when one should do this, I would probably encourage one to revert when the number of repetitions, N, is large enough so that the probability of an event occurring at least once has passed some threshold, p, decided by the user. Certainly p should probably be higher than 0.5.

Lets now take another example: I am on Deal or No Deal, and there are three boxes left: $100000, $25000 and $.01. The banker has just given me a deal of $20000 (no doubt to much audience booing). Should I take that? Expected gains maximisation says certainly not! After all my expectation is more than double that offer! Risk aversion could be applied here, but I've actually got a good chance (0.66) of beating the bankers offer, so maybe its worth it? Except... well if I had $20000 dollars there’s quite a lot I could do with that- perhaps its enough to get a mortgage on a house, or pay for that dream holiday I've always wanted. Sure, $100000 would be nice, but 1/3 of the time I'm going home with nothing- I've effectively lost that $20000 I wanted, and 1/3 of the time I'm only getting $5000 more, which isn't going to make a huge difference to me.

Different amounts of money are valued very differently. The first million one earns will be quite a bit more important than the second million, and so on. Again, this is a perfectly reasonable criteria to have: the first amount of money lets us pay for things we need, the second for things we want, for a crude comparison. Yes, the banker is going to offer us below our expected gains, but his expectation is based on us valuing totals all the same. If that first $20,000 is what I really want, the utility of higher sums may be much smaller than one might consider. So again, naively maximising expectation could leave one disappointed.

Obviously defining one's nonlinearity may be difficult. One of the advantages of trying to work out ones expected utility is it allows us to overwrite our brains, which don't necessarily always think very clearly about our expected gains, and allow us to do much better overall. But if we don't define our function carefully enough, then we are cheating ourselves. While I am not claiming that instinct is always correct about what will make us happy in the long run, using to simple a method to try and overwrite ourselves will not help.

Recently there has been a couple of articles in the discussion page asking whether rationalists should do action A. Now such questions are not uninteresting, but by saying "rationalist" they are poorly phrased.

The rational decision at any time is the decision, given a human with a specific utility function B, and information C, should make to maximise B, given their knowledge (and knowledge about their knowledge) of C. It's not a decision a rationalist should make, it's a decision any human should make. If Omega popped into existence and carefully explained why action A is the best thing for this human to do given their function B, and their information C, then said human should agree.

The important question is not what a rationalist should do, but what your utility function and current information is. This is a more difficult question. Humans are often wrong about what they want in the long term, and it's questionable how much we should value happiness now over happiness in the future (in particular, I suspect current and future me might disagree on this point). Quantifying our current information is also rather hard- we are going to make bad probability estimates, if we can make them at all, which lead us into incorrect decisions just because we haven't considered the evidence carefully enough.

Why is this an important semantic difference? Well it's important for the cause of refining rationality that we don't get caught with associating the notion of rationality with certain goals. Some rationalists believe that they want to save the world, and the best way to do it is by creating friendly AI. This is because they have certain utility functions, and certain beliefs about the probabilities of the singularity. Not all rationalists have these utility functions. Some just want to have a happy home life, meet someone nice, and raise a family. These are different goals, and they can be helped by rationality, because rationality IS the art of winning. Being able to clearly state ones goals and work out the best way to acheieve them is useful pretty much no matter what those goals are. (pretty much to prevent silly examples here!)

Reading Being a teacher made me think about my experiences tutoring university students. I'm a PhD student, so my teaching currently consists of helping first year undergraduates on problem sheets. I think I'm reasonably good- I try to appreciate what difficulties they are having and anticipate them, by explaining what they are doing means, and approaching the problem in different ways.

One constant frustration I get though is that, having explained a problem to a particular student, the student will give me a blank look, and then mutter "ok". I know what that look means, and will ask "so do you understand that?" "sort of...." "well look at it this way....".

Now some of this may come from me- I'm explaining too fast or in a way they don't understand, and my familiarity with the subject, but I suspect some comes from them. It can be difficult to admit one's ignorance, from my own experience. I, and I suspect others who go on to do university maths, was used to being the best or near the best in school, with "being smart" being part of my core identity, something that made me distinct from my more attractive or more fit peers. Getting to university and realising one is having difficulty with even basic questions can be a knock to ones identity. So I hid my ignorance, and did myself damage. I might lose the respect of a tutor, or even a lecturer by admitting my ignorance, but the alternative is to remain ignorant.

I suspect this is a problem that is common among us all. Its a lot easier to pretend we understand, and sometimes it may help- if we want to impress a potential employer we shouldn't admit ignorance (unless the alternative makes us look more ignorant)- but in general admitting ignorance helps us learn. There is (almost) always someone with more knowledge on a particular subject than you, and a failure to use that resource is a failure of rationality.