We are inclined to use expected return when we should use expected utility
A well-known point that goes back to Bernoulli and the very dawn of the expected utility formalism - except that conventionally this is illustrated by explaining why you should not buy lottery tickets that seem to have a positive expected return.
Your main post is rather an attempt to defend behavior as "rational" which on the surface appears to be "irrational". This may make sense when you're looking at a hedge-fund trader who seemingly lost huge amounts of money through "stupid" Black Swan trades, and yet who is, in fact, living comfortably in a mansion based on prior payouts. The fact that he's living in a mansion gives us good reason to suspect that his actions are not so "stupid" as they seemed.
The case for suspecting the hidden rationality of crack users is not so clear-cut. Is it really the case that before ever taking that first hit, the original potential drug user, looking over their entire futures with a clear eye free of such biases as the Peak-End Rule, would still choose the crack-user future?
People in general are crazy. We are, for example, hyperbolic discounters. Sometimes the different behavior of "unusual" people stems not from any added stupidity, but from added motives given their situation. Crack users are not mutants. Their baseline level of happiness is lower, they are more desperate for change, their life expectancy is short; none of this is stupidity per se. But like all humans they are still hyperbolic discounters who will value short-term pleasure over the long-term consequences to their future self. To suppose that being in poverty they must also stop being hyperbolic discounters, so that their final decision is inhumanly flawless and we can praise their hidden rationality, is a failure mode that we might call Pretending To Be An Economist.
Don't blame the readers, you killed your own post: humans in general are flawed beings, and buying lottery tickets is an illustration thereof. Trying to make it come out as an amazing counterintuitive demonstration of rationality was your mistake. To illustrate the difference between expected return and expected utility, you should have picked some example whose final answer added up to normality (like "Don't play the Martingale") rather than abnormality ("Buy lottery tickets now!").
like all humans they are still hyperbolic discounters who will value short-term pleasure over the long-term consequences to their future self.
Just a nitpick: As Carl Shulman observed, this is not irrational. It's just a different discounting function than yours.
Trying to make it come out as an amazing counterintuitive demonstration of rationality was your mistake.
Really? So you found a mistake in anything that I wrote? I must have missed it. All I see is you presenting just-so arguments along the lines of either "C causes people to play th...
The lottery came up in a recent comment, with the claim that the expected return is negative - and the implicit conclusion that it's irrational to play the lottery. So I will explain why this is not the case.
It's convenient to reason using units of equivalent value. Dollars, for instance. A utility function u(U) maps some bag of goods U (which might be dollars) into a value or ranking. In general, u(kn) / u(n) < k. This is because a utility function is (typically) defined in terms of marginal utility. The marginal utility to you of your first dollar is much greater than the marginal utility to you of your 1,000,000th dollar. It increases the possible actions available to you much more than your 1,000,000th dollar does.
Utility functions are sigmoidal. A serviceable utility function over one dimension might be u(U) = k * ([1 / (1 + e-U)] - .5). It's steep around U=0, and shallow for U >> 0 and U << 0.
Sounds like I'm making a dry, academic mathematical point, doesn't it? But it's not academic. It's crucial. Because neglecting this point leads us to make elementary errors such as asserting that it isn't rational to play the lottery or become addicted to crack cocaine.
For someone with $ << 0, the marginal utility of $5 to them is minimal. They're probably never going to get out of debt; someone has a lien on their income and it's going to be taken from them anyway; and if they're $5 richer it might mean they'll lose $4 in government benefits. It can be perfectly reasonable, in terms of expected utility, for them to play the lottery.
Not in terms of expected dollars. Dollars are the input to the utility function.
Rationally, you might expect that u(U) = 0 for all U < 0. Because you can always kill yourself. Once your life is so bad that you'd like to kill yourself, it could make perfect sense to play the lottery, if you thought that winning it would help. Or to take crack cocaine, if it gives you a few short intervals over the next year that are worth living.
Why is this important?
Because we look at poor folks playing the lottery, and taking crack cocaine, and we laugh at them and say, Those fools don't deserve our help if they're going to make such stupid decisions.
When in reality, some of them may be making <EDITED> much more rational decisions than we think. </EDITED>
If that doesn't give you a chill, you don't understand.
(I changed the penultimate line in response to numerous comments indicating that the commenters reserve the word "rational" for the unobtainable goal of perfect utility maximization. I note that such a definition defines itself into being irrational, since it is almost certainly not the best possible definition.)