I agree that attitudes have been internalized that make ratings skewed. I will add, however, that the rating for "mean performance" on a scale is context-dependent. Examples off the top of my head: 80% is an okay-ish grade in most US schools, but 50% is atrocious. Contrast this with attractiveness on a 0-10 scale: an 8 is a superior specimen, whereas a 5 is average.
With customer service in particular, I can attest to feeling a lot of pressure to give a high rating (if I must rate) because I don't want an employee punished as a result. Heck, this goes beyond ratings. I would be a dishonest juror if I thought the defendant were guilty of a minor crime but they were facing an extreme sentence.
I would argue that inconsistency of preferences isn't necessarily a sign of irrationality. Come to think of it, it may hinge greatly on how you frame the preference.
Consider changing tastes. As a child, I preferred some sweets to savory items, and those preferences reversed as I aged. Is that irrational? No and, indeed, you needn't even view it as a preference reversal. The preference "I prefer to eat what tastes good to me" has remained unchanged, after all. Is my sense of taste itself a preference? It seems like this would devolve into semantics quickly.
My reluctance to characterize preferences as rational or irrational is that I see these as prescriptive terms. But you can't prescribe preferences. You either have them or you don't. Only decision rules are chosen.
I've read a good chunk of Eliezer's paper on TDT, and it's in that context that I am interpreting reflection. Forgive me if I misunderstand some of it; it's new to me.
TDT is motivated by requiring a decision rule that is consistent under reflection. It doesn't seem to pass judgment on preferences themselves, only on how actions ought to be chosen given preferences. Am I mistaken here?
Perhaps I should have been clearer with Voldemort's "revealed" preferences. JKR writes him as a fairly simple character and I did take for granted that what we saw was what we got. I agree that in general actions aren't indicative of beliefs.
EDIT: Ah, there is an exception. Eliezer is quite critical in the paper of preferring a decision rule for its own sake.
There is a distinction people often fail to make, which is commonly seen in analyses of fictional characters' actions but also those of real people. It is the distinction between behaving irrationally and having extreme preferences.
If we look at actions and preferences the way decision theorists do, it is clear that preferences cannot be irrational. Indeed, rationality is defined as tendency toward preference-satisfaction. To say preferences are irrational is to say that someone's tastes can be objectively wrong.
Example: Voldemort is very stubborn in JKR's Harry Potter. He could have easily arranged for a minion to kill Harry, but he didn't, and this is decried as irrational. Or even more to the point, he could have been immortal if only he hid in a cave somewhere and didn't bother anyone.
But that is ignoring Voldemort's revealed preference relation and just treating survival as his chief end. What is the point of selling your soul to become the most powerful lich of all time so you can live as a hermit? That would be irrational, as it would neglect Voldemort's preferences.
I think this is closely related to the more colloquial concept of "necessary evils". I always felt the term was a bit of a misnomer--we feel they are evils, I suspect, because their necessity is questionable. Actually necessary things aren't assigned moral value, because that would be pointless. You can't prescribe behavior that is impossible (to paraphrase Kant).
As a recent example, someone argued that school bullying is a necessary evil because bullying in the adult world is inevitable and the schoolyard version is preparation. In that case it seems there was a sort of "all-or-nothing" fallacy, i.e., if we can't eliminate it, we might as well not even mitigate it.
When I first read about Newcomb's Problem, I will admit that it struck me as artificial at first. But not unfamiliar! Similar dilemmas seem common in film and television.
For example, consider Disney's Hercules. Hercules spends the entire movie trying to regain his status as a god. He is told that he must become a hero to do so, so of course he sets out doing what seem to be heroic things. In the end he succeeds by jumping into the Styx to save Meg despite being told he would die. Heroism in his world evidently requires irrationality!
While it isn't identical to Newcomb's Problem, there is the same theme of "just ignore the apparent causal chain and it will all work out". There are countless other instances: let the girl go and you'll end up with the girl, or admit to cheating in that contest and you'll win the prize, etc. The message is that denying ourselves is virtuous, creating Newcomblike scenarios.
I've always had philosophical leanings, so I find myself asking often what decision theory sets out to do, even as I grapple with a concrete mathematical application. This seems important to me if I want a realistic model of an actual decision an agent may face. My concerns keep returning to utility and what it represents.
Utility is used as a measure for many things: wealth, usefulness, satisfaction, happiness, scoring in games, etc. Our treatment of it suggests that what it represents doesn't matter--that the default aim of a decision theory should be to maximize an objective function, and we just call that function the utility function. It doesn't seem to me that this is always obvious.
One may object that the VNM Utility Theorem assures this is so. But VNM (at least the version with which I am familiar) covers only simple decision problems. It would be faster to list scenarios it can handle, but let's summarize what it doesn't:
It has nothing to say when there are infinite outcomes, or when the timing of utility gains matters (auxiliary discounting functions must be introduced, their motivation varied). It doesn't address apples-to-oranges comparisons, because everything must by the end be possible to convert to utils. It offers no insight into how the weights guaranteed by the continuity axiom can be calculated for an agent, so you can't construct their utility function without already knowing all of their preferences (which is what you were trying to infer by using the utility function!). It doesn't enable comparisons between agents, so it isn't a basis for a social choice theory.
The result is that utility ends up being the drawer of miscellaneous items we cram with stuff whose proper place isn't yet known; a black box that produces the result we want in that context. The limitations are mostly ignored. We use unbounded utility functions defined on unbounded domains whose growth rates are chosen for convenience, we discount their future values as if we will live forever, and we pretend every combination of assets we may acquire will fit into their domains.
As an example, we may try to investigate why people are inclined to play the lottery despite being risk-averse in other respects. A common explanation is that people overweight small probabilities with extreme outcomes. But it is also observed that some people simply enjoy the thrill of taking a risk. So do you use a weighting function or do you use a convex utility function? It isn't clear at all, partly because the utility of having money and the utility of having a thrill don't seem to be comparable. They certainly don't feel comparable.
To conclude, the mathematical convenience of turning decision-making into optimization shouldn't seduce us into being lazy like this.
It occurred to me today that the VNM utility functions model preferences concerning income rather than wealth in general.
Consider the continuity axiom, for example. This axiom seems to imply that a rational agent would be willing to gamble their entire life savings for an extra dollar provided that the probability of losing is small enough. Barring the possibility of charity, going broke is tantamount to death, since it costs money to make money. It seems reasonable to me that a rational agent would treat their own death as infinitely bad. Under this assumption no probability of losing is small enough.
This criticism doesn't apply if lotteries are only allowed positive payouts, of course, but no such assumption is ever made. This is what I mean when I say that the axioms describe preferred income streams rather than wealth levels. The obvious fix is to add a parameter for current wealth, but I'm unsure if a result will follow that is analogous to the VNM Utility Theorem.