fubarobfusco comments on SotW: Check Consequentialism - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (311)
Cleverness-related failure mode (that actually came up in the trial unit):
One shouldn't try too hard to rescue non-consequentialist reasons. This probably has to be emphasized especially with new audiences who associate "rationality" to Spock and university professors, or audiences who've studied pre-behavioral economics, and who think they score extra points if they come up with amazingly clever ways to rescue bad ideas.
Any decision-making algorithm, no matter how stupid, can be made to look like expected utility maximization through the transform "Assign infinite negative utility to departing from decision algorithm X". This in essence is what somebody is doing when they say, "Aha! But if I stop my PhD program now, I'll have the negative consequence of having abandoned a sunk cost!" (Sometimes I feel like hitting people with a wooden stick when they do this, but that act just expresses an emotion rather than having any discernible positive consequences.) This is Cleverly Failing to Get the Point if "not wanting to abandon a sunk cost", i.e., the counterintuitive feel of departing from the brain's previous decision algorithm, is treated as an overriding consideration, i.e., an infinite negative utility.
It's a legitimate future consequence only if the person says, "The sense of having abandoned a sunk cost will make me feel sick to my stomach for around three days, after which I would start to adjust and adapt a la the hedonic treadmill". In this case they have weighed the intensity and the duration of the future hedonic consequence, rather than treating it as an instantaneous infinite negative penalty, and are now ready to trade that off against other and probably larger considerations like the total amount of work required to get a PhD.
One of the other models people have for the rationalizing sort of "rationality" is that of lawyers.
Lawyers are very good at logic — the LSAT, the entrance examination for U.S. law schools, leans heavily on logic puzzles — but the whole point of being a trial or appeals lawyer is to come up with clever (and socially respectable) arguments for whatever position your client may have at the moment.
This extends past real-world lawyerhood. The tabletop role-playing game crowd have the expression "rules lawyer" for a person who comes up with clever arguments for why their character should get away with whatever they want to at the moment.
Indeed I think this is the central problem with the way most people use their powers of reasoning. (It even has a name: "the argumentative theory of reason".) They start with a conclusion, and work backwards to find rational (or at least rational-sounding) ways of supporting that conclusion.
We all do this automatically; it may be the very thing our brains evolved to do. We have to work very hard to get ourselves to do the opposite, start with evidence and use reasoning based on the evidence to decide on our conclusion. I'd say most scientists manage to do this right maybe half the time, and most laypeople almost never manage it.