A few years ago I listened to a fascinating podcast interview featuring former Democratic presidential candidates Andrew Yang and Marianne Williamson. They agreed that politics is a mess and politicians are constantly doing bad things that harm the people they are supposed to serve. But they couldn’t agree on how bad that made the politicians as people.

Yang wanted to view the politicians as normal people responding to bad incentives, but Williamson wanted to call them evil for failing to exercise courage in the face of these bad incentives.

Morally, the notion that you can’t blame people when they are following incentives is akin to the “just following orders” excuse that Nazis tried to use at the Nuremberg trials. But what’s the alternative? In practice, we can’t and don’t expect people to always do the right thing even when everyone else around them isn’t.

There’s a point at which “everyone else is doing it” really is an acceptable excuse, because everyone else really is doing it, and not doing so puts you at a significant and unfair disadvantage. But there are also absolutes, where this excuse is never acceptable -- things like genocide.

Most of the time it’s something more complicated: Doing the right thing means being a bit better on the margin. If everyone else in your class is cheating and using AI to do their homework, it could mean living by a principle where you only use AI for parts of the assignment that are clearly useless busy work -- and letting this be known.

A colleague recently said something that sums it up nicely: “A person’s moral strength is exactly their ability to resist bad incentives.” *(paraphrased)

Are the incentives in the room with us right now?

But this post is not ultimately about ethics. I want to ask a more basic question: what do we really mean when we say someone is “following incentives”?

I think most of the time, it’s not at all clear that it’s true in a literal sense. My take is that “apparent short-term incentive-like vibes” might be a better description for what they are actually following. Things that have more “incentive-y” vibes are those that are more associated with selfishness and vices like greed. Money: incentive!! Admiration of your peers: incentive???

I think often what “incentive” is really referring to is more like a feeling of competitive pressure, or a belief that “if I don’t do this, someone else will, and then I’ll be a sucker and a failure.”

When I was in grad school, the people around me generally felt a lot of pressure to publish a lot of papers. But the people who really stood out and succeeded often were more focused on making real contributions that were actually valuable to others in the field, even if it meant publishing less. The apparent incentive to publish constantly was almost exactly backwards!

Often people do actually get short-term benefits for doing something that’s not in their long-term interest. So it might be a case of following short-term incentives in particular (and potentially being confused about what’s good in the longer term). Publishing more often made it seem like a student was more productive or impressive in the short term, and unlocked travel funding to go to conferences. But what you really want to advance your career is to become known throughout the field for something you did; no amount of mediocre publications would ever get you there.

“One-shot thinking” is commonly misapplied

A special case of following short-term incentives, which is maybe the most puzzlingly common, is one-shot thinking. You’ve likely been in a situation where someone says something like: “Of course the other side won’t cooperate -- there’s no incentive to! So we can’t either!” and people listening treat this as the sophisticated, hard-nosed take. But failing to cooperate leaves value on the table. And when you have the chance to negotiate, build trust, and/or set-up enforcement mechanisms to make sure all parties follow through on a commitment, it seems like you should at least consider trying to find a way to cooperate. The basic mistake here is treating an interaction as an isolated “one-shot” game, after which everyone walks away and never interacts in any way ever again. Acting like a situation is “one-shot” when it’s not isn’t sophisticated, it’s stupid.

This also means that saying you did something bad because of “the incentives” doesn’t work as an excuse. You’ve done the thing. The “one-shot” part is over. You are now in the position of being judged for your previous behavior, but treating something as a one-shot game is only valid if you will never be in a position to be judged for your behavior during the game.

Applying these insights to AI is left as an exercise for the reader.