I agree that the true PD never happens in human existence, and that's yet another reason why I'm outraged at using a mathematically flawed decision theory to teach incoming students of rationality that they ought to betray their friends. (C, C) FTW!

(Actually, that would make a nice button.)

But I defend the use of simple models for the sake of understanding problems with mathematical clarity; if you can't model simple hypothetical things correctly, how does it help to try to model complex real things correctly first? In real life, no one is an economic agent; in real life, no laws except basic physics and theorems therefrom have universal force; in real life, an asteroid can always strike at any time; in real life, we can never use Bayesian reasoning... but knowing a bit of math still helps, even if it never applies perfectly above the level of quarks.

Within moral philosophy, at least, there are two related senses in which philosophers’ typical practice of thought-experiments can seem ill-advised:

1. They may deal with situations that are strongly unlike the situations in which we actually need to make decisions. Perhaps you’ll never be faced with a runaway trolley, with decisions concerning 3^^^3 dust specks, or with any decision simple enough that you can easily apply your thinking about trolley problems or dust specks.

2. They may highlight situations that disorient or break our moral intuitions or our notions of value.

   To elaborate a plausible mechanism: The human categories “birds”, “vegetables” (vs. “fruits”, or “herbs”), and “morally right” are all better understood as family resemblance terms (capturing “clusters in thingspace”) than as crisp, explicitly definable, schematic categories that entities do or don’t fall into. Such family resemblance terms arguably gain their meaning, in our heads, from our exposure to many different central examples. Show a person carrots, mushrooms, spinach, and broccoli, with a “yes these are Xes”, and strawberries, cayenne, and rice with a “these aren’t Xes”, and the person will construct the concept “vegetables”. Add in a bunch of borderline cases (“are mustard greens a vegetable or an herb? what exact features point toward and against?”) and the person’s notion of “vegetable” will lose its some of its intuitive “is a category”-ness. If there are enough borderline examples in their example-space, “vegetable” won’t be a cluster for them anymore.

   “Is morally right” may similarly be a cluster formed by seeing what kinds of intra- and inter-personal situations work well, or can be expected to be judged well, and may break or weaken when faced with non-“ecologically valid” thought-experiments.

I spent two years in a graduate philosophy department before leaving academic philosophy to try to reduce existential risks. In my grad philosophy courses, I used to express disdain for dust specks vs. torture type problems, and to claim arguments along the lines of both (1) and (2) for why I should fail to engage such questions. My guess is that (2) was my actual motivation -- I could feel aspects of my moral concern breaking when I considered trolley problems and the like -- and, having not read OB, and tending to believe that arguments were like soldiers, I then argued for (1) as well.

When I left philosophy, though, and started actually thinking about what kind of a large-scale world we want, I was surprised to find that the discussions I'd claimed were inapplicable (with argument (1)) were glaringly applicable. If you’re considering what people shouldn’t tile the light-cone with, or even if you’re just considering aid to Africa, large-scale schematic beliefs about how to navigate tradeoffs are, in fact, a better guide than are folk moral intuitions about what a good friendship looks like. The central examples around which human moral intuitions are built just don’t work well for some of the most important decisions we do in fact need to make.

But despite its inconvenience, (2) may in fact pose a problem, AFAICT.

Many physical models, like gravity, have the nice property of stably approximating reality. Perturbing the positions of planets by one millimeter doesn't explode the Solar System the next second.

The stability of orbits when perturbing the position of planets is a nice property (from our perspective of not wanting to crash into the sun) of the physical system of gravity. The fact that our model of gravity explains this stability is a nice property of the model, just as, if the physical system did not have the stability, not predicting stability that is not there would be a nice property of the model. As Tarski would say, if orbits are stable, we want our model to predict stable orbits, if orbits are unstable, we want our model to predict unstable orbits.

I forget where, but I read a blog post that described these sorts of things as controlled experiments - you want to test one part of your decision apparatus, not have it confounded by all the others.

Also, you are anthropomorphizing the paperclipper AI. It would accept your bargain, but demand not just a handful but as many paperclips as you could be arm-twisted into making - where your pain at the expense and effort is just below where you'd prefer the billion deaths. And then it would exploit your exhausted state to stab you in the back anyway. It's not psychopathic, it's just incrementing a number by expedient means. You can't negotiate with something that won't stay bought.

If you pose someone the Monty Hall Problem, and their response is "It doesn't matter whether I switch doors or not! They're going to move the prize so that I don't end up getting it anyway!" Do you think they've understood the point of the exercise?

I am sympathetic to the sentiment underlying this post, but I would stress that the value of "realism" depends on what you're trying to model, and why. If your purpose is to generate reasonable solutions to non-extreme problems/parlor games, then you can lose your purpose by artificially hardening your models beyond what such parlor games require. But if your purpose is to find generally-applicable decision rules that will be robust to extreme circumstances, then you can lose by failing to harden your models sufficiently.

I don't see the purpose of such thought experiments as being to model reality (we've already got a perfectly good actual reality for that), but to simplify it. Hypothesizing omnipotent beings and superpowers may not seem like simplification, but it is in one key aspect: it reduces the number of variables.

Reality is messy, and while we have to deal with it eventually, it's useful to consider simpler, more comprehensible models, and then gradually introduce complexity once we understand how the simpler system works. So the thought experiments arbitrarily set certain variables (such as predictive ability) to 100% or 0% simply to remove that aspect from consideration.

This does give a fundamentally unrealistic situation, but that's really the point - they are our equivalent of spherical cows. Dealing with all those variables at once is too hard. In the situations where it isn't and we have "real" situations we can fruitfully consider, there's no need for the thought experiment in the first place. Once we can understand the simpler system, we have somewhere to start from once we start adding back in the complexity.

So I'm making a modest proposal. If you invent an interesting decision problem, please, first model it as a parlor game between normal people with stakes of around ten dollars. If the attempt fails, you have acquired a bit of information about your concoction; don't ignore it outright.

Absolutely not. If I want to play parlor games for stakes around ten dollars I'll do that at home. But that's not what I'm here for. Questions such as "How long do you wait before giving up (on the flakey date)?" are simply not best answered by the types of thinking we are trying to train here. They are best answered by embracing the dark side with both hands and following our social instincts and conditioning.

any natural out-of-the-box solution like giving the damn thing some paperclips or bargaining with it would ruin the game.

LOL'd and upvoted.

In fairness, though, "giving the damn thing some paperclips" doesn't work in the long run, even for humans above a certain level of fanaticism. Nonetheless, IAWYC.