Every year, a handful of small children, and visiting tourists, and people chasing stray pets, and people who get lost in the dark, accidentally wander up into the monster's cave and into its mouth.
How big does this number have to be before it's worth whipping up the village to kill the monster by all walking in together?
What if the monster has only recently settled in the mountain, so no wayward children have been eaten yet. The town holds a vote: either we can commit to drilling it into our children, and our stumbling drunks, and our visiting tourists to never, ever, ever, ever go up the mountain, or we can go up as one and get rid of the monster now. How do you think most people will vote?
"It seems like everyone will pick red pill"
-- but in the actual poll, they didn't! So, something has gone wrong in the reasoning here, even if there is some normative sense in which it should work.
My understanding is that 70% of Twitter respondents chose "blue", and I'd expect the Twitter poll was both seen by, and responded to, at higher rates by people with an interest in game theory and related topics, i.e. the people more likely to understand the principles necessary to arrive at "red" as an answer.
Obviously a Twitter poll isn't the real life situation, but a) it is far from clear that "blue"s are committing suicide and b) if you find yourself arguing that a supermajority of humanity is below your intellectual threshold of concern, I think that's a good sign in general to reflect on how much you really mean that.
I liked this post a lot!
I can't be the only person who read this and thought of the three major varieties of moral theory: Decision Theory roughly corresponding to naive Act Utilitarianism; Policy Theory roughly matching Rule Utilitarianism or Deontology; and Agent Theory being sort of Virtue Ethics-y.
One thing that would be interesting, is to think about what features of the world would lead us to prefer different levels of analysis: my impression is, the move from CDT to EDT/FDT/UDT etc. is driven by Newcomb-like problems and game theory-ish situations, where your decision process can affect the outcomes you will face; usually this seems to arise in situations where there are agents who have comparable or greater computational power than you. But I'm not totally sure I understand what circumstances would make the Agent Theoretic level more appealing than the Policy Theoretic level.
In the post you appear skeptical of the value of going beyond Decision Theory; while some of the examples used to push FDT and so forth seem at least a little contrived, I think the game-theory-ish stuff feels somewhat compelling to me; and even the contrived examples seem to be telling us something about the value of being able to pre-commit to an action ahead of time.
My final thought: is there any reason to stop at Agent Theory? Are there a class of problems that would push one to consider (what we might call) Society Theories, at a level above individual agents?