Well, also part of it is that for most people, utility isn't linear in money.

Yeah. This assumption of linearity is annoyingly common; I wish more people were aware of the problems with it when contructing their various thought experiments. Not just with money, either.