The soft (and hard) failure seems to be a tempting but unnecessary case of pseudo-rationalization.

I'd have called it "the danger of falling in love with your model". The mathematics of having a utility function is far more elegant than what we actually have, a thousand shards of desire that Dutch-book you into working for the propagation of your genes. So people try to work like they have a utility function, and this leaves them open to ordinary human-level exploits since assuming you have a utility function still doesn't work.