I am definitely not saying "trust your current probability estimates".

What I'm saying is that probability should reflect reality as closely as possible, whereas utility should reflect preferences as closely as possible.

Modifying the preference function in an ad-hoc way to get the right behavior is a bad idea, but modifying our expectation about how reality actually might be is even worse. The probability function should be modified exclusively in response to considerations about how reality might be. The utility function should be modified exclusively in response to considerations about our preferences.