The problem is when people decide that they believe / do not believe some proposition P, and then consider only the expected utility of the case where P is true / false.

Agree that this is widespread, and is faulty thinking. And my $.02, which you should feel free to ignore: your main post would be clearer, I think, if you focused more on the math of why this is so: find an example where different actions are appropriate based on the probability, and collapsing the probability into a 1 or 0 forces the choice of an inappropriate action; explain the example thoroughly; and only then name the concept with the labels believe/disbelieve. Hearing them right from the start put me on the wrong trail entirely.

I thought this was a post about language usage, but it's actually a post about how not to do math with probabilities.