Reducing a probability to a binary decision clearly loses information. You can't argue with that.

No, I can't. But I can argue that no reduction occurs.

To be fair, I see your point in the case of politicians or people who are otherwise indisposed to changing their minds: once they say they believe something there are costs to subsequently saying they don't. That effectively makes it a binary distinction for them.

However, for people not in such situations, if I hear they believe X, that gives me new information about their internal state (namely, that they give X something like 55-85% chance of being the case). This doesn't lose information. I think this comprises most uses of believe/disbelieve.

So I would argue that it's not the believe/disbelieve distinction that is the problem; it's the feedback loop that results from us not letting people change their minds that causes issues to be forced into yes/no terms, combined with the need for politicians/public figures to get their thought to fit into a soundbite. I don't see how using other terms will ameliorate either of those problems.