As an alternative to probability thresholds and bounded utilities (already mentioned in the comments), you could constrain the epistemic model such that for any state and any candidate action, the probability distribution of utility is light-tailed.

The effect is similar to a probability threshold: the tails of the distribution don't dominate the expectation, but this way it is "softer" and more theoretically principled, since light-tailed distributions, like those in the exponential family, are in a certain sense, "natural".