The problem goes away if you add finiteness in any of a bunch of different places: restrict agents to only output decisions of bounded length, or to only follow strategies of bounded length, or expected utilities are constrained to finitely many distinct levels. (Making utility a bounded real number doesn't work, but only because there are infinitely many distinct levels close to the bound).

The problem also goes away if you allow agents to output a countable sequence of successively better decisions, and define an optimal sequence as one such that for any possible decision, a decision at least that good appears somewhere in the sequence. This seems like the most promising approach.