Strictly speaking, you don't always need the sums to converge. To choose between two actions you merely need the sign of difference between utilities of two actions, which you can represent with divergent sum. The issue is that it is not clear how to order such sum or if it's sign is even meaningful in any way.