I think I've found the core of our disagreement. I want an algorithm that considers all possible paths through time. It decides on a set of actions, not just for the current time step, but for all possible future time steps.

So, I think you might be interested in UDT. (I'm not sure what the current best reference for that is.) I think that this requires actual omniscience, and so is not a good place to look for decision algorithms.

(Though I should add that typically utilities are defined over world-histories, and so any decision algorithm typically identifies classes of 'equivalent' actions, i.e. acknowledges that this is a thing that needs to be accepted somehow.)