One relevant idea is that there is a duality between assigning utilities to actions (or equivalently, being able to pick your favorite option out of a probabilistic mix of actions), and assigning utilities to outcomes. Acting consistently in one way implies that you are also acting consistently in the other.

Since humans are much better at picking actions than we are at evaluating entire world-states, this is pretty handy (though it comes nowhere near solving the entire problem). Paul Christiano has a writeup of what a naive-ish application of this idea would look like here.