Previously I'd been thinking in terms of a more general agent, which needn't use a concept of utility and whose performance relative to an objective is found in retrospect.

It doesn't need to use utility explicitly. It's just whatever objective it tends to gravitate towards.

I'm not entirely sure what you're saying in the rest of the comment.

The reason I'm talking about "expected value" is that an optimizer must be able to work in a variety of environments. This is equivalent to talking about a probability distribution of environments.