I agree that it's unclear that it makes sense to talk about humans having utility functions; my use of the term was more a manner of speaking than anything else.

They do have them - in this sense:

It would be convenient if we could show that all O-maximizers have some characteristic behavior pattern, as we do with reward maximizers in Appendix B. We cannot do this, though, because the set of O-maximizers coincides with the set of all agents; any agents can be written in O-maximizer form. To prove this, consider an agent A whose behavior is specied by yk = A(yx<k). Trivially, we can construct an O-maximizer whose utility is 1 if each yn in its interaction history is equal to A(yx<n), and 0 otherwise. This O-maximizer will maximize its utility by behaving as A does at every time n. In this way, any agent can be rewritten as an O-maximizer.