It's not obvious to me that Qiaochu would endorse utility functions as a standard for "ideal rationality". I, for one, do not.

Talking about utility functions can be useful if one believes any of the following about ideal rationality, as a concrete example of what one means if nothing else.

1. An ideally rational agent uses one of the standard decision theories (vNM, EDT, CDT, etc.)
2. An ideally rational agent does EU maximization.
3. An ideally rational agent is consequentialist.
4. An ideally rational agent, when evaluating the consequences of its actions, divides up the domain of evaluation into two or more parts, evaluates them separately, and then adds their values together. (For example, for an EU maximizer, the "parts" are possible outcomes or possible world-histories. For a utilitarian, the "parts" are individual persons within each world.)
5. An ideally rational agent has values/preferences that are (or can be) represented by a clearly defined mathematical object.

I guess when you say you don't "endorse utility functions" you mean that you don't endorse 1 or 2. Do you endorse any of the others, and if so what would you use instead of utility functions to illustrate what you mean?