It's possible that the confusion here is (yet again) due to people using the word utility to mean different things. If PJ is using utility to refer to, e.g. an emotional or cognitive state, then he's right that our utility cannot respond directly to the territory. But broader notions of utility, well-being, and preference are possible, and nothing PJ has said is especially relevant to whether they are coherent or not.