"statement x is not currently the case and is probably unfeasible" does in fact mean we shouldn't try to act on it. Maybe we can try to act to make statement x true, but we shouldn't act as if it already is. For a more concrete example, imagine this: "I've never done a backflip. It's not even clear I can do one". We know backflips are possible, and with training you're probably going to be able to do one. But at the time you're making that statement, saying "doesn't mean you shouldn't try" is TERRIBLE advice that could get you a broken neck.
Firstly, that's kind of an uncharitable reading. If I said "I'm going to try and pass an exam" you'd naturally understand me as planning to do the requisite work first. "Backflip" just pattern-matches to 'the sort of thing silly people try to do without training'.
However, that said, I'm being disingenuous. What I really truly meant at the time I typed that was moral-should, not practical-should, which come apart if one isn't a perfect consequentialist. Which I ain't, which is at least partly the point.
I think we should stop talking about utility functions.
In the context of ethics for humans, anyway. In practice I find utility functions to be, at best, an occasionally useful metaphor for discussions about ethics but, at worst, an idea that some people start taking too seriously and which actively makes them worse at reasoning about ethics. To the extent that we care about causing people to become better at reasoning about ethics, it seems like we ought to be able to do better than this.
The funny part is that the failure mode I worry the most about is already an entrenched part of the Sequences: it's fake utility functions. The soft failure is people who think they know what their utility function is and say bizarre things about what this implies that they, or perhaps all people, ought to do. The hard failure is people who think they know what their utility function is and then do bizarre things. I hope the hard failure is not very common.
It seems worth reflecting on the fact that the point of the foundational LW material discussing utility functions was to make people better at reasoning about AI behavior and not about human behavior.