As it turns out, I care about the world directly, and that's the meaning of "affect" here - affecting my utility, not affecting my perception of my utility.

In that case TheOtherDave's statement is completely vacuous.