My argument is more like: If our prior says almost everyone optimizes for Y, and few optimize for X, but many people like to say they optimize for X, and you point to an action A you have taken which increases X and say "Look how I'm optimizing for X!" and A increases Y as well, and there is an obvious alternative B which an X-optimizer would have noticed and which increases X more than A at the cost of not increasing Y, then we've just explained away A as evidence that you optimize for X and should stop privileging that hypothesis.

Edit: one more thing is important here, which is that A is something you'd do if you were optimizing Y, not just something you'd only do if you were jointly optimizing X and Y. You might be thinking "Well, just donate sperm/eggs!". But that is not an option that the Y-agent controlling the Thread Starter really "considers", because the selective forces that shape parental instincts did most of their work before the presence of that option.

Depending on how you define "care", maybe you can care about X without optimizing for it, but that doesn't really matter to me, as that kind of caring doesn't do X any good.