It's enough that I don't care about living for a very long time.

And if I say that you do, what is the criterion for telling which statement is the correct one? That criterion is what I referred to as the fact of the matter about what you (should) care about.

This seems wrong to me. That Toby should X does not imply that Toby does X, so determining what Toby should want does not settle whether Toby in fact wants it.

you can't claim that the thing you should do is to not care.

Toby does not seem to be making that claim, though perhaps implicitly so. (Much like it could be argued that "X" implies "I believe that X", it could be argued that "I did X" implies "I should have done X". But that fails on common usage, where "I did X but I should not have done X" is ordinary.)