Our values are fragile, some see this as a reason to not be too concerned with them.

I think it really depends on the exact value change we're talking about. There's an analogue for death/aging--you'd probably greatly prefer aging another 10 years, then being frozen at that biological age forever, over aging and dying normally. In the same way, I might not consider a small drift in apparently unimportant values to big a deal in the grand scheme of things, and might not choose to spend resources guarding against this (slippery slope scenarios aside).

In practice, people don't seem to be that concerned with guarding against small value changes. They do things like travel to new places, make new friends, read books, change religions, etc., all of which are likely to change what they value, often in unpredictable ways.