I think one reason why people (including me, in the past) have difficulty accepting the way you present this argument is that you're speaking in too abstract terms, while many of the values that we'd actually like to preserve are ones that we appreciate the most if we consider them in "near" mode. It might work better if you gave concrete examples of ways by which there could be a catastrophic value drift, like naming Bostrom's all-work-and-no-fun scenario where

what will maximize fitness in the future will be nothing but non-stop high-intensity drudgery, work of a drab and repetitive nature, aimed at improving the eighth decimal of some economic output measure

or some similar example.