The world is full of people who may want to edit my values ever-so-slightly while I'm not looking, in order to further their own agenda.

Hm, at this point it sounds similar to the point Phil Goetz was making in "Reason as memetic immune disorder".