TimS comments on We Don't Have a Utility Function - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (123)
I'll admit it's rather shaky and I'd be saying the same thing if I'd merely been brainwashed. It doesn't feel like it was precipitated by anything other than legitimate moral argument, though. If I can be brainwashed out of my "terminal values" so easily, and it really doesn't feel like something to resist, then I'd like a sturdier basis on which to base my moral reasoning.
What is a conversation metaphor? I'm afraid I don't see what you're getting at.
I still value freedom in what feels like a fundamental way, I just also value hierarchy and social order now. What is gone is the extreme feeling of ickyness attached to authority, and the feeling of sacredness attached to freedom, and the belief that these things were terminal values.
The point is that things I'm likely to identify as "terminal values", especially in the contexts of disagreements, are simply not that fundamental, and are much closer to derived surface heuristics or even tribal affiliation signals.
I feel like I'm not properly responding to your comment though.
My fault for failing to clarify. There are roughly three ways one can talk about changes to an agent's terminal values.
(1) Such changes never happen. (At a society level, this proposition appears to be false).
(2) Such changes happen through rational processes (i.e. reasoning).
(3) Such changes happen through non-rational processes (e.g. tribal affiliation + mindkilling).
I was using "conversion" as a metaphorical shorthand for the third type of change.
BTW, you might want to change "conversation" to "conversion" in the grandparent.
Ah! Thanks.
Ok. Then my answer to that is roughly this:
This could of course use more detail, unless you understand what I'm getting at.