Gandhi does not prefer to murder. He prefers to not-murder. His human brain contains the wiring to implement "frothing lunacy", sure, and a little pill might bring it out, but a pill is not a fact. It's not even an argument.

No pills required. People are not 100% conditionable, but they are highly situational in their behaviour. I'll stand by the idea that, for example, anyone who has ever fantasized about killing anyone can be situationally manipulated over time to consciously enjoy actual murder. Your subconscious doesn't seem to actually know the difference between imagination and reality, even if you do.

Perhaps Gandhi could not be manipulated in this way due to preexisting highly built up resistance to that specific act. If there is any part of him, at all, that enjoys violence, though, it's a question only of how long it will take to break that resistance down, not of whether it can be.

People do experience dramatic and beneficial preference reversals through experiencing things that, on the whole, they had dispreferred previously.

Yes, they do. And if I expected that an activity would cause a dramatic preference reversal, I wouldn't do it.

Of course. And that is my usual reaction, too, and probably even the standard reaction -- it's a good heuristic for avoiding derangement. But that doesn't mean that it is actually more optimal to not do the specified action. I want to prefer to modify myself in cases where said modification produces better outcomes. In these circumstances if it can be executed it should be. If I'm a FAI, I may have enough usable power over the situation to do something about this, for some or even many people, and it's not clear,as it would be for a human, that "I'm incapable of judging this correctly".

In case it's not already clear, I'm not a preference utilitarian -- I think preference satisfaction is too simple a criteria to actually achieve good outcomes. It's useful mainly as a baseline.

Huh? She's just changing people's plans by giving them chosen information, she's not performing surgery on > their values Did you notice that you just interpreted 'preference' as 'value'? This is not such a stretch, but they're not obviously equivalent either.

I'm not sure what 'surgery on values' would be. I'm certainly not talking about physically operating on anybody's mind, or changing that they like food, sex, power, intellectual or emotional stimulation of one kind or another, and sleep, by any direct chemical means, But how those values are fulfilled, and in what proportions, is a result of the person's own meaning-structure -- how they think of these things. Given time, that is manipulable. That's what CelestAI does.. it's the main thing she does when we see her in interactiion with Hofvarpnir employees.

In case it's not clarified by the above: I consider food, sex, power, sleep, and intellectual or emotional stimulation as values, 'preferences' (for example, liking to drink hot chocolate before you go to bed) as more concrete expressions/means to satisfy one or more basic values, and 'morals' as disguised preferences.