I've realized that my sibling comment is logically rude, because I've left out some relevant detail. Most relevantly, I tend to self-describe as a virtue ethicist.

I've noticed at least 3 things called 'virtue ethics' in the wild, which are generally mashed together willy-nilly:

1. an empirical claim, that humans generally act according to habits of action and doing good things makes one more likely to do good things in the future, even in other domains
2. the notion that ethics is about being a good person and living a good life, instead of whether a particular action is permissible or leads to a good outcome
3. virtue as an achievement; a string of good actions can be characterized after the fact as virtuous, and that demonstrates the goodness of character.

There are virtue ethicists who buy into only some of these, but most often folks slip between them without noticing. One fellow I know will often say that #1 being false would not damage virtue ethics, because it's really about #2 and #3 - and yet he goes on arguing in favor of virtue ethics by citing #1.

I'm still not sold on the idea that an intelligent being would slavishly follow its utility function.

If it's really your utility function, you're not following it "slavishly" - it is just what you want to do.

For AI, there are no questions about the meaning of life then? Just keep on U maximizing?

If "questions about the meaning of life" maximize utility, then yes, there are those. Can you unpack what "questions about the meaning of life" are supposed to be, and why you think they're important? ('meaning of "life"' is fairly easy, and 'meaning of life' seems like a category error).

I follow the virtue-ethics approach, I do actions that make me like the person that I want to be. The acquisition of any virtue requires practice, and holding open the door for old ladies is practice for being altruistic. If I weren't altruistic, then I wouldn't be making myself into the person I want to be.

It's a very different framework from util maximization, but I find it's much more satisfying and useful.

What makes us think that AI would stick with the utility function they're given?

There are very few situations in which an agent can most effectively maximise expected utility according to their current utility function by modifying themselves to have a different utility function. Unless the AI is defective or put in a specially contrived scenario it will maintain its current utility function because that is an instrumentally useful thing to do.

If you are a paperclip maximiser then becoming a staples maximiser is a terribly inefficient strategy for maximising paperclips unless Omega is around making weird bargains.

I change my utility function all the time, sometimes on purpose.

No you don't. That is, to the extent that you "change your utility function" at all you do not have a utility function in sense meant when discussing AI. It only makes sense to model humans as having 'utility functions' when they are behaving in a manner that can be vaguely approximated as expected utility maximisers with a particular preference function.

Sure, it is possible to implement AIs that aren't expected utility maximisers either and those AIs could be made to do all sorts of arbitrary things including fundamentally change their goals and behavioral strategies. But if you implement an AI that tries to maximise a utility function then it will (almost always) keep trying to maximise that same utility function.

The point was not necessarily to advocate torture. It's to take the math seriously.

In fact, the mental discomfort caused by people who heard of the torture would swamp the disutility from the dust specks.

Just how many people do you expect to hear about the torture? Have you taken seriously how big a number 3^^^3 is? By how many utilons do you expect their disutility to exceed the disutility from the dust specks?

I was very surprised to find that a supporter of the Complexity of Value hypothesis and the author who warns against simple utility functions advocates torture using simple pseudo-scientific utility calculus.

My utility function has constraints that prevent me from doing awful things to people, unless it would prevent equally awful things done to other people. That this is a widely shared moral intuition is demonstrated by the reaction in the comments section. Since you recognize the complexity of human value, my widely-shared preferences are presumably valid.

In fact, the mental discomfort caused by people who heard of the torture would swamp the disutility from the dust specks. Which brings us to an interesting question - is morality carried by events or by information about events? If nobody else knew of my choice, would that make it better?

For a utilitarian, the answer is clearly that the information about morally significant events is what matters. I imagine so-called friendly AI bots built on utilitarian principles doing lots of awful things in secret to achieve its ends.

Also, I'm interested to hear how many torturers would change their mind if we kill the guy instead of just torturing him. How far does your "utility is all that matters" philosophy go?

What makes us think that AI would stick with the utility function they're given? I change my utility function all the time, sometimes on purpose.

"Long-term monogamy should not be done on the pretense that attraction and arousal for one's partner won't fade. It will."

This is precisely the point of monogamy. Polyamory/sleeping around is a young man's game. Long-term monogamy is meant to maintain strong social units throughout life, long after the thrill is gone.

"Eliezer is basically lamenting that when people behave rationally, they refuse to act against their self-interest, and damn it, it's hurting the rational tribe. That's informative, and sort of my point."

So if that's Eliezer's point, and it's also your point, what is it that you actually disagree about?

I take Eliezer to be saying that sometimes rational individuals fail to co-operate, but that things needn't be so. In response, you seem to be asking him to prove that rational individuals must co-operate - when he already appears to have accepted that this isn't true.

Isn't the relevant issue whether it is possible for rational individuals to co-operate? Provided we don't make silly mistakes like equating rationality with self-interest, I don't see why not - but maybe this whole thread is evidence to the contrary. ;)