pjeby comments on Post Your Utility Function - Less Wrong

28 Post author: taw 04 June 2009 05:05AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (273)

You are viewing a single comment's thread. Show more comments above.

Comment author: pjeby 04 June 2009 06:53:50AM 1 point [-]

If CEV outputs a null utility function, that would seem to imply that human preferences are completely symmetrically distributed, which seems hard to believe.

If by "null utility function", you mean one that says, "don't DO anything", then do note that it would not require that we all have balanced preferences, depending on how you do the combination.

A global utility function that creates more pleasure for me by creating pain for you would probably not be very useful. Heck, a function that creates pleasure for me by creating pain for me might not be useful. Pain and pleasure are not readily subtractable from each other on real human hardware, and when one is required to subtract them by forces outside one's individual control, there is an additional disutility incurred.

These things being the case, a truly "Friendly" AI might well decide to limit itself to squashing unfriendly AIs and otherwise refusing to meddle in human affairs.

Comment author: loqi 04 June 2009 07:08:32AM 1 point [-]

These things being the case, a truly "Friendly" AI might well decide to limit itself to squashing unfriendly AIs and otherwise refusing to meddle in human affairs.

I wouldn't be particularly surprised by this outcome.