Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Richard_Hollerith2 comments on Disputing Definitions - Less Wrong

48 Post author: Eliezer_Yudkowsky 12 February 2008 12:15AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (42)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Richard_Hollerith2 13 February 2008 05:34:39PM 3 points [-]

Eliezer said that another time you can't just generate new words is when a category boundary like "person" or "human" or "baby" makes a direct appearance in your utility function.

Which gently suggests that when defining a utility function that might remain in force for billions of years, one should prefer functions that do not have category boundaries.

I would be happy to exhibit functions of that sort that have the property that even after an explosion of engineered intelligence, the humans probably retain enough expected utility to keep them flourishing and protected from exploitation although they probably do not retain enough expected utility to cause the majority of the future light cone's space, time, matter, free energy and other resources to be devoted to them.