Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Arthur comments on Prolegomena to a Theory of Fun - Less Wrong

27 Post author: Eliezer_Yudkowsky 17 December 2008 11:33PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (49)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Arthur 18 December 2008 05:42:11AM 2 points [-]

"...if you would prefer not to become orgasmium, then why should you?"

I'd prefer not to become orgasmium, because I value my consciousness and humanity, my ability to think, decide, and interact. However, it's unclear to me what exactly preference is, other than the traversal of pathways we've constructed, whether we're aware of them or not, leading to pleasure, or away from pain. To drastically oversimplify, those values exist in me as a result of beliefs I've constructed, linking the lack of those things to an identity that I don't want, which in turn is eventually linked to an emotional state of sadness and loss that I'd like to avoid. There's also probably a link to a certain identity that I do want, which leads to a certain sense of pride and rightness, which leads me to a positive emotional state.

Eliezer, you said there was nothing higher to override your preference to increase intelligence gradually. But what about the preferences that led you to that one? What was it about the gradual increase of intelligence, and your beliefs about what that means, that compelled you to prefer it? Isn't that deeper motivation closer to your actual volition? How far down can we chase this? What is the terminal value of fun, if not orgasmium?

Or is "fun" in this context the pursuit specifically of those preferences that we're consciously aware of as goals?