Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

pnrjulius comments on Prolegomena to a Theory of Fun - Less Wrong

27 Post author: Eliezer_Yudkowsky 17 December 2008 11:33PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (49)

Sort By: Old

You are viewing a single comment's thread.

Comment author: pnrjulius 06 June 2012 08:00:33PM 0 points [-]

How do we know that our own preferences are worth trusting? Surely you believe in possible preference systems that are defective (I'm reminded of another post involving giant cheesecakes). But how do we know that ours isn't one of them? It seems plausible to me that evolution would optimize for preferences that aren't morally optimal, because its utility function is inclusive fitness.

This requires us to ask what metric we would use, outside our own preferences; not an easy question, but one I think we have to face up to asking. Otherwise, we'll end up making giant cheesecakes.