You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

torekp comments on Goal retention discussion with Eliezer - Less Wrong Discussion

56 Post author: MaxTegmark 04 September 2014 10:23PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (26)

You are viewing a single comment's thread. Show more comments above.

Comment author: torekp 05 September 2014 11:57:38PM *  2 points [-]

In the link you suggest that ontological crises might lead to nihilism, but I think a much more likely prospect is that they lead to relativism, with respect to the original utility function. That is, there are solutions to the re-interpretation problem, which, for example, allow us to talk of "myself" and "others" despite the underlying particle physics. But there are more than one of those solutions, none of which are forced. Thus the original "utility function" fails to be such, strictly speaking. It does not really specify which actions are preferred. It only does so modulo a choice of interpretation.

So, all we need to do is figure out each of the possible ways physics might develop, and map out utility functions in terms of that possible physics! Or, we could admit that talk of utility functions needs to be recognized as neither descriptive nor truly normative, but rather as a crude, mathematically simplified, approximation to human values. (Which may be congruent to your conclusions - I just arrive by a different route.)