alexflint comments on What makes us think _any_ of our terminal values aren't based on a misunderstanding of reality? - Less Wrong

17 Post author: bokov 25 September 2013 11:09PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (89)

You are viewing a single comment's thread.

Comment author: alexflint 01 October 2013 03:23:13AM *  2 points [-]

Another way to think about Dave's situation is that his utility function assigns the same value to all possible futures (i.e. zero) because the one future that would've been assigned a non-zero value turned out to be unrealizable. His real problem is that his utility function has very little structure: it is zero almost everywhere.

I suspect our/my/your utility function is structured in a way that even if broad swaths of possible futures turn out to be unrealizable, the remainder will still contain gradients and local maxima, so there will be some more desirable and some less desirable possibilities.

Of course this is not guaranteed, but most utility functions have gradients and local maxima over most sets. You need a very special utility function and a very special set of realizable futures in order for all futures to be assigned exactly the same value.