drnickbone comments on Pascal's Muggle: Infinitesimal Priors and Strong Evidence - Less Wrong

43 Post author: Eliezer_Yudkowsky 08 May 2013 12:43AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (404)

You are viewing a single comment's thread. Show more comments above.

Comment author: drnickbone 11 May 2013 08:21:28AM *  0 points [-]

As lukeprog says here, this really needs to be written up. It's not clear to me that just because the measure over observers (or observer moments) sums to one then the expected utility is bounded.

Here's a stab. Let's use s to denote a sub-program of a universe program p, following the notation of my other comment. Each s gets a weight w(s) under UDASSA, and we normalize to ensure Sum{s} w(s) = 1.

Then, presumably, an expected utility looks like E(U) = Sum{s} U(s) w(s), and this is clearly bounded provided the utility U(s) for each observer moment s is bounded (and U(s) = 0 for any sub-program which isn't an "observer moment").

But why is U(s) bounded? It doesn't seem obvious to me (perhaps observer moments can be arbitrarily blissful, rather than saturating at some state of pure bliss). Also, what happens if U bears no relationship to experiences/observer moments, but just counts the number of paperclips in the universe p? That's not going to be bounded, is it?

Comment author: paulfchristiano 11 May 2013 05:08:58PM 0 points [-]

I agree it would be nice if things were better written up; right now there is the description I linked and Hal Finney's.

If individual moments can be arbitrarily good, then I agree you have unbounded utilities again.

If you count the number of paperclips you would again get into trouble; the analogous thing to do would be to count the mesure of paperclips.