= 27d567bc2d48d9f40e9ccc20ea370b15)
cousin_it comments on Open thread, August 5-11, 2013 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (307)
Just a fun little thing that came to my mind.
If "anthropic probabilities" make sense, then it seems natural to use them as weights for aggregating different people's utilities. For example, if you have a 60% chance of being Alice and a 40% chance of being Bob, your utility function is a weighting of Alice's and Bob's.
If the "anthropic probability" of an observer-moment depends on its K-complexity, as in Wei Dai's UDASSA, then the simplest possible observer-moments that have wishes will have disproportionate weight, maybe more than all mankind combined.
If someday we figure out the correct math of which observer-moments can have wishes, we will probably know how to define the simplest such observer-moment. Following SMBC, let's call it Felix.
All parallel versions of mankind will discover the same Felix, because it's singled out by being the simplest.
Felix will be a utility monster. The average utilitarians who believe the above assumptions should agree to sacrifice mankind if that satisfies the wishes of Felix.
If you agree with that argument, you should start preparing for the arrival of Felix now. There's work to be done.
Where is the error?
That's the sharp version of the argument, but I think it's still interesting even in weakened forms. If there's a mathematical connection between simplicity and utility, and we humans aren't the simplest possible observers, then playing with such math can strongly affect utility.
How would being moved by this argument help me achieve my values? I don't see how it helps me to maximize an aggregate utility function for all possible agents. I don't care intrinsically about Felix, nor is Felix capable of cooperating with me in any meaningful way.
How does your aggregate utility function weigh agents? That seems to be what the argument is about.
Felix exists as multiple copies in many universes/Everett branches, and it's measure is the sum of the measures of the copies. Each version of mankind can only causally influence (e.g., make happier) the copy of Felix existing in the same universe/branch, and the measure of that copy of Felix shouldn't be much higher than that of an individual human, so there's no reason to treat Felix as a utility monster. Applying acausal reasoning doesn't change this conclusion either. For example all the parallel versions of mankind could jointly decide to make Felix happier, but while the benefit of that is greater (all the copies of Felix existing near the parallel versions of mankind would get happier), so would the cost.
If Felix is very simple it may be deriving most of its measure from a very short program that just outputs a copy of Felix (rather than the copies existing in universes/branches containing humans), but there's nothing humans can do to make this copy of Felix happier, so its existence doesn't make any difference.
Why? Even within just one copy of Earth, the program that finds Felix should be much shorter than any program that finds a human mind...
Are you thinking that the shortest program that finds Felix in our universe would contain a short description of Felix and find it by pattern matching, whereas the shortest program that finds a human mind would contain the spacetime coordinates of the human? I guess which is shorter would be language dependent... if there is some sort of standard language that ought to be used, and it turns out the former program is much shorter than the latter in this language, then we can make the program that finds a human mind shorter by for example embedding some kind of artificial material in their brain that's easy to recognize and doesn't exist elsewhere in nature. Although I suppose that conclusion isn't much less counterintuitive than "Felix should be treated as a utility monster".
Yeah, there's a lot of weird stuff going on here. For example, Paul said sometime ago that ASSA gives a thick computer larger measure than a thin computer, so if we run Felix on a computer that is much thicker than human neurons (shouldn't be hard), it will have larger measure anyway. But on the other hand, the shortest program that finds a particular human may also do that by pattern matching... I no longer understand what's right and what's wrong anymore.
Hal Finney pointed out the same thing a long time ago on everything-list. I also wrote a post about how we don't seem to value extra identical copies in a linear way, and noted at the end that this also seems to conflict with UDASSA. My current idea (which I'd try to work out if I wasn't distracted by other things) is that the universal distribution doesn't tell you how much you should value someone, but only puts an upper bound on how much you can value someone.
http://xkcd.com/687/
Or to put it another way - probability is not just a unit. You need to keep track of probability of what, and to whom, or else you end up like the bad dimensional analysis comic.
A version of this that seems a bit more likely to me at least; the thing that matters is not the simplicity of the mind itself, but rather the ease of pointing it out among the rest of the universe; this'd mean that, basically, a a planet sized Babbage engine running a single human equivalent mind, would get more weight than a planet sized quantum computer running trillions and trillions of such minds. It'd also mean that all sorts of implementation details of how close the experiencing level is to raw physics would matter a lot, even if the I/O behaviour is identical. This is highly counter-intuitive.
One flaw; Felix almost certainly resides outside our causal reach and doesn't care about what happens here.