I've heard that point of view from several people. It's a natural extension of LW-style beliefs, but I'm not sure I buy it yet. There are several lines of attack, the most obvious one is trying to argue that coinflips still behave as coinflips even when the person betting on them is really stupid and always bets on heads. But we've already explored that line a little bit, so I'm gonna try a different one:

Are you saying that evolution has equipped our minds with a measure of caring about all possible worlds according to simplicity? If yes, can you guess which of our ancestor organisms were already equipped with that measure, and which ones weren't? Monkeys, fishes, bacteria?

(As an alternative, there could be some kind of law of nature saying that all minds must care about possible worlds according to simplicity. But I'm not sure how that could be true, given that you can build a UDT agent with any measure of caring.)

A: Wait, if simplicity changes preferences, but does not change the level of existence, how do you explain the fact that we appear to be in a world that is simple? Isn’t that a priori extremely unlikely?

B: This is where it gets a little bit fuzzy, but I do not think that question makes sense. Unlikely by what measure? You are presupposing an existence measure on the collection of theoretical worlds just to ask that question.

I saw a good explanation of this point (in the #lesswrong IRC sometime last month), which I'll try to reproduce here.

Imagine that there is a specific flavor of Universal Turing Machine which runs everything in Tegmark IV; all potential-universe objects are encoded as tapes for this UTM to simulate.

Take a simple set of rules. There is only one tape that corresponds to these precise rules. Now add an incredibly-specific edge case which applies in exactly one situation that could possibly arise, but otherwise have the same simple set of rules hold; this adds another tape that will produce precisely the same observations from the inside as the simple rules, except in one case that is grotesquely uncommon and very likely will never occur (because it happens to req... (read more)

My personal objection to this seems to be: "But what if some things DO exist?" IE, what if existence actually does turn out to be a thing, in limited supply?

Let me try to unfold this gut reaction into something which makes more sense.

I think what I'm "really" saying is that, for me, this view does not add up to normality. If I were to discover that "existence" is meaningless as a global predicate, and (to the extent that it's meaningful) all possibilities exist equally, I would change the way I care about things.

I think I valu... (read more)

Thank you all so much for all of your comments.

Three separate comment threads simultaneously lead to the refutation that I seem to be unfairly biased towards agents that happen to be born in simple universes. I think this is a good point, so I am starting a new comment thread to discuss that issue. Here is my counter point.

First, notice that our finite intuitions do not follow over nicely here. In saying that beings in universes with 20 fewer bits are a million times as important, I am not saying that the happiness of this one person is more important tha... (read more)

As I understand it, your attempted solution to the Problem of Induction is this:

(a) Deny that there is a fact of the matter about what our future experiences will be like

(b) Care about things in inverse proportion to the Kolmogorov complexity of the structure in which they are embedded.

This is why it all adds up to normality. Without (a), people could say: Go ahead, care about whatever you want, but under your belief system you ought to expect the world to dissolve into high-complexity chaos immediately! And without (b), people could say: Go ahead, deny th... (read more)

Interesting approach. The way I would put it, the number you want to maximize is expectation of U(X) where U is a utility function and X is a random universe taken from a Solomonoff ensemble. The way you put it, the number is the same but you don't interpret the sum over universes as expectation value, you just take it to be part of your utility function.

What I feel is missing in your approach is that U by its nature is arbitrary / complex, whereas the Solomonoff prior is simple and for some reason has to be there. I.e. something would go wrong on the phil... (read more)

So why haven't you tried walking into a casino and caring really hard about winning? I'm not just being a prick here, this is the most concise way I could think to frame my core objection to your thesis.

What are the semantics of "exist" on this view? When people say things like "Paris exists but Minas Tirith doesn't" are they saying something meaningful? True? It seems like such statements do convey actual information to someone who knows little about both a French history book and a Tolkein book. Why not just exercise some charitable interpretation upon your fellow language users when it comes to "exist"? We use "existence" concepts to explain our experiences, without any glaring difficulties that I can see (so a charitable interpretation is likely possible). But maybe I've missed some glaring difficulties.

Reminds me of some discussions I had long ago.

The general principle was: Take some human idea or concept and drive it to its logical conclusion. To its extreme. The result: It either became trivial or stopped to make sense.

The reason for this is that sometimes you can't separate out components of a complex system and try to optimize them in isolation without losing important details.

You can't just optimize happyness. It leads to wireheading and that's not what you want.

You can't just optimize religious following. It leads to crussades and witch hunts.

You ... (read more)

Nice dialogue.

It's true that probability and importance are interchangeable in an expected-utility calculation, but if you weight A twice as much as B because you care twice as much about it, that implies equal probabilities between the two. So if you use a Solomonoff-style prior based on how much you care, that implies a uniform prior on the worlds themselves. Or maybe you're saying expected utility is the sum of caring times value, with no probabilities involved. But in that case your probability is just how much you care.

If we were in a complex world, i... (read more)

I'm not beginning to think I identified a real flaw in this approach.

The usual formulation of UDT assumes the decision algorithm to be known. In reality, the agent doesn't know its own decision algorithm. This means there is another flavor of uncertainty w.r.t. which the values of different choices have to be averaged. I call this "introspective uncertainty". However, introspective uncertainty is not independent: it is strongly correlated with indexical uncertainty. Since introspective uncertainty can't be absorbed into the utiltity function, indexical uncertainty cannot either.

I have a precise model of this kind of UDT in mind. Planning to write about it soon.

Instead of saying that you care about simpler universes more, couldn't a similar preference arise out of pure utilitarianism? Simpler universes would be more important because things that happen within them will be more likely to also happen within more complicated universes that end up creating a similar series of states. For every universe, isn't there an infinite number of more complicated universes that end up with the simpler universe existing within part of it?