LESSWRONG
LW

All of winwonce's Comments + Replies

UDT might not pay a Counterfactual Mugger
winwonce20

I don't think Nomega has to simulate you interacting with Omega in order to know how to would react should you encounter it, in the same way that you can predict the output of many computer programs without simulating them.

By the time you get mugged, you could be 100% sure that you are in the Omega world, rather than the Nomega world, but the principle is that your decision in the Omega world affects the Nomega world, and so before knowing UDT commits to making the decision that maximizing EV across both worlds.

This logic operates in the same way for the c... (read more)

Reply
2interstice
By 'simulating' I just mean that it's reasoning in some way about your behavior in another universe, it doesn't have to be a literal simulation. But the point remains -- of all the ways that Nomega could choose to act, for some reason it has chosen to simulate/reason about your behavior in a universe containing Omega, and then give away its resources depending on how it predicts you'll act. What this means is that, from a Kolmogorov complexity perpective, Nomega is strictly more complex than Omega, since the definition of Nomega includes simulating/reasoning about Omega. Worlds containing Nomega will be discounted by a factor proportional to this additional complexity. Say it takes 100 extra bits to specify Nomega. Then worlds containing Nomega have 2−100 less measure under the Solomonoff prior than worlds with Omega, meaning that UDT cares much less about them. (My comment above was reasoning as if Nomega could choose to simulate/reason about many different possible universes, not just the ones with Omega. Then, perhaps, its baseline complexity might be comparable to Omega. Either way, the result is that the worlds where Nomega exists and you have influence don't have very high measure) What I meant by "Nomega world" in that paragraph was a world where Nomega exists but does not simulate/reason about your behavior in the Omega world. The analogous situation to the tails/heads world here is the "Omega"/"Nomega simulating omega" world. I acknowledge that you would have counterfactual influence over this world. The difference is that the heads/tails worlds have equal measure, whereas the "Nomega simulates omega" world has much less measure than the Omega world(under a 'reasonable' measure such as Solomonoff)
UDT might not pay a Counterfactual Mugger
winwonce*10

But UDT's decision on how to interact woth Omega does direct affect worlds in which Nomega exists instead of Omega.

Again overly simplistic prior:

50% chance: Omega exists, and we get counterfactually mugged, half of the times heads and half of the times tails.

50% chance: Nomega exists, guesses what we would do if Omega existed and the coin came up tails, and pays out accordingly.

There is only one decision -- do you pay if Omega exists and the coin comes up tails, and that decision affects both (or all three) possible worlds.

Even once you see that Omega exists, UDT already recognized that in order to maximize utility it should precommit (or just decide or whatever) to not pay.

Reply
1interstice
UDT's behavior here is totally determined by its prior. The question is which prior is more reasonable. 'Closeness to Solomonoff induction' is a good proxy for reasonableness here. I think a prior putting greater weight on Omega, given that one has seen Omega, is much more reasonable. Here's the reasoning. Let's say that the description complexity of both Omega and Nomega is 1000 bits. Before UDT has seen either of them, it assigns a likelihood of 2−1000 to worlds where either of them exist. So it might seem that it should weight them equally, even having seen Omega. However, the question then becomes -- why is Nomega choosing to simulate the world containing Omega? Nomega could choose to simulate any world. In fact, a complete description of Nomega's behavior must include a specification of which world it is simulating. This means that, while it takes 1000 bits to specify Nomega, specifying that Nomega exists and is simulating the world containing Omega actually takes 2000 bits.[1] So UDT's full prior ends up looking like: * 999/1000: Normal world * 2−1000: Omega exists * 2−1000: Nomega exists * 2−2000: Nomega exists and is simulating the world containing Omega Thus, in a situation where UDT has seen Omega, it has influence over the Omega world and Nomega/Omega world, but no influence over the normal world and Nomega world. Since the Omega world has so much more weight than the Omega/Nomega world, UDT will effectively act as if it's in the Omega world. ---------------------------------------- 1. You might object that Nomega is defined by its property of messing with Omega, so it will naturally simulate worlds with Omega. In that case, it's strictly more complex to specify than Omega, probably by several hundred bits due to the complexity of 'messing with' ↩︎
UDT might not pay a Counterfactual Mugger
winwonce10

If the answer is that you have a higher prior towards Omega before the mugging, then fine that solves the problem. But if you think Omega is more likley to exist only because you see Omega in front of you, then doesnt that violate UDTs principle of never updating?

Reply
1interstice
Although UDT is formally updateless, the 'mathematical intuition module' which it uses to determine the effects of its actions can make it effectively act as though it's updating. Here's a simple example. Say UDT's prior over worlds is the following: * 75% chance: you will see a green and red button, and a sign saying "press the red button for $5" * 25% chance: same buttons, but the sign says "press the green button for $5" Now, imagine the next thing UDT sees is the sign saying that it should press the green button. Of course, what it should do is press the green button(assuming the signs are truthful), even though in expectation the best thing to do would be pressing the red button. So why does it do this? UDT doesn't update -- it still considers the worlds where it sees the red button to be 3X more important -- however, what does change is that, once it sees the green button sign, it no longer has any influence over the worlds where it sees the red button sign. Thus it acts as though it's effectively updated on seeing the green button sign, even though its distribution over worlds remains unchanged. By analogy, in your scenario, even though Omega and Nomega might be equally likely a priori, UDT's influence over Omega's actions is far greater given that it has actually seen Omega. Or to be more precise -- in the situation where UDT has both seen Omega and the coin comes up heads, it has a lot of predictable influence over Omega's behavior in a(equally valuable by its prior) world where Omega is real and the coin comes up tails. It has no such predictable influence over worlds where Nomega exists.
UDT might not pay a Counterfactual Mugger
winwonce10

Hmm perhaps I am still a little confused as to how UDT works. My understanding is that you don't make your decisions based on the information you have observed, but instead, when you "boot up" your UDT, you consider all of the possible world states you may find yourself in and their various mesures, and then for each decision, "precommit" to making the one that maximizes your expected utility across all of the possible world states that this decision affects.

If this understanding is correct, then unless we have some sort of prior telling us, when we "boot ... (read more)

Reply
UDT might not pay a Counterfactual Mugger
winwonce10

Whoops -- EV re-updated.

Perhaps I am misunderstanding the setup of the counterfactual mugging -- do we live in a world in which Omega is a known being (and just hasn't yet interacted with us), or do we live in a world in which we have roughly equal credence of the existence of Omega vs Nomega (vs any other arbitrary God-like figure). If it's the former, then sure UDT says precommit and pay.

But if its the latter, I still don't see why UDT tells us to pay -- not because not precommitting is some sort of default (which is I agree UDT says isn't relevant) but ... (read more)

Reply
3Vladimir_Nesov
In the first approximation, the point is not that counterfactual mugging (or any other thought experiment) is actually defined in a certain way, but how it should be redefined in order to make it possible to navigate the issue. Unless Nomegas are outlawed, it's not possible to do any calculations, therefore they are outlawed. Not because they were already explicitly outlawed or were colloquially understood to be outlawed. But when we look at this more carefully, the assumption is not actually needed. If nonspecified Nomegas are allowed, the distribution of their possible incentives is all over the place, so they almost certainly cancel out in the expected utility of alternative precommitments. The real problem is not with introduction of Nomegas, but with managing to include the possibilities involving Omega in the calculations (as opposed to discarding them as particular Nomegas), taking into account the setting that's not yet described at the point where precommitment should be made. In counterfactual mugging, there is no physical time when the agent is in the state of knowledge where the relevant precommitment can be made (that's the whole point). Instead, we can construct a hypothetical state of knowledge that has updated on the description of the thought experiment, but hasn't updated on the fact of how the coin toss turned out. The agent never holds this state of knowledge as a description of all that's actually known. Why retract knowledge of the coin toss, instead of retracting knowledge of the thought experiment? No reason, UDT strives to retract all knowledge and make a completely general precommitment to all eventualities. But in this setting, retracting knowledge of the coin toss while retaining knowledge of Omega creates a tractable decision problem, thus UDT that notices the possibility will make a precommitment. Similarly, it should precommit to not paying Omega in a situation where a Nomega punishing for paying up $100 to Omega (as described in thi
UDT might not pay a Counterfactual Mugger
winwonce10

Hi Vladimir, thanks for your response.

Upon further reflection, I think the crux of my argument is that by precommiting you are essentially pascals wagering yourself -- you are making a decision looking to maximize yoir reward should a certain type of God (Omega) exist. Unless (before you get mugged) you have some reason to believe that this type of God is more likley to exist then the opposite type (Nomega), then precommiting is getting wagered (as far as I can tell). You cant wait until you find out that Omega exists to preccomit because by then you have ... (read more)

Reply
3Vladimir_Nesov
I think I see what you mean. The situation where you'd make a precommitment, which is described by the same state of knowledge that UDT makes its decision under, occurs before the setting of the thought experiment is made clear. Thus it's not yet clear what kinds of Nomegas can show up with their particular incentives, and the precommitment can't rely on their absense. With some sort of risk-averse status-quo-anchored attitude it seems like "not precomitting" is therefore generally preferable. But optimization of expected utility doesn't work like that. You have the estimates for possible decisions, and pick the option that's estimated to be the best available. Whether it's the status quo ("not precommitting") or not has no bearing on the decision unless it's expressed as a change in the esimate of expected utility that makes it lower or greater than expected utility of the alternative decisions. Thus when a thought experiment talks about precommitments or any UDT decisions, bringing in arbitrary Nomegas is a problem because it makes the expected utility of precommitments similarly arbitrary, and it's these expected utilities that determine decisions. (Whether to make some precommitment or not is itself a decision.) The obvious way of making it possible to perform the calculation of expected utilities of precommitments is to make the assumption of absense of Nomegas, or more generally to construct the settings of precommitments based only on what's already in the thought experiment. (Mistakes in expected values in the post are a tiny bit relevant (one of the values is still wrong after the correction), as they vaguely signal lack of reliable knowledge of what expected utility is, although the issue here is mostly informal and correct calculation won't by itself make things clear. General experience with mathematical proofs might be closer to being helpful, as the issue is that the actual algorithms being discussed screen off a lot of informal considerations such a