Comment author: Wei_Dai 06 February 2010 06:47:30PM *  2 points [-]

In UDT1, I would model this problem using the following world program. (For those not familiar with programming convention, 0=False, and 1=True.)

def P(i):
E = (Pi(i) == 0)
D = Omega_Predict(S, i, "box contains $1M")
if D ^ E:
C = S(i, "box contains $1M")
payout = 1001000 - C * 1000 + E * 1e9
else:
C = S(i, "box is empty")
payout = 1000 - C * 1000 + E * 1e9

We then ask, what function S maximizes the expected payout at the end of P? When S sees "box is empty" clearly it should return 0. What should it do when it sees "box contains $1M"?

If it returns 0 (i.e. two-boxes), then

  • with probability .1, E=1, D^E=1, and payout = 1e9 + 1001000,
  • with probability .9, E=0, D^E=0, and payout = 1000

If it returns 1 (i.e. one-boxes), then

  • with probability .1, E=1, D^E=0, and payout = 1e9 + 1000,
  • with probability .9, E=0, D^E=1, and payout = 1000000

So returning 1 maximizes expected payout. If S=UDT1, then whenever it's called, it performs the above computation to determine what the optimal S* is, then returns the same value that S* would given that input.

The updateless part of the solution is that when determining the counterfactual dependencies that are necessary to find the optimal S*, UDT1 doesn't look at its input, so that even when called with "box contains $1M", it still doesn't "know" that D^E=1, in which case E is clearly independent of what it returns.

Comment author: Gary_Drescher 28 February 2010 04:10:30PM *  2 points [-]

That's very elegant! But the trick here, it seems to me, lies in the rules for setting up the world program in the first place.

First, the world-program's calling tree should match the structure of TDT's graph, or at least match the graph's (physically-)causal links. The physically-causal part of the structure tends to be uncontroversial, so (for present purposes) I'm ok with just stipulating the physical structure for a given problem.

But then there's the choice to use the same variable S in multiple places in the code. That corresponds to a choice (in TDT) to splice in a logical-dependency link from the Platonic decision-computation node to other Platonic nodes. In both theories, we need to be precise about the criteria for this dependency. Otherwise, the sense of dependency you're invoking might turn out to be wrong (it makes the theory prescribe incorrect decisions) or question-begging (it implicitly presupposes an answer to the key question that the theory itself is supposed to figure out for us, namely what things are or are not counterfactual consequences of the decision-computation).

So the question, in UDT1, is: under what circumstances do you represent two real-world computations as being tied together via the same variable in a world-program?

That's perhaps straightforward if S is implemented by literally the same physical state in multiple places. But as you acknowledge, you might instead have distinct Si's that diverge from one another for some inputs (though not for the actual input in this case). And the different instances need not have the same physical substrate, or even use the same algorithm, as long as they give the same answers when the relevant inputs are the same, for some mapping between the inputs and between the outputs of the two Si's. So there's quite a bit of latitude as to whether to construe two computations as "logically equivalent".

So, for example, for the conventional transparent-boxes problem, what principle tells us to formulate the world program as you proposed, rather than having:

def P1(i):
const S1;
E = (Pi(i) == 0)
D = Omega_Predict(S1, i, "box contains $1M")
if D ^ E:
C = S(i, "box contains $1M")
payout = 1001000 - C * 1000
else:
C = S(i, "box is empty")
payout = 1000 - C * 1000

(along with a similar program P2 that uses constant S2, yielding a different output from Omega_Predict)?

This alternative formulation ends up telling us to two-box. In this formulation, if S and S1 (or S and S2) are in fact the same, they would (counterfactually) differ if a different answer (than the actual one) were output from S—which is precisely what a causalist asserts. (A similar issue arises when deciding what facts to model as “inputs” to S—thus forbidding S to “know” those facts for purposes of figuring out the counterfactual dependencies—and what facts to build instead into the structure of the world-program, or to just leave as implicit background knowledge.)

So my concern is that UDT1 may covertly beg the question by selecting, among the possible formulations of the world-program, a version that turns out to presuppose an answer to the very question that UDT1 is intended to figure out for us (namely, what counterfactually depends on the decision-computation). And although I agree that the formulation you've selected in this example is correct and the above alternative formulation isn't, I think it remains to explain why.

(As with my comments about TDT, my remarks about UDT1 are under the blanket caveat that my grasp of the intended content of the theories is still tentative, so my criticisms may just reflect a misunderstanding on my part.)

Comment author: Eliezer_Yudkowsky 07 February 2010 02:40:08AM 3 points [-]

Okay, then we have a logical link from C-platonic to D-platonic, and causal links descending from C-platonic to C-physical, E-platonic to E-physical, and D-platonic to D-physical to F-physical = D-physical xor E-physical. The idea being that when we counterfactualize on C-platonic, we update D-platonic and its descendents, but not E-platonic or its descendents.

I suppose that as written, this requires a rule, "for purposes of computing counterfactuals, keep in the causal graph rather than the logical knowledge base, any mathematical knowledge gained by observing a fact descended from your decision-output or any logical implications of your decision-output". I could hope that this is a special case of something more elegant, but it would only be hope.

Comment author: Gary_Drescher 07 February 2010 12:33:27PM 2 points [-]

Ok. I think it would be very helpful to sketch, all in one place, what TDT2 (i.e., the envisioned avenue-2 version of TDT) looks like, taking care to pin down any needed sense of "dependency". And similarly for TDT1, the avenue-1 version. (These suggestions may be premature, I realize.)

Comment author: Eliezer_Yudkowsky 07 February 2010 12:18:43AM 1 point [-]

In my view, the chief form of "dependence" that needs to be discriminated is inferential dependence and causal dependence. If earthquakes cause burglar alarms to go off, then we can infer an earthquake from a burglar alarm or infer a burglar alarm from an earthquake. Logical reasoning doesn't have the kind of directionality that causation does - or at least, classical logical reasoning does not - there's no preferred form between ~A->B, ~B->A, and A \/ B.

The link between the Platonic decision C and the physical decision D might be different from the link between the physical decision D and the physical observation F, but I don't know of anything in the current theory that calls for treating them differently. They're just directional causal links. On the other hand, if C mathematically implies a decision C-2 somewhere else, that's a logical implication that ought to symmetrically run backward to ~C-2 -> ~C, except of course that we're presumably controlling/evaluating C rather than C-2.

Thinking out loud here, the view is that your mathematical uncertainty ought to be in one place, and your physical uncertainty should be built on top of your mathematical uncertainty. The mathematical uncertainty is a logical graph with symmetric inferences, the physical uncertainty is a directed acyclic graph. To form controlling counterfactuals, you update the mathematical uncertainty, including any logical inferences that take place in mathland, and watch it propagate downward into the physical uncertainty. When you've already observed facts that physically depend on mathematical decisions you control but you haven't yet made and hence whose values you don't know, then those observations stay in the causal, directed, acyclic world; when the counterfactual gets evaluated, they get updated in the Pearl, directional way, not the logical, symmetrical inferential way.

Comment author: Gary_Drescher 07 February 2010 12:43:23AM *  2 points [-]

The link between the Platonic decision C and the physical decision D

No, D was the Platonic simulator. That's why the nature of the C->D dependency is crucial here.

Comment author: Eliezer_Yudkowsky 06 February 2010 09:33:21PM 1 point [-]

Or would you try to build one big graph that encompasses physical and logical facts alike, and then use Pearl's decision procedure without further modification?

I definitely want one big graph if I can get it.

Wait, isn't it decision-computation C—rather than simulation D—whose “effect” (in the sense of logical consequence) on E we're concerned about here?

Sorry, yes, C.

Even with the node structure you suggest, we can still infer E from C and from the physical node that matches (D xor E)—unless the new rule prohibits relying on that physical node, which I guess is the idea. But what exactly is the prohibition? Are we forbidden to infer any mathematical fact from any physical indicator of that fact?

No, but whenever we see a physical fact F that depends on a decision C/D we're still in the process of making plus Something Else (E), then we express our uncertainty in the form of a causal graph with directed arrows from C to D, D to F, and E to F. Thus when we compute a counterfactual on C, we find that F changes, but E does not.

Comment author: Gary_Drescher 07 February 2010 12:02:10AM 2 points [-]

No, but whenever we see a physical fact F that depends on a decision C/D we're still in the process of making plus Something Else (E),

Wait, F depends on decision computation C in what sense of “depends on”? It can't quite be the originally defined sense (quoted from your email near the top of the OP), since that defines dependency between Platonic computations, not between a Platonic computation and a physical fact. Do you mean that D depends on C in the original sense, and F in turn depends on D (and on E) in a different sense?

then we express our uncertainty in the form of a causal graph with directed arrows from C to D, D to F, and E to F.

Ok, but these arrows can't be used to define the relevant sense of dependency above, since the relevant sense of dependency is what tells us we need to draw the arrows that way, if I understand correctly.

Sorry to keep being pedantic about the meaning of “depends”; I know you're in thinking-out-loud mode here. But the theory gives wildly different answers depending (heh) on how that gets pinned down.

Comment author: Eliezer_Yudkowsky 05 February 2010 08:11:26PM 9 points [-]

Logical uncertainty has always been more difficult to deal with than physical uncertainty; the problem with logical uncertainty is that if you analyze it enough, it goes away. I've never seen any really good treatment of logical uncertainty.

But if we depart from TDT for a moment, then it does seem clear that we need to have causelike nodes corresponding to logical uncertainty in a DAG which describes our probability distribution. There is no other way you can completely observe the state of a calculator sent to Mars and a calculator sent to Venus, and yet remain uncertain of their outcomes yet believe the outcomes are correlated. And if you talk about error-prone calculators, two of which say 17 and one of which says 18, and you deduce that the "Platonic answer" was probably in fact 17, you can see that logical uncertainty behaves in an even more causelike way than this.

So, going back to TDT, my hope is that there's a neat set of rules for factoring our logical uncertainty in our causal beliefs, and that these same rules also resolve the sort of situation that you describe.

If you consider the notion of the correlated error-prone calculators, two returning 17 and one returning 18, then the most intuitive way to handle this would be to see a "Platonic answer" as its own causal node, and the calculators as error-prone descendants. I'm pretty sure this is how my brain is drawing the graph, but I'm not sure it's the correct answer; it seems to me that a more principled answer would involve uncertainty about which mathematical fact affects each calculator - physically uncertain gates which determine which calculation affects each calculator.

For the (D xor E) problem, we know the behavior we want the TDT calculation to exhibit; we want (D xor E) to be a descendant node of D and E. If we view the physical observation of $1m as telling us the raw mathematical fact (D xor E), and then perform mathematical inference on D, we'll find that we can affect E, which is not what we want. Conversely if we view D as having a physical effect, and E as having a physical effect, and the node D xor E as a physical descendant of D and E, we'll get the behavior we want. So the question is whether there's any principled way of setting this up which will yield the second behavior rather than the first, and also, presumably, yield epistemically correct behavior when reasoning about calculators and so on.

That's if we go down avenue (2). If we go down avenue (1), then we give primacy to our intuition that if-counterfactually you make a different decision, this logically controls the mathematical fact (D xor E) with E held constant, but does not logically control E with (D xor E) held constant. While this does sound intuitive in a sense, it isn't quite nailed down - after all, D is ultimately just as constant as E and (D xor E), and to change any of them makes the model equally inconsistent.

These sorts of issues are something I'm still thinking through, as I think I've mentioned, so let me think out loud for a bit.

In order to observe anything that you think has already been controlled by your decision - any physical thing in which a copy of D has already played a role - then (leaving aside the question of Omega's strategy that simulated alternate versions of you to select a self-consistent problem, and whether this introduces conditional-strategy-dependence rather than just decision-dependence into the problem) there have to be other physical facts which combine with D to yield our observation.

Some of these physical facts may themselves be affected by mathematical facts, like an implemented computation of E; but the point is that in order to have observed anything controlled by D, we already had to draw a physical, causal diagram in which other nodes descended from D.

So suppose we introduce the rule that in every case like this, we will have some physical node that is affected by D, and if we can observe info that depends on D in any way, we'll view the other mathematical facts as combining with D's physical node. This is a rule that tells us not to draw the diagram with a physical node being determined by the mathematical fact D xor E, but rather to have a physical node determined by D, and then a physical descendent D xor E. (Which in this particular problem should descend from a physical node E that descends from the mathematical fact E, because the mathematical fact E is correlated with our uncertainty about other things, and a factored causal graph should have no remaining correlated sources of background uncertainty; but if E didn't correlate to anything else in particular, we could just have D descending to (D xor E) via the (xor with E) rule.)

When I evaluate this proposed solution for ad-hoc-ness, it does admittedly look a bit ad-hoc, but it solves at least one other problem than the one I started with, and which I didn't think of until now. Suppose Omega tells me that I make the same decision in the Prisoner's Dilemma as Agent X. This does not necessarily imply that I should cooperate with Agent X. X and I could have made the same decision for different (uncorrelated) reasons, and Omega could have simply found out by simulating the two of us that X and I gave the same response. In this case, presumably defecting; but if I cooperated, X wouldn't do anything differently. X is just a piece of paper with "Defect" written on it.

If I draw a causal diagram of how I came to learn this correlation from Omega, and I follow the rule of always drawing a causal boundary around the mathematical fact D as soon as it physically affects something, then, given the way Omega simulated both of us to observe the correlation, I see that D and X separately physically affected the correlation-checker node.

On the other hand, if I can analyze the two pieces of code D and X and see that they return the same output, without yet knowing the output, then this knowledge was obtained in a way that doesn't involve D producing an output, so I don't have to draw a hard causal boundary around that output.

If this works, the underlying principle that makes it work is something along the lines of "for D to control X, the correlation between our uncertainty about D and X has to emerge in a way that doesn't involve anyone already computing D". Otherwise D has no free will (said firmly tongue-in-cheek). I am not sure that this principle has any more elegant expression than the rule, "whenever, in your physical model of the universe, D finishes computing, draw a physical/causal boundary around that finished computation and have other things physically/causally descend from it".

If this principle is violated then D ends up "correlated" to all sorts of other things we observe, like the price of fish and whether it's raining outside, via the magic of xor.

Comment author: Gary_Drescher 06 February 2010 04:27:33PM 3 points [-]

If we go down avenue (1), then we give primacy to our intuition that if-counterfactually you make a different decision, this logically controls the mathematical fact (D xor E) with E held constant, but does not logically control E with (D xor E) held constant. While this does sound intuitive in a sense, it isn't quite nailed down - after all, D is ultimately just as constant as E and (D xor E), and to change any of them makes the model equally inconsistent.

I agree this sounds intuitive. As I mentioned earlier, though, nailing this down is tantamount to circling back and solving the full-blown problem of (decision-supporting) counterfactual reasoning: the problem of how to distinguish which facts to “hold fixed”, and which to “let vary” for consistency with a counterfactual antecedent.

In any event, is the idea to try to build a separate graph for math facts, and use that to analyze “logical dependency” among the Platonic nodes in the original graph, in order to carry out TDT's modified “surgical alteration” of the original graph? Or would you try to build one big graph that encompasses physical and logical facts alike, and then use Pearl's decision procedure without further modification?

If we view the physical observation of $1m as telling us the raw mathematical fact (D xor E), and then perform mathematical inference on D, we'll find that we can affect E, which is not what we want.

Wait, isn't it decision-computation C—rather than simulation D—whose “effect” (in the sense of logical consequence) on E we're concerned about here? It's the logical dependents of C that get surgically altered in the graph when C gets surgically altered, right? (I know C and D are logically equivalent, but you're talking about inserting a physical node after D, not C, so I'm a bit confused.)

I'm having trouble following the gist of avenue (2) at the moment. Even with the node structure you suggest, we can still infer E from C and from the physical node that matches (D xor E)—unless the new rule prohibits relying on that physical node, which I guess is the idea. But what exactly is the prohibition? Are we forbidden to infer any mathematical fact from any physical indicator of that fact? Or is there something in particular about node (D xor E) that makes it forbidden? (It would be circular to cite the node's dependence on C in the very sense of "dependence" that the new rule is helping us to compute.)

Comment author: Tyrrell_McAllister 05 February 2010 05:40:45PM *  0 points [-]

The proportion of two-boxer simulations that end up with the digit equal to zero is no different than the proportion of one-boxer simulations that end up with the digit equal to zero (both are approximately .1). But the proportion of the one-boxer simulations that end up with an actual $1M is much higher (.9) than the proportion of two-boxer simulations that end up with an actual $1M (.1).

But the proportion of two-boxers that saw $1M in the box that end up

  • with their digit being 0 and
  • with the $1M

is even higher (1). I already saw the $1M, so, by two-boxing, aren't I just choosing to be one of those who see their E module output True?

Comment author: Gary_Drescher 05 February 2010 07:17:00PM 2 points [-]

I already saw the $1M, so, by two-boxing, aren't I just choosing to be one of those who see their E module output True?

Not if a counterfactual consequence of two-boxing is that the large box (probably) would be empty (even though in fact it is not empty, as you can already see).

That's the same question that comes up in the original transparent-boxes problem, of course. We probably shouldn't try to recap that whole debate in the middle of this thread. :)

Comment author: Eliezer_Yudkowsky 05 February 2010 01:16:44AM 7 points [-]

And this was my reply:

This is an unfinished part of the theory that I've also thought about, though your example puts it very crisply (you might consider posting it to LW?)

My current thoughts on resolution tend to see two main avenues:

1) Construct a full-blown DAG of math and Platonic facts, an account of which mathematical facts make other mathematical facts true, so that we can compute mathematical counterfactuals.

2) Treat differently mathematical knowledge that we learn by genuinely mathematical reasoning and by physical observation. In this case we know (D xor E) not by mathematical reasoning, but by physically observing a box whose state we believe to be correlated with D xor E. This may justify constructing a causal DAG with a node descending from D and E, so a counterfactual setting of D won't affect the setting of E.

Currently I'd say that (2) looks like the better avenue. Can you come up with an improper mathematical dependency where E is inferred from D, and shouldn't be seen as counterfactually affected, based on mathematical reasoning only without postulating the observation of a physical variable that descends from both E and D?

Incidentally, note that an unsolvable problem that should stay unsolvable is as follows: I'm asked to pick red or green, and told "A simulation of you given this information as well picked the wrong color and got shot." Whichever choice I make, I deduce that the other choice was better. The exact details here will depend on how I believe the simulator chose to tell me this, but ceteris paribus it's an unsolvable problem.

Comment author: Gary_Drescher 05 February 2010 06:24:51PM *  5 points [-]

2) Treat differently mathematical knowledge that we learn by genuinely mathematical reasoning and by physical observation. In this case we know (D xor E) not by mathematical reasoning, but by physically observing a box whose state we believe to be correlated with D xor E. This may justify constructing a causal DAG with a node descending from D and E, so a counterfactual setting of D won't affect the setting of E.

Perhaps I'm misunderstanding you here, but D and E are Platonic computations. What does it mean to construct a causal DAG among Platonic computations? [EDIT: Ok, I may understand that a little better now; see my edit to my reply to (1).] Such a graph links together general mathematical facts, so the same issues arise as in (1), it seems to me: Do the links correspond to logical inference, or something else? What makes the graph acyclic? Is mathematical causality even coherent? And if you did have a module that can detect (presumably timeless) causal links among Platonic computations, then why not use that module directly to solve your decision problems?

Plus I'm not convinced that there's a meaningful distinction between math knowledge that you gain by genuine math reasoning, and math knowledge that you gain by physical observation.

Let's say, for instance, that I feed a particular conjecture to an automatic theorem prover, which tells me it's true. Have I then learned that math fact by genuine mathematical reasoning (performed by the physical computer's Platonic abstraction)? Or have I learned it by physical observation (of the physical computer's output), and hence be barred from using that math fact for purposes of TDT's logical-dependency-detection? Presumably the former, right? (Or else TDT will make even worse errors.)

But then suppose the predictor has simulated the universe sufficiently to establish that U (the universe's algorithm, including physics and initial conditions) leads to there being $1M in the box in this situation. That's a mathematical fact about U, obtained by (the simulator's) mathematical reasoning. Let's suppose that when the predictor briefs me, the briefing includes mention of this mathematical fact. So even if I keep my eyes closed and never physically see the $1M, I can rely instead on the corresponding mathematically derived fact.

(Or more straightforwardly, we can view the universe itself as a computer that's performing mathematical reasoning about how U unfolds, in which case any physical observation is intrinsically obtained by mathematical reasoning.)

Comment author: Eliezer_Yudkowsky 05 February 2010 01:16:44AM 7 points [-]

And this was my reply:

This is an unfinished part of the theory that I've also thought about, though your example puts it very crisply (you might consider posting it to LW?)

My current thoughts on resolution tend to see two main avenues:

1) Construct a full-blown DAG of math and Platonic facts, an account of which mathematical facts make other mathematical facts true, so that we can compute mathematical counterfactuals.

2) Treat differently mathematical knowledge that we learn by genuinely mathematical reasoning and by physical observation. In this case we know (D xor E) not by mathematical reasoning, but by physically observing a box whose state we believe to be correlated with D xor E. This may justify constructing a causal DAG with a node descending from D and E, so a counterfactual setting of D won't affect the setting of E.

Currently I'd say that (2) looks like the better avenue. Can you come up with an improper mathematical dependency where E is inferred from D, and shouldn't be seen as counterfactually affected, based on mathematical reasoning only without postulating the observation of a physical variable that descends from both E and D?

Incidentally, note that an unsolvable problem that should stay unsolvable is as follows: I'm asked to pick red or green, and told "A simulation of you given this information as well picked the wrong color and got shot." Whichever choice I make, I deduce that the other choice was better. The exact details here will depend on how I believe the simulator chose to tell me this, but ceteris paribus it's an unsolvable problem.

Comment author: Gary_Drescher 05 February 2010 05:03:45PM *  5 points [-]

1) Construct a full-blown DAG of math and Platonic facts, an account of which mathematical facts make other mathematical facts true, so that we can compute mathematical counterfactuals.

“Makes true” means logically implies? Why would that graph be acyclic? [EDIT: Wait, maybe I see what you mean. If you take a pdf of your beliefs about various mathematical facts, and run Pearl's algorithm, you should be able to construct an acyclic graph.]

Although I know of no worked-out theory that I find convincing, I believe that counterfactual inference (of the sort that's appropriate to use in the decision computation) makes sense with regard to events in universes characterized by certain kinds of physical laws. But when you speak of mathematical counterfactuals more generally, it's not clear to me that that's even coherent.

Plus, if you did have a general math-counterfactual-solving module, why would you relegate it to the logical-dependency-finding subproblem in TDT, and then return to the original factored causal graph? Instead, why not cast the whole problem as a mathematical abstraction, and then directly ask your math-counterfactual-solving module whether, say, (Platonic) C's one-boxing counterfactually entails (Platonic) $1M? (Then do the argmax over the respective math-counterfactual consequences of C's candidate outputs.)

Comment author: whpearson 05 February 2010 01:44:02AM *  0 points [-]

Have some Omega thought experiments been one shot, never to be repeated type deals or is my memory incorrect?

Yes I wasn't thinking through what would happen when the ith digit wasn't 0. You can't switch to one boxing in that case because you don't know when that would be, or rather when you see an empty box you are forced to do the same as when you see a full box due to the way the game is set up.

Comment author: Gary_Drescher 05 February 2010 03:23:45PM 1 point [-]

Have some Omega thought experiments been one shot, never to be repeated type deals or is my memory incorrect?

Yes, and that's the intent in this example as well. Still, it can be useful to look at the expected distribution of outcomes over a large enough number of trials that have the same structure, in order to infer the (counterfactual) probabilities that apply to a single trial.

Comment author: JGWeissman 05 February 2010 03:25:52AM 0 points [-]

I drew a causal graph of this scenario (with the clarification you just provided), and in order to see the problem with TDT you describe, I would have to follow a causation arrow backwards, like in Evidential Decision Theory, which I don't think is how TDT handles counterfactuals.

Comment author: Gary_Drescher 05 February 2010 03:11:51PM 1 point [-]

The backward link isn't causal. It's a logical/Platonic-dependency link, which is indeed how TDT handles counterfactuals (i.e., how it handles the propagation of "surgical alterations" to the decision node C).

View more: Prev | Next

Powered by Reddit Powered by Reddit