= 27d567bc2d48d9f40e9ccc20ea370b15)
private_messaging comments on Newcomblike problems are the norm - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (108)
Again, folk physics. You make your action available to your world model at the time t where t is when you take that action. You propagate the difference your action makes (to avoid re-evaluating everything). So you need back in time magic.
Let's look at the equation here: http://www.hutter1.net/ai/uaibook.htm . You have a world model that starts at some arbitrary point well in the past (e.g. big bang), which proceeds from that past into the present, and which takes the list of past actions and the current potential action as an input. Action which is available to the model of the world since it's very beginning. When evaluating potential action 'take 1 box', the model has money in the first box, when evaluating potential action 'take 2 boxes', the model doesn't have money in the first box, and it doesn't do any fancy reasoning about the relation between those models and how those models can and can't differ. It just doesn't perform this time saving optimization of 'let first box content be x, if i take 2 boxes, i get x+1000 > x'.
Why do you need "back in time magic", exactly? That's a strictly more complex world model than the non-back-in-time-magic version. If Solomonoff induction results in a belief in the existence of back-in-time magic when what's happening is just perfectly normal physics, this would be a massive failure in Solomonoff induction itself. Fortunately, no such thing occurs; Solomonoff induction works just fine.
I'm arguing that, because the box already either contains the million or does not, AIXI will (given a reasonable but not particularly large amount of evidence) massively downweight models that do not correctly describe this aspect of reality. It's not doing any kind of "fancy reasoning" or "time-saving optimization", it's simply doing Solomonoff induction, and dong it correctly.
Then it can, for experiment' sake, take 2 boxes if theres something in the first box, and take 1 otherwise. The box contents are supposedly a result of computing AIXI and as such are not computable; or for a bounded approximation, not approximable. You're breaking your own hypothetical and replacing the predictor (which would have to perform hypercomputation) with something that incidentally coincides. AIXI responds appropriately.
edit: to stpop talking to one another: AIXI does not know if there's money in the first box. The TM where AIXI is 1boxing is an entireliy separate TM from one where AIXI is 2boxing. AIXI does not in any way represent any facts about the relation between those models, such as 'both have same thing in the first box'.
edit2: and , it is absoloutely correct to take 2 boxes if you don't know anything about the predictor. AIXI represents the predictor as the surviving TMs using the choice action value as omega's action to put/not put money in the box. AIXI does not preferentially self identify with the AIXI formula inside the robot that picks boxes, over AIXI formula inside 'omega'.
If you have to perform hypercomputation to even approximately guess what AIXI would do, then this conversation would seem like a waste of time/
Precisely.
Besides that, if you can't even make a reasoned guess as to what AIXI would do in a given situation, then AIXI itself is pretty useless even as a theoretical concept, isn't it?
Omega doesn't have to actually evaluate the AIXI formula exactly; it can simply reason logically to work out what AIXI will do without performing those calculations. Sure, AIXI itself can't take those shortcuts, but Omega most definitely can. As such, there is no need for Omega to perform hypercomputation, because it's pretty easy to establish AIXI's actions to a very high degree of accuracy using the arguments I've put forth above. Omega doesn't have to be a "perfect predictor" at all.
In this case, AIXI is quite easily able to predict the chain of reasoning Omega takes, and so it can easily work out what the contents of the box are. This straightforwardly results in AIXI two-boxing, and because it's so straightforward it's quite easy for Omega to predict this, and so Omega only fills one box.
The problem with AIXI is not that it preferentially self-identifies with the AIXI formula inside the robot that picks boxes vs the "AIXI formula inside Omega". The problem with AIXI is that it doesn't self-identify with the AIXI formula at all.
One could argue that the simple predictor is "punishing" AIXI for being AIXI, but this is really just the same thing as the CDT agent who thinks Omega is punishing them for being "rational". The point of this example is that if the AIXI algorithm were to output "one-box" instead of "two-box" for Newcomb's problem, then it would get a million dollars. Instead, it only gets $1000.
Well, to make an object-level observation, it's not entirely clear to me what it means for AIXI to occupy the epistemic state required by the problem definition. The "hypotheses" of AIXI are general sequence predictor programs rather than anything particularly realist. So while present program state can only depend on AIXI's past actions, and not future actions, nothing stops a hypothesis from including a "thunk" that is only evaluated when the program receives the input describing AIXI's actual action. In fact, as long as no observations or rewards depend on the missing information, there's no need to even represent the "actual" contents of the boxes. Whether that epistemic state falls within the problem's precondition seems like a matter of definition.
If you restrict AIXI's hypothesis state to explicit physics simulations (with the hypercomputing part of AIXI treated as a black box, and decision outputs monkeypatched into a simulated control wire), then your argument does follow, I think; the whole issue of Omega's prediction is just seen as some "physics stuff" happening, where Omega "does some stuff" and then fills the boxes, and AIXI then knows what's in the boxes and it's a simple decision to take both boxes.
But, if the more complicated "lazily-evaluating" sort of hypotheses gain much measure, then AIXI's decision starts actually depending on its simulation of Omega, and then the above argument doesn't really work and trying to figure out what actually happens could require actual simulation of AIXI or at least examination of the specific hypothesis space AIXI is working in.
So I suppose there's a caveat to my post above, which is that if AIXI is simulating you, then it's not necessarily so easy to "approximately guess" what AIXI would do (since it might depend on your approximate guess...). In that way, having logically-omniscient AIXI play kind of breaks the Newcomb's Paradox game, since it's not so easy to make Omega the "perfect predictor" he needs to be, and you maybe need to think about how Omega actually works.
I think it's implicit in the Newcomb's problem scenario that it takes place within the constraints of the universe as we know it. Obviously we have to make an exception for AIXI itself, but I don't see a reason to make any further exceptions after that point. Additionally, it is explicitly stated in the problem setup that the contents of the box are supposed to be predetermined, and that the agent is made aware of this aspect of the setup. As far as the epistemic states are concerned, this would imply that AIXI has been presented with a number of prior observations that provide very strong evidential support for this fact.
I agree that AIXI's universe programs are general Turing machines rather than explicit physics simulations, but I don't think that's a particularly big problem. Unless we're talking about a particularly immature AIXI agent, it should already be aware of the obvious physics-like nature of the real world; it seems to me that the majority of AIXI's probability mass should be occupied by physics-like Turing machines rather than by thunking. Why would AIXI come up with world programs that involve Omega making money magically appear or disappear after being presented significant evidence to the contrary?
I can agree that in the general case it would be rather difficult indeed to predict AIXI, but in many specific instances I think it's rather straightforward. In particular, I think Newcomb's problem is one of those cases.
I guess that in general Omega could be extremely complex, but unless there is a reason Omega needs to be that complex, isn't it much more sensible to interpret the problem in a way that is more likely to comport with our knowledge of reality? Insofar as there exist simpler explanations for Omega's predictive power, those simpler explanations should be preferred.
I guess you could say that AIXI itself cannot exist in our reality and so we need to reinterpret the problem in that context, but that seems like a flawed approach to me. After all, the whole point of AIXI is to reason about its performance relative to other agents, so I don't think it makes sense to posit a different problem setup for AIXI than we would for any other agent.
If AIXI has been presented with sufficient evidence that the Newcomb's problem works as advertised, then it must be assigning most of its model probability mass to programs where the content of the box, however internally represented, is correlated to the next decision.
Such programs exist in the model ensemble, hence the question is how much probability mass does AIXI assign to them. If it not enough to dominate its choice, then by definition AIXI has not been presented with enough evidence.
What do you mean by "programs where the content of the box, however internally represented, is correlated to the next decision"? Do you mean world programs that output $1,000,000 when the input is "one-box" and output $1000 when the input is "two-box"? That seems to contradict the setup of Newcomb's to me; in order for Newcomb's problem to work, the content of the box has to be correlated to the actual next decision, not to counterfactual next decisions that don't actually occur.
As such, as far as I can see it's important for AIXI's probability mass to focus down to models where the box already contains a million dollars and/or models where the box is already empty, rather than models in which the contents of the box are determined by the input to the world program at the moment AIXI makes its decision.
AIXI world programs have no inputs, they just run and produce sequences of triples in the form: (action, percept, reward).
So, let's say AIXI has been just subjected to Newcomb's problem. Assuming that the decision variable is always binary ("OneBox" vs "TwoBox"), of all the programs which produce a sequence consistent with the observed history, we distinguish five classes of programs, depending on the next triple they produce:
1: ("OneBox", "Opaque box contains $1,000,000", 1,000,000)
2: ("TwoBox", "Opaque box is empty", 1,000)
3: ("OneBox", "Opaque box is empty", 0)
4: ("TwoBox", "Opaque box contains $1,000,000", 1,001,000)
5: Anything else (eg. ("OneBox", "A pink elephant appears", 42)).
Class 5 should have a vanishing probability, since we assume that the agent already knows physics.
Therefore:
E("OneBox") = (1,000,000 * p(class1) + 0 * p(class3)) / (p(class1) + p(class3))
E("TwoBox") = (1,000 * p(class2) + 1,001,000 * p(class4)) / (p(class2) + p(class4))
Classes 1 and 2 are consistent with the setup of Newcomb's problem, while classes 3 and 4 aren't.
Hence I would say that if AIXI has been presented with enough evidence to believe that it is facing Newcomb's problem, then by definition of "enough evidence", p(class1) >> p(class3) and p(class2) >> p(class4), implying that AIXI will OneBox.
EDIT: math.
The problem is not "programs that make money magically (dis)appear for the box after the fact" but rather programs that don't explicitly represent the presence or nonpresence of money at all until it is known. For example, a constraint solver that seeks a proof of AIXI's observations when they are called for (using a logic that expresses normal physics). This gives all the right answers, and is fairly simple but does allow the content of the box to be controlled by the decision.
Such models would generally not offer good explanations for why Omega is so good at predicting all those other agents who aren't AIXI, and would be penalized for this. On the other hand, any model that explains Omega's general predictive power would be made more complex by adding a special case just for AIXI.
I don't understand what you mean by "a constraint solver that seeks a proof of AIXI's observations when they are called for." Can you explain it further?
A proof system that starts with some axioms describing the physical world (excluding the AIXI machine itself), and when run with input
a_1 .. a_mbeing AIXI's actions so far, plugs them in as axioms about AIXI's control wires, and attempts to prove a statement of the form 'AIXI's input wire observes a 1 at timet' or 'AIXI's input wire observes a 0 at timet'. And returns the first answer it finds.Alternatively, what about a version of Newcomb's problem where the predictor's source code is shown to AIXI before it makes its decision?
So you don't predict anything, just put nothing in the first box, and advertise this fact clearly enough for the agent making the choice.
Newcomb's original problem did not include the clause 'by the way, there's nothing in the first box'. You're adding that clause by making additional assertions regarding what AIXI knows about "Omega".
There's a truly crazy amount of misunderstandings with regards to what Solomonoff Induction can learn about the world, on LW.
Let's say you run AIXI, letting it oversee some gigabytes of webcam data, at your location. You think AIXI can match the exact location of raindrops on your roof, hours in advance? You think AIXI is going to know all about me - the DNA I have, how may I construct a predictor, etc?
A version of the problem in which Omega is predictable is hardly the same thing as a version of the problem in which the first box is always empty. Other algorithms get the million dollars; it's just that AIXI does not. Moreover, AIXI is not being punished simply for being AIXI; AIXI not getting the million dollars is a direct consequence of the output of the AIXI algorithm.
Of course it didn't include that clause; it would be a rather stupid problem if it did include that clause. On the other hand, what is in the statement of Newcomb's problem is "By the time the game begins, and the player is called upon to choose which boxes to take, the prediction has already been made, and the contents of box B have already been determined." Moreover, it is quite clearly stated that the agent playing the game is made fully aware of this fact.
If we stipulate, for the sake of argument, that AIXI cannot work out the contents of the opaque box, AIXI still fails and two-boxes. By the problem statement AIXI should already be convinced that the contents of the boxes are predetermined. Consequently, the vast majority of weight in AIXI's distribution over world models should be held by models in which AIXI's subsequent action has no effect on the contents of the box, and so AIXI will rather straightforwardly calculate two-boxing to have higher utility. Moreover, it's easy for Omega to deduce this, and so the first box will be empty, and so AIXI gets $1000.
Setting the stipulation aside, I still think it should be pretty easy for AIXI to deduce that the box is empty. Given Omega's astounding predictive success it is far more likely that Omega has a non-trivial capacity for intelligent reasoning and uses this reasoning capacity with a goal of making accurate predictions. As such, I would be surprised if an Omega-level predictor was not able to come across the simple argument I gave above. Of course, as I said above, it doesn't really matter if AIXI can't deduce the contents of the box; AIXI two-boxes and loses either way.
No, I don't think that.
Really? I thought your predictor didn't evaluate the algorithm, so how is that a 'direct consequence'?
Yeah, and in the Turing machine provided with the tape where the action is "choose 1 box" (the tape is provided at the very beginning), the content of the box is predetermined to have 1 million, while in the entirely different Turing machine provided with the tape where the action is "choose 2 boxes", the box is predetermined to have nothing. What is so hard to get about it? Those are two entirely different Turing machines, in different iterations of the argmax loop. Are you just selectively ignoring the part of the statement where the predictor, you know, is actually being correct?
edit: as I said, it's a word problem, only suitable for sloshy and faulty word reasoning using folk physics. You end up ignoring some part of the problem statement.
The predictor doesn't have to fully evaluate the algorithm to be able to reason about the algorithm.
Nowhere in the problem statement does it say that Omega is necessarily always correct. If it's physically or logically impossible, Newcomb's problem is basically just asking "would you prefer a million dollars or a thousand dollars." The whole point of Newcomb's problem is that Omega is just very, very good at predicting you.
Anyways, I think you're misunderstanding the AIXI equation. If there are two Turing machines that are consistent with all observations to date, then both of those Turing machines would be evaluated in the one-boxing argmax iteration, and both would be evaluated in the two-boxing argmax iteration as well. There is no possible reason that either world machine would be excluded from either iteration.
As such, if in one of those Turing machines the box is predetermined to have 1 million, then it's pretty obvious that when given the input "two-box" that Turing machine will output $1,001,000. More generally there would of course be infinitely many such Turing machines, but nonetheless the expected value over those machines will be very nearly that exact amount.
What exactly is the reason you're suggesting for AIXI excluding the million-dollar Turing machines when it considers the two-boxing action? Where in the AIXI equation does this occur?
This is getting somewhere.
AIXI does S.I. multiple times using multiple machines differing in what they have on the extra actions tape (where the list of actions AIXI will ever take is written). All the machines used to evaluate the consequence of 1-boxing have different extra actions tape from all the machines used to evaluate the consequences of 2 boxing.
From
http://www.hutter1.net/ai/uaibook.htm
"where U is a universal (monotone Turing) machine executing q given a1..am."
The U used for one boxing is different U from U used for two boxing, running the same q (which can use the action from the extra tape however it wants; to alter things that happen at the big bang, if it sees fit).
With regards to the content of the boxes, there are 3 relevant types of program. One is 'there's nothing in the box', other is 'there's a million in the box', but the third, and this is where it gets interesting, is 'a bit from the extra input tape determines if there's money in the box'. Third type can in principle be privileged over repeated observation of correct prediction as it does not have to duplicate the data provided on the third tape for the predictions to be correct all the time.
The third type evaluates to money in the box when the action (provided on the actions tape, which is available to the machine from the beginning) is to take 1 box, and evaluates to no money in the box when the action is to take 2 boxes.
If AIXI learns or is pre-set to know that there's prediction of the decision happening, I take it as meaning that the third type of machine acquires sufficient weight. edit: and conversely, if the AIXI is not influenced by the program that reads from the actions tape to determine the movements of the 'predictor', I take it as AIXI being entirely ignorant of the predicting happening.
edit: clearer language regarding the extra actions tape
edit2: and to clarify further, there's machines where a bit of information in q specifies that "predictor" has/hasn't put money in the box, and there's machines where a bit in the another tape, a1...am , determines this. Because it's not doing any sort of back in time logic (the a1..am is here from the big bang), the latter are not that apriori improbable and can be learned just fine.