= 27d567bc2d48d9f40e9ccc20ea370b15)
I don't understand what you mean by "a constraint solver that seeks a proof of AIXI's observations when they are called for." Can you explain it further?
A proof system that starts with some axioms describing the physical world (excluding the AIXI machine itself), and when run with input a_1 .. a_m being AIXI's actions so far, plugs them in as axioms about AIXI's control wires, and attempts to prove a statement of the form 'AIXI's input wire observes a 1 at time t' or 'AIXI's input wire observes a 0 at time t'. And returns the first answer it finds.
I think it's implicit in the Newcomb's problem scenario that it takes place within the constraints of the universe as we know it. Obviously we have to make an exception for AIXI itself, but I don't see a reason to make any further exceptions after that point. Additionally, it is explicitly stated in the problem setup that the contents of the box are supposed to be predetermined, and that the agent is made aware of this aspect of the setup. As far as the epistemic states are concerned, this would imply that AIXI has been presented with a number of prior observations that provide very strong evidential support for this fact.
I agree that AIXI's universe programs are general Turing machines rather than explicit physics simulations, but I don't think that's a particularly big problem. Unless we're talking about a particularly immature AIXI agent, it should already be aware of the obvious physics-like nature of the real world; it seems to me that the majority of AIXI's probability mass should be occupied by physics-like Turing machines rather than by thunking. Why would AIXI come up with world programs that involve Omega making money magically appear or disappear after being presented significant evidence to the contrary?
I can agree that in the general case it would be rather difficult indeed to predict AIXI, but in many specific instances I think it's rather straightforward. In particular, I think Newcomb's problem is one of those cases.
I guess that in general Omega could be extremely complex, but unless there is a reason Omega needs to be that complex, isn't it much more sensible to interpret the problem in a way that is more likely to comport with our knowledge of reality? Insofar as there exist simpler explanations for Omega's predictive power, those simpler explanations should be preferred.
I guess you could say that AIXI itself cannot exist in our reality and so we need to reinterpret the problem in that context, but that seems like a flawed approach to me. After all, the whole point of AIXI is to reason about its performance relative to other agents, so I don't think it makes sense to posit a different problem setup for AIXI than we would for any other agent.
I agree that AIXI's universe programs are general Turing machines rather than explicit physics simulations, but I don't think that's a particularly big problem. Unless we're talking about a particularly immature AIXI agent, it should already be aware of the obvious physics-like nature of the real world; it seems to me that the majority of AIXI's probability mass should be occupied by physics-like Turing machines rather than by thunking. Why would AIXI come up with world programs that involve Omega making money magically appear or disappear after being presented significant evidence to the contrary?
The problem is not "programs that make money magically (dis)appear for the box after the fact" but rather programs that don't explicitly represent the presence or nonpresence of money at all until it is known. For example, a constraint solver that seeks a proof of AIXI's observations when they are called for (using a logic that expresses normal physics). This gives all the right answers, and is fairly simple but does allow the content of the box to be controlled by the decision.
Precisely.
Besides that, if you can't even make a reasoned guess as to what AIXI would do in a given situation, then AIXI itself is pretty useless even as a theoretical concept, isn't it?
Omega doesn't have to actually evaluate the AIXI formula exactly; it can simply reason logically to work out what AIXI will do without performing those calculations. Sure, AIXI itself can't take those shortcuts, but Omega most definitely can. As such, there is no need for Omega to perform hypercomputation, because it's pretty easy to establish AIXI's actions to a very high degree of accuracy using the arguments I've put forth above. Omega doesn't have to be a "perfect predictor" at all.
In this case, AIXI is quite easily able to predict the chain of reasoning Omega takes, and so it can easily work out what the contents of the box are. This straightforwardly results in AIXI two-boxing, and because it's so straightforward it's quite easy for Omega to predict this, and so Omega only fills one box.
The problem with AIXI is not that it preferentially self-identifies with the AIXI formula inside the robot that picks boxes vs the "AIXI formula inside Omega". The problem with AIXI is that it doesn't self-identify with the AIXI formula at all.
One could argue that the simple predictor is "punishing" AIXI for being AIXI, but this is really just the same thing as the CDT agent who thinks Omega is punishing them for being "rational". The point of this example is that if the AIXI algorithm were to output "one-box" instead of "two-box" for Newcomb's problem, then it would get a million dollars. Instead, it only gets $1000.
Well, to make an object-level observation, it's not entirely clear to me what it means for AIXI to occupy the epistemic state required by the problem definition. The "hypotheses" of AIXI are general sequence predictor programs rather than anything particularly realist. So while present program state can only depend on AIXI's past actions, and not future actions, nothing stops a hypothesis from including a "thunk" that is only evaluated when the program receives the input describing AIXI's actual action. In fact, as long as no observations or rewards depend on the missing information, there's no need to even represent the "actual" contents of the boxes. Whether that epistemic state falls within the problem's precondition seems like a matter of definition.
If you restrict AIXI's hypothesis state to explicit physics simulations (with the hypercomputing part of AIXI treated as a black box, and decision outputs monkeypatched into a simulated control wire), then your argument does follow, I think; the whole issue of Omega's prediction is just seen as some "physics stuff" happening, where Omega "does some stuff" and then fills the boxes, and AIXI then knows what's in the boxes and it's a simple decision to take both boxes.
But, if the more complicated "lazily-evaluating" sort of hypotheses gain much measure, then AIXI's decision starts actually depending on its simulation of Omega, and then the above argument doesn't really work and trying to figure out what actually happens could require actual simulation of AIXI or at least examination of the specific hypothesis space AIXI is working in.
So I suppose there's a caveat to my post above, which is that if AIXI is simulating you, then it's not necessarily so easy to "approximately guess" what AIXI would do (since it might depend on your approximate guess...). In that way, having logically-omniscient AIXI play kind of breaks the Newcomb's Paradox game, since it's not so easy to make Omega the "perfect predictor" he needs to be, and you maybe need to think about how Omega actually works.
Then it can, for experiment' sake, take 2 boxes if theres something in the first box, and take 1 otherwise. The box contents are supposedly a result of computing AIXI and as such are not computable; or for a bounded approximation, not approximable. You're breaking your own hypothetical and replacing the predictor (which would have to perform hypercomputation) with something that incidentally coincides. AIXI responds appropriately.
edit: to stpop talking to one another: AIXI does not know if there's money in the first box. The TM where AIXI is 1boxing is an entireliy separate TM from one where AIXI is 2boxing. AIXI does not in any way represent any facts about the relation between those models, such as 'both have same thing in the first box'.
edit2: and , it is absoloutely correct to take 2 boxes if you don't know anything about the predictor. AIXI represents the predictor as the surviving TMs using the choice action value as omega's action to put/not put money in the box. AIXI does not preferentially self identify with the AIXI formula inside the robot that picks boxes, over AIXI formula inside 'omega'.
If you have to perform hypercomputation to even approximately guess what AIXI would do, then this conversation would seem like a waste of time/
This quote implies a connection from "people react less strongly to emotional expressions in a foreign language" to "dilemmas in a foreign language don't touch the very core of our moral being". Furthermore, it connects or equates being more willing to sacrifice one person for five and "touch[ing] the core of our moral being" less. All rational people should object to the first implication, and most should object to the second one. This is a profoundly anti-rational quote, not a rationality quote.
I think you're reading a lot into that one sentence. I assumed that just to mean "there should not be inconsistencies due to irrelevant aspects like the language of delivery". Followed by a sound explanation for the unexpected inconsistency in terms of system 1 / system 2 thinking.
(The final paragraph of the article begins with "Our research does not show which choice is the right one.")
No. BOT^CDT = DefectBot. It defects against any opponent. CDT could not cause it to cooperate by changing what it does.
If it cooperated, it would get CC instead of DD.
Actually if CDT cooperated against BOT^CDT it would get $3^^^3. You can prove all sorts of wonderful things once you assume a statement that is false.
Depending on the exact setup, "irrelevant details in memory" are actually vital information that allow you to distinguish whether you are "actually playing" or are being simulated in BOT's mind.
OK... So UDT^Red and UDT^Blue are two instantiations of UDT that differ only in irrelevant details. In fact the scenario is a mirror matchup, only after instantiation one of the copies was painted red and the other was painted blue. According to what you seem to be saying UDT^Red will reason:
Well I can map different epistemic states to different outputs, I can implement the strategy cooperate if you are painted blue and defect if you are painted red.
Of course UDT^Blue will reason the same way and they will fail to cooperate with each other.
Well I can map different epistemic states to different outputs, I can implement the strategy cooperate if you are painted blue and defect if you are painted red. Of course UDT^Blue will reason the same way and they will fail to cooperate with each other.
No, because that's a silly thing to do in this scenario. For one thing, UDT will see that they are reasoning the same way (because they are selfish and only consider "my color" vs "other color"), and therefore will both do the same thing. But also, depending on the setup, UDT^Red's prior should give equal probability to being painted red and painted blue anyway, which means trying to make the outcome favour red is silly.
Compare to the version of newcomb's where the bot in the room is UDT^Red, while Omega simulates UDT^Blue. UDT can implement the conditional strategy {Red => two-box, Blue => one-box}. This is obviously unlikely, because the point of the Newcomb thought experiment is that Omega simulates (or predicts) you. So he would clearly try to avoid adding such information that "gives the game away".
However in this scenario you say that BOT simulates UDT "by coincidence", not by mind reading. So it is far more likely that BOT simulates (the equivalent) of UDT^Blue, while the UDT actually playing is UDT^Red. And you are passed the code of BOT as input, so UDT can simply implement the conditional strategy {cooperate iff the color inside BOT is the same as my color}.
It's hard to see how this doesn't count as "reading your mind".
So... UDT's source code is some mathematical constant, say 1893463. It turns out that UDT does worse against BOT^1893463. Note that it does worse against BOT^1893463 not BOT^{you}. The universe does not depend on the source code of the person playing the game (as it does in mirror PD). Furthermore, UDT does not control the output of its environment. BOT^1893463 always cooperates. It cooperates against UDT. It cooperates against CDT. It cooperates everything.
But this isn't due to any intrinsic advantage of CDT's algorithm. It's just because they happen to be numerically inequivalent.
No. CDT does at least as well as UDT against BOT^CDT. UDT does worse when there is this numerical equivalence, but CDT does not suffer from this issue. CDT does at least as well as UDT against BOT^X for all X, and sometimes does better. In fact, if you only construct counterfactuals this way, CDT does at least as well as anything else.
An instance of UDT with literally any other epistemic state than the one contained in BOT would do just as well as CDT here.
This is silly. A UDT that believes that it is in a mirror matchup also loses. A UDT that believes it is facing Newcomb's problem does something incoherent. If you are claiming that you want a UDT that differs from the encoding in BOT because, of some irrelevant details in its memory... well then it might depend upon implementation, but I think that most attempted implementations of UDT would conclude that these irrelevant details are irrelevant and cooperate anyway. If you don't believe this then you should also think that UDT will defect in a mirror matchup if it and its clone are painted different colors.
I take it back, the scenario isn't that weird. But your argument doesn't prove what you think it does:
Consider the analogous scenario, where CDT plays against BOT = CDT(BOT). CDT clearly does the wrong thing here - it defects. If it cooperated, it would get CC instead of DD. Note that if CDT did cooperate, UDT would be able to freeload by defecting (against BOT = CDT(BOT)). But CDT doesn't care about that because the prisoner's dilemma is defined such that we don't care about freeloaders. Nevertheless CDT defects and gets a worse result than it could.
CDT does better than UDT against BOT = UDT(BOT) because UDT (correctly) doesn't care that CDT can freeload, and correctly cooperates to gain CC.
If you are claiming that you want a UDT that differs from the encoding in BOT because, of some irrelevant details in its memory...
Depending on the exact setup, "irrelevant details in memory" are actually vital information that allow you to distinguish whether you are "actually playing" or are being simulated in BOT's mind.
Actually, I think that you are misunderstanding me. UDT's current epistemic state (at the start of the game) is encoded into BOT^UDT. No mind reading involved. Just a coincidence. [Really, your current epistemic state is part of your program]
Your argument is like saying that UDT usually gets $1001000 in Newcomb's problem because whether or not the box was full depended on whether or not UDT one-boxed when in a different epistemic state.
Okay, you're saying here that BOT has a perfect copy of the UDT player's mind in its own code (otherwise how could it calculate UDT(BOT) and guarantee that the output is the same?). It's hard to see how this doesn't count as "reading your mind".
Yes, sometimes its advantageous to not control the output of computations in the environment. In this case UDT is worse off because it is forced to control both its own decision and BOT's decision; whereas CDT doesn't have to worry about controlling BOT because they use different algorithms. But this isn't due to any intrinsic advantage of CDT's algorithm. It's just because they happen to be numerically inequivalent.
An instance of UDT with literally any other epistemic state than the one contained in BOT would do just as well as CDT here.
Fine. Your opponent actually simulates what UDT would do if Omega had told it that and returns the appropriate response (i.e. it is CooperateBot, although perhaps your finite prover is unable to verify that).
Err, that's not CooperateBot, that's UDT. Yes, UDT cooperates with itself. That's the point. (Notice that if UDT defects here, the outcome is DD.)
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Took the survey. However, my answer for the probability of MWI is "Since MWI makes the same predictions as the standard interpretation, asking for the probability of MWI is meaningless. It is like asking "this glass is 50% full of water. What is the probability that it is half empty? What is the probability that it is half full?" I put 0 for the MWI question, but I'm not sure what you want for that.
For some of the other probability questions, my answer is "I don't have enough information to come up with a good estimate, and I also don't have enough information to come up with a probability that takes into account my inability to come up with a good estimate". Again, I put 0.
Also, after the test, I'm starting to get worried how you anonymize the questions. Releasing the data without a name attached is not anonymization, if the answers people give are enough to identify them.
Um, nothing complicated is required here. Consult your gut feeling and put whatever number comes up. Turning this into an actual probability is why we have calibration questions. Putting 0 is definitely a wrong way of refusing to answer the question (you can already leave the box blank, if you really want, and thereby avoid polluting other results).
As an aside, given we had at least three people with such strange interpretations of the MWI question so far, perhaps the survey should include a question asking one's opinion on map-vs-territory...
I believe that effective anonymization (apart from removing names) is in general impossible. Although, for specific analyses a reduced dataset can be effective, such as all pairwise joint histograms, which I think preserves a certain amount of anonymity depending how unique each person's answers are.