If I am given a thing, like a mug, I now have one more mug than I had before. My need for mugs has therefore decreased. If I am to sell the mug, I must examine how much I will need the mug after it is gone and place a price on that loss of utility. If I am buying a mug I must set a price on how much I need it after I have it and place a price on that increase of utility. If the experiment is not worded carefully then the thought process could go along the lines of...

I have 2 mugs, and often take a tea break with my mate Steve. To sell one of those mugs would make me lose out on this activity... $10. I don't hugely need another mug unless it breaks, but it is handy to have a spare... $2.

In real life people will attribute more value to their stuff than other stuff as in general they would not have got the stuff if they did not value it higher than the cost of getting it. It is not a failiure of rationality to want something more than what you paid for it, and while it is a failiure of rationality to over value something just because you own it, it is not a failiure of rationality to ask a higher price first in case the person you are selling to is willing to pay more.

It would be difficult to adjust for these factors in designing an experiment.

I have a couple of questions about this subject...

Does it still count if the AI "believes" that it needs humans when it, in fact, does not?

For example does it count if you code into the AI the belief that it is being run in a "virtual sandbox," watched by a smarter "overseer" and that if it takes out the human race in any way, then it will be shut down/tortured/highly negative utilitied by said overseer?

Just because an AI needs humans to exist, does that really mean that it won't kill them anyway?

This argument seems to be contingent on the AI wishing to live. Wishing to live is not a function of all inteligence. If an AI was smarter than anything else out there but depended on lesser, and provenly irrational beings for its continued existence this does not mean that it would want to "live" that way forever. It could either want to gain independance, or cease to exist, neither of which are necessarily healthy for its "supporting units".

Or, it could not care either way whether it lives or dies, as stopping all work on the planet is more important for slowing the entropic death of the universe.

It may be the case that an AI does not want to live reliant on "lesser beings" and sees the only way of ensuring its permanent destruction as the destruction of any being capable of creating it again, or the future possibilty of such life evolving. It may decide to blow up the universe to make extra sure of that.

Come to think of it a suicidal AI could be a pretty big problem...

Mm. I'm not really understanding your thinking here.

Until I know how and why I know it is predicting the result I am not sure how I would act in the real case.

Fair enough.

For my own part, I find that I often act on my beliefs in a situation without stopping to consider what my basis for those beliefs is, so it's not too difficult for me to imagine acting on my posited beliefs about Omega's predictive ability while ignoring the question of where those beliefs came from. I simply accept, for the sake of the exercise, that I do believe it and act accordingly.

Another way of looking at it you might find helpful is to leave aside altogether the question of what I would or wouldn't do, and what I can and can't believe, and instead ask what the right thing to do would be were this the actual situation.

E.g., if you give me a device that is indistinguishable from a revolver, but is designed in such a way that placing it to my temple and firing the trigger doesn't put a bullet in my skull but instead causes Vast Quantities of Really Good Stuff to happen, the right thing to do is put the device to my temple and fire the trigger. I won't actually do that, because I have no way of knowing what the device actually does, but whether I do it or not it's the right thing to do.

I agree that I can't imagine any justified way of coming to believe Omega has the properties that I am presumed to believe Omega to have. So, yes, the thought experiment either assumes that I've arrived at that state in some unjustified way (as you say, assume I'm irrational, at least sometimes) or that I've arrived at it in some justified way I currently have no inkling of (and therefore cannot currently imagine).

Assuming that I'm irrational sometimes, and sometimes therefore arrive at beliefs that aren't justified, isn't too difficult for me; I have a lot of experience with doing that. (Far more experience than I have with riding a trolley on a spaceship, come to that.)

But, sure, I can see where people whose experience doesn't include that, or whose self-image rejects it regardless of their experience, or who otherwise have trouble imagining themselves arriving at beliefs that aren't rationally justified, might balk at that step.

Without knowing how can one make the best choice?

If by "best choice" we mean the choice that has the best possible results, then in this case we either cannot make the best choice except by accident, or we always make the best choice, depending on whether the things that didn't in fact happen were possible before they didn't happen, which there's no particular reason to believe.

If by "best choice" we mean the choice that has the highest expected value given what we know when we make it, then we make the best choice by evaluating what we know.

Yeah, this comes up a lot.

My usual way of approaching it is to acknowledge that the thought experiment is asking me to imagine being in a particular epistemic state, and then asking me for my intuitions about what I would do, and what it would be right for me to do, given that state. The fact that the specified epistemic state is not one I can imagine reaching is beside the point.

This is common for thought experiments. If I say "suppose you're on a spaceship traveling at .999999999c, and you get in a trolley inside the ship that runs in the direction the ship is travelling at 10 m/s, how fast are you going?" it isn't helpful to reply "No such spaceship exists, so that condition can't arise." That's absolutely true, but it is beside the point.

Sorry, I'm new here, I am having trouble with the Idea that anyone would consider taking both boxes in a real world situation. How would this puzzle be modeled differently, versus how would it look differently if it were Penn and Teller flying Omega?

If Penn and Teller were flying Omega then they would have been able to produce exactly the same results as seen, without violating causality or time travelling or perfectly predicting people by just cheating and emptying the box after you choose to take both.

Given that "it's cheating" is a significantly more rational idea than "it's smart enough to predict 100 people" in terms of simplicity and results seen, why not go with that as a rational reason to pick just box B? The only reason one would take both is if it proved it was not cheating, how it could do that without also convincing me of its predictive powers I don't know, and once convinced of is predictive powers I would have to take Box B.

So taking both boxes only makes sense if you know it is not cheating, and know it can be wrong. I notice I am confused, how can you both know it is not cheating, and not know that it is correct in it's prediction.

I think that the reason this puzzle begets irrationality is that one of the fundamental things you must do to parse the puzzle is irrational, that is 'believe that the machine is not cheating', given the alternatives and no further info.

I expect it depends rather a lot on the nature of the problem, and on just what exactly we mean by "science," "magical thinking," and "Bayes".

I find, thinking about your question, that I'm not really sure what you mean by these terms. Can you give me a more concrete example of what you have in mind? That is, OK, there's a team comprising A, B, and C. What would lead me to conclude that A is a "magical thinker", B is a "Bayesian," and C is a "scientist"?

For my own part, I would say that the primary difference has to do with how evidence is evaluated.

For example, I would expect A, in practice, to examine the evidence holistically and arrive at intuitive conclusions about it, whereas I would expect B and C to examine the evidence more systematically. In a situation where the reality is highly intuitive, I would therefore expect A to arrive at the correct conclusion with confidence quickly, and B and C to confirm it eventually. In a situation where the reality is highly counterintuitive, I would expect A to arrive at the wrong conclusion with confidence quickly, while B and C become (correctly) confused.

For example, I would expect B and C, in practice, to try and set up experimental conditions under which all observable factors but two (F1 and F2) are held fixed, and F1 is varied and F2 measured and correlations between F1 and F2 calculated. In a situation where such conditions can be set up, and strong correlations are observed between certain factors, I would expect C to arrive at correct conclusions about causal links with confidence slowly, and B to confirm them even more slowly. In a situation where such conditions cannot be set up, or where no strong correlations are observed between evaluated factors, I would expect C to arrive at no positive conclusions about causal links, and B to arrive at weak positive conclusions about causal links.

Are these expectations consistent with what you mean by the terms?

Do you expect everything that's possibly enjoyable to be enjoyable immediately?

Why do people take the time to develop "aquired tastes". "That was an unpleasant experience", somehow becomes "I will keep doing it until I like it."

My guess is social conditioning, but then how did it become popular enough for that to be a factor?