Someone wishes he were Hofstadter.

Replace "matrix" with "light cone" and see if you would still endorse that.

they'll each go on international television and denounce me for being a terrible person and ask me over and over

There's not enough time.

I'd be tempted on an emotional level to say yes and give him the money

If I ever ascend into dietyhood I'll be sorely tempted to go around offering Pascal's wager in various forms and inverting the consequences from what i said, or having no other consequences no matter what they do.

"Accept Christianity, and have a chance of heaven." (Create heaven only for those who decline my Wager.)

"Give me five dollars, or I will torture some people" (Only torture people if they give me five dollars.)

Check the multiverse to see how many beings will threaten people with Pascal's wager. Create 3^^(a large number of up arrows)^^^3 unique beings for each philosopher con man. Ask each new being: "Give me five dollars, or I will torture some people" (Do nothing. Let them live out normal lives with the benefit of their money. [Don't worry, for each such reply I will add five dollars worth of goods and services to their world, to avoid deflation and related issues.])

Why everyone is assuming the probability they are confronting a trickster testing them is zero, or that it is in any case a smaller probability than something different that they can't get a handle on because it is too small, I have no idea.

Since people are so taken with only taking beings at their word, wouldn't a being telling them it will trick them if it gets the power confound them?

Maybe I'm missing the point here, but why do we care about any number of simulated "people" existing outside the matrix at all? Even assuming that such people exist, they'll never effect me, nor effect anyone in the world I'm in. I'll never speak to them, they'll never speak to anyone I know and I'll never have to deal with any consequences for their deaths. There's no expectation that I'll be punished or shunned for not caring about people from outside the matrix, nor is there any way that these people could ever break into our world and attempt to punish me for killing them. As far as I care, they're not real people and their deaths or non-deaths do not factor into my utility function at all. Unless Pascal's mugger claims he can use his powers from outside the matrix to create 3^^^3 people in our world (the only one I care about) and then kill them here, my judgement is based soley on that fact that U(me loosing 5$) < 0.

So, let's assume that we're asking about the more interesting case and say that Pascal's mugger is instead threatening to use his magic extra-matrix powers to create 3^^^3 people here on Earth one by one and that they'll each go on international television and denounce me for being a terrible person and ask me over and over why I didn't save them and then explode into chunks of gore where everyone can see it before fading back out of the matrix (to avoid black hole concerns) and that all of this can be avoided with a single one time payment of 5$. What then?

I honestly don't know. Even one person being created and killed that way definitely feels worse than imagining any number of people outside the matrix getting killed. I'd be tempted on an emotional level to say yes and give him the money, despite my more intellectual parts saying this is clearly a setup and that something that terrible isn't actually going to happen. 3^^^3 people, while I obviously can't really imagine that many, is only worse since it will keep happening over and over and over until after the stars burn out of the sky.

The only really convincing argument, aside from the argument from absurdity ("That's stupid, he's just a philosopher out trying to make a quick buck.") is Polymeron's argument here

"When is pain worst?" an is important and deeply related question which is, fortunately for us, much easier to examine directly. I feel worse to have a papercut than it is to have an equally painful, but ultimately more damaging cut elsewhere. I feel worse to have a chronic pain that will not go away than I do when I feel a brief pain that goes away shortly after. I feel worse if I am unable to fix the injury that is causing me pain. It feels unfair, awful and even unbearable to ache for days during a bad bout of the flu. I know that the pain doesn't serve me any useful purpose, it isn't going to make me any less likely to catch the flu, which makes it all the worse. Likewise, if I've hurt my foot and it keeps on hurting even as I walk over to the cabinet to get out a bandage and some disinfectant, that is worse than a pain that hurts just as bad for a second and then stops once I've stopped doing the thing that injured me.

This seems to indicate that pain is worst when there is a conflict between my objective assessment of my injuries and what my "gut" tells me about my injuries through the proxy of pain. There was a post somewhere on this site, perhaps by Yvain and I'd thank anyone that could find it and link it, about how there is not a single unitary self, but rather many seperate selves. I suspect that the main reason why pain is bad is due to a conflict between these many parts of me. Pain is at its worst when "rational, far-mode assessor of injury" me is at odds with "hindbrain, near-mode tabulator of pain nerves" me and the former has no way to get the near-mode brain to stop sending pain signals even after my initial panic at being injured has been overridden and all useful steps towards recovery and preventing future injuries have been taken, while the latter can't keep the far-mode brain from constantly ignoring these dire warnings about how my nerves are reporting bad things and how my skin is cut and how I'm probably bleeding and how it hasn't stopped and I need to lie still and wait to heal. The two argue in circles somewhere in the back of my mind leaving me with a certain unease that I can't but at rest by either laying still and resting or by ignoring my pain and going on with my day.

That is why pain is bad. Because, on some level or another, it causes different parts of me to come into conflict, neither being able to overcome the other (and neither should, for I shudder to think of what would happen if one did win out) and neither being able to let me rest.

One method I've seen no mention of is distraction from the essence of an argument with pointless pedantry. The classical form is something along the lines of "My opponent used X as an example of Y. As an expert in X, which my opponent is not, I can assure you that X is not an example of Y. My opponent clearly has no idea how Y works and everything he says about it is wrong." which only holds true of X and Y are in the same domain of knowledge.

A good example: Eliezer said in the first paragraph that a geologist could tell a pebble from the beach from a driveway. As a geologist, I know that most geologists, myself included, honestly couldn't tell the difference. Most pebbles found in concrete, driveways and so forth are taken from rivers and beaches, so a pebble that looks like a beach pebble wouldn't be suprising to find in someone's driveway. That doesn't mean that Eliezer's point is wrong, since he could have just as easily said "a pebble from a mountaintop" or "a pebble from under the ocean" and the actual content of this post wouldn't have changed a bit.

In a more general sense, this an example of assuming an excessively convenient world to fight the enemy arguments in, but I think this specific form bears pointing out, since it's a bit less obvious than most objections of that sort.

The correct moral response to the king's sadistic choice (in any of the 4 forms mentioned) is not sacrifice yourself OR to let the other 10 die instead. The correct answer is that you, knowing the king was doing this, should have founded/joined/assisted an organization devoted to deposing the evil king and replacing him with someone who isn't going to randomly kill his subjects.

So to with charity. The answer isn't to sacrifice all of your comforts and wealth to save the lives of others, but to assist with, petition for and otherwise attempt to inact sanctions, reforms and revolutions to force the leaders of the world's most impoverished nations to end the policies that are leading to their populations starving to death. There is already enough food to feed everyone in the world twice over, it is simply a matter of making sure that nobody is prevented from obtaining it by a cruel or uncarring outside institution.

The Outcome Pump resets the universe whenever a set period of time passes without an "Accept Outcome" button being pressed to prevent the reset.

This creates a universe where the Accept Outcome button gets pressed, not necessarily one that has a positive outcome. e.g. if the button was literally a button, something might fall on to it; or if it was a state in a computer, a cosmic ray might flip a bit.

Welcome!

The Outcome Pump resets the universe whenever a set period of time passes without an "Accept Outcome" button being pressed to prevent the reset.

So no more problem if it kills you! But what if it kills you and destroys itself in the process?

Hello! quinesie here. I discovered LessWrong after being linked to HP&MoR, enjoying it and then following the links back to the LessWrong site itself. I've been reading for a while, but, as a rule, I don't sign up with a site unless I have something worth contributing. After reading Eliezer's Hidden Complexity of Wishes post, I think I have that:

In the post, Eliezer describes a device called an Outcome Pump, which resets the universe repeatedly until the desired outcome occurs. He then goes on to describe why this is a bad idea, since it can't understand what it is that you really want, in a way that is analogous to unFriendly AI being programed to naively maximize something (like paper clips) that humans say they want maximized even when they really something much more complex that they have trouble articulating well enough to describe to a machine.

My idea, then, is to take the Outcome Pump and make a 2.0 version that uses the same mechanism as the orginal Outcome Pump, but with a slightly different trigger mechanism: The Outcome Pump resets the universe whenever a set period of time passes without an "Accept Outcome" button being pressed to prevent the reset. To convert back to AI theory, the analogous AI would be one which simulates the world around it, reports the projected outcome to a human and then waits for the results to be accepted or rejected. If accepted, it implements the solution. If rejected, it goes back to the drawing board and crunches numbers until it arrives at the next non-rejected solution.

This design could of course be improved upon by adding in parameters to automatically reject outcomes with are obviously unsuitable, or which contain events, ceteris paribus, we would prefer to avoid, just as with the standard Outcome Pump and its analogue in unFriendly AI. The chief difference between the two is that the failure mode for version 2.0 isn't a catastrophic "tile the universe with paper clips/launch mother out of burning building with explosion" but rather the far more benign "submit utterly inane proposals until given more specific instructions or turned off".

This probably has some terrible flaw in it that I'm overlooking, of course, since I am not an expert in the field, but if there is, the flaws aren't obvious enough for a layman to see. Or, just as likely, someone else came up with it first and published a paper describing exactly this. So I'm asking here.