Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

# The Aliens have Landed!

33 19 May 2011 05:09PM

"General Thud! General Thud! Wake up! The aliens have landed. We must surrender!" General Thud's assistant Fred turned on the lights and opened the curtains to help Thud wake up and confront the situation. Thud was groggy because he had stayed up late supervising an ultimately successful mission carried out by remotely piloted vehicles in some small country on the other side of the world. Thud mumbled, "Aliens? How many? Where are they? What are they doing?" General Thud looked out the window, expecting to see giant tripods walking around and destroying buildings with death rays. He saw his lawn, a bright blue sky, and hummingbirds hovering near his bird feeder.

Fred was trying to bring Thud up to speed as quickly as possible. "Thousands of them, General! 2376, to be precise. They gave us a map; we know where they all are. They aren't doing anything overt, but the problem is their computation! I have one here, if you'd like to look." Fred removed a black sphere two inches in diameter from his pocket and gave it to Thud.

Thud sat on his bed holding the small sphere and staring at it dumbfounded. "Okay, you think we should surrender to a few thousand small spheres. Why is that, exactly?" The sphere seemed a little flexible in Thud's hand. As he experimented a few seconds to see just how flexible, it collapsed in his hand, converting itself into a loose clump of alien sand that landed in his lap and started to dribble onto his bed and the floor. Thud stood up and brushed the rest of the sand off of his pyjamas and bed, and thought for a moment about where he left his vacuum cleaner bags. He was not impressed with these aliens.

Fred said "I don't think you wanted to do that, sir. Their ultimatum states that for every alien we destroy, they'll manufacture two in the outer reaches of the Solar System where we'll never find them!"

Thud said, "Okay, so now you think we should surrender to 2375 small spheres, and two or more small spheres that are out of the battlefield for the moment. Why is that?"

Fred said "Well, you remember a few years back when some people copied their brain state into a computer and posted it to the Internet? Apparently somebody copied the data across an unencrypted wireless link, the aliens picked it up with their radio telescopes, and now they are simulating those poor people in these black spheres and torturing the simulations! They sent us videos!" Fred held up his cell phone, pushed a button, and showed the video to Thud.

Thud looked at the video for a moment and said, "Yep, that's torture. Do these people know anything potentially useful to the aliens?"

Fred said, "Well, they know how to break into a laboratory that has brain scanning tools and push some buttons. That was apparently the high point of their lives.  But none of that matters, the aliens don't seem to be torturing them for information anyway."

Thud was still suffering from morning brain fog. He rubbed his eyes. "And why should we surrender?"

Fred said, "The aliens have made a trillion copies of these poor people and will run the torture simulations on the little black spheres until we march all of our citizens into the death camps they demand we build! We have analyzed these black spheres and the engineering diagrams the aliens gave us, and we know this to be true. We only have ten billion citizens, and this simulated torture is much worse than simulated death, so the total utility is much greater if we surrender!"

Thud yawned.  "Fred, you're fired. Get out of my house." As Fred left, Thud closed his curtains and tried to get back to sleep.

-----

Michael said "So I take it you no longer assist Thud. What are you doing now?"

Fred reclined comfortably on the analyst's couch. "I help out at the cafeteria as a short order cook. But I'm not worried about my career right now. I have nightmares about all these simulated people being tortured in the flimsy alien spheres."

"Thud surely knows the simulations are being tortured too. Do you think he has nightmares about this?"

"No, he doesn't seem to care."

"Have you always cared about the well-being of simulations?"

"No, when I was a teenager I was self-centered and conceited and didn't care about anybody else, including simulated people."

"So at some point you self-modified to care about simulations. If it helps you, you could self-modify again."

"But I don't want to!"

"Did you want to self-modify to care about simulations in the first place?"

"No, it just sort of happened as I grew up."

"Is there any logical inconsistency in Thud's position?"

Fred thought for a bit.  "Not that I can see.  The value one assigns to simulations seems to be an arbitrary choice.  Ignoring the alien invasion certainly hasn't harmed his career."

"Concern about simulations seems to give the aliens more influence over you than Thud would prefer. What would you prefer?"

"Well, I'd also prefer the aliens not to be able to jerk me around. I really don't have room in my life for it now.  In the grand scheme of things, it seems just wrong -- they shouldn't be able to genocide a species with a few thousand stupid spheres that just sit there converting sunlight to heat."

Michael passed Fred a piece of paper with a short list of bulleted items.  "This is the procedure I teach my clients who want to change their preferences.  After you've learned it, you can decide whether and how you want to use it..."

Sort By: Best
Comment author: 20 May 2011 01:20:51AM *  22 points [-]

The relevant question is, how does surrendering, or not surrendering, control the probability of the ultimatum having been given? If it doesn't, we should surrender. If the aliens sufficiently more likely wouldn't make the ultimatum if we wouldn't surrender if they did, we shouldn't surrender. Furthermore, we should look for third options whose choosing could also control aliens' actions.

Since this information is not given in the story, and the only thing we can go on is anthropomorphic intuition that we shouldn't give in to blackmail (since having the property to not give in really does control the probability of getting blackmailed by humans), the correct answer wasn't suggested, which defeats part of the appeal of a puzzle like this and can wreak some unnecessary memetic hazard.

For the same reason, focusing on whether one "cares about simulations" in this context is misleading, a false dilemma, since this is not the most relevant consideration. It's like asking whether you should cross a road on prime-numbered minutes, and pointing out examples of people who did cross the road on prime-numbered minutes and were run over, instead of focusing on how you should respond to traffic lights.

Comment author: 19 May 2011 05:56:45PM 7 points [-]

Stuart Armstrong posted a similar scenario earlier. A lot of the discussion there are relevant here.

Comment author: 19 May 2011 06:04:24PM *  7 points [-]

Some of the responses on that thread are hilarious, when did LW become so serious?

(I guess the other thing to do is to look at other similarly old threads and see if it has equally irreverent comments)

Comment author: 19 May 2011 06:20:49PM 13 points [-]

Would it make a difference if instead of simulation, they had gotten human dna and were speed-growing clones to torture?

Comment author: 19 May 2011 08:41:09PM *  4 points [-]

It shouldn't, IMO. The reasonable job description for Thud should be some combination of protecting a particular set of people and a particular piece of land, and that doesn't cover rescuing synthesized humans.

Comment author: 20 May 2011 04:29:22AM 4 points [-]

If that is the extent of his job description, should he stop all humanitarian aid to foreign nations where it has no direct benefit to his own set of people?

And would that allow him to eliminate the rest of humanity for some marginal benefit to his countrymen?

Comment author: 08 December 2012 11:18:23PM 1 point [-]

Along these lines, why do each of us individually do pretty much nothing, or at best something pretty minimal, to help the millions of people in the real world living in poverty or dying from preventable diseases? It seems to me our empathy has only a "limited range", something like the "monkeysphere" effect I suppose, whereby we only really care about those closest to us. We have some abstract empathy for the less fortunate, but not enough to really do much about it.

I can imagine also that empathy asymptotes to almost zero at the emotional distance of simulated people (and that we can only think about large numbers of people logarithmically, so that there is hardly any difference between millions and trillions of people to the empathy portion of our utility function). As the scenario demonstrates, it seems like this behaviour has great survival value.

And to answer your question, the real world demonstrates the answer. The amount of foreign aid given to developing countries is abysmally small compared to the value we should place on the lives of those living in disease and poverty, if we indeed cared about them anything like as much as we cared about those who affect our lives more directly. The current amount of foreign aid given is more closely proportional to its UN and local PR value. It is politically, not morally, motivated.

For the second question, well we are generally more adverse to direct negative intervention that to lack of positive intervention, such as in the case of the runaway train that is about to kill a group of schoolchildren, and a big fat man either a) happens to fall in the path of the train (stopping it), but you could intervene and save him - but this would result in the deaths of the schoolchildren; or b) you are in a position where you could push the big fat man in front of the train, thus saving the children but killing the fat man. Most people would not save the fat man, but they would not push him in front of the train either.

I expect the explanation for this is that killing people for the immediate benefit of the group destroys social cohesion, since those in the group live in fear of whether or not their comrades will suddenly turn on them for the benefit of a naive kind of global utility. Likewise eliminating most of humanity for the benefit of said countrymen doesn't seem like it could be done without similarly introducing extreme suspicion into the remaining society. This problem is less evident with the foreign aid situation, although of course the possibility that no foreign aid might easily extend to reduced domestic aid might erode trust in those furthest from central power.

Letting the simulated people suffer is not likely to induce similar social chaos, unless perhaps the aliens can convince the citizens of the Earth that they themselves might be the simulated ones.

Comment author: 02 December 2014 06:16:28PM 0 points [-]

"And would that allow him to eliminate the rest of humanity for some marginal benefit to his countrymen?" That doesn't sound too far from the principle which many militaries throughout history and in the present follow, including what some very large factions within American politics want the military to follow.

For example, 50,000 civilians were killed in the war in Afghanistan (and that estimate is a few years old); if the Afghanistan war is justified because of 9/11, that gives a value of *at least 16 Afghans per American. If we assume that the war wasn't just barely justified- that nobody would switch their opinion due to it being 60,000 rather than 50,000 casualties- a ratio of 20 Afghans to 1 American or probably higher seems quite reasonable. Extrapolating from that we can see that killing 6-7 Billion Afghans to save ~300 million Americans would be quite reasonable. (if we count based on the Iraq War, it would be to save 30 million Americans)

the actual utilitarian preventative calculations are complicated here, namely because the war in Afghanistan most likely made further terrorist attacks *more likely rather than less, so I'm just assuming revenge is our utility function. I'm also not counting the American soldiers lost in Afghanistan.

Now, to clarify, if the war in Afghanistan is justified because even more Afghans would have died under Taliban rule otherwise, then that wouldn't apply. But it seems to me that the main argument presented in favor of the war, even by liberals who support it, is over "preventing terrorism" rather than humanitarian interests, which seems to be widely appealing to the vast majority of the American public.

Obviously that's to actually avenge or prevent the deaths of a large portion of the country, not "some marginal benefit". Still, I don't think it's too far off to say that quite a few people do hold that view and would not be at all inconsistent, in that regard, in wanting General Thud to control their armed forces.

Comment author: 02 December 2014 09:54:33PM 1 point [-]

You failed to consider a combination. For instance, the main objective of the war is preventing terrorism, but the fact that the Taliban would have killed people if left in power changes it from "lots of lives lost to prevent terrorism" to "few lives lost to prevent terrorism". Just because preventing the Taliban from killing wasn't our main goal doesn't mean that it can't affect the balance in favor of the goal that we did have.

Comment author: 19 May 2011 11:26:49PM 4 points [-]

If the aliens care so much about us building death camps and marching our citizens into those, why don't they simulate that directly instead of simulating torture in an indirect attempt to achieve the primary goal?

Comment author: 21 May 2011 12:26:17AM 2 points [-]

If the aliens care so much about us building death camps and marching our citizens into those, why don't they simulate that directly instead of simulating torture in an indirect attempt to achieve the primary goal?

They want real estate. That is, real real estate, not simulated real estate. If they persuade us to kill ourselves they can occupy the solar system without any opposition.

Comment author: 21 May 2011 09:13:46AM 4 points [-]

What's so much better about the "real" substrate, as opposed to the simulated one, that makes such an attempt worth investing the massive resources it must take to build all these computers - and commit to using them for torture?

The "trade" that the aliens are proposing seems to be as follows: you have something we want, namely a rather inefficient world capable of supporting ten billion. We have a substrate capable of convincingly simulating trillions of people. Give us your inefficient world or we will devote our efficient substrate to making many copies of your folk miserable.

This trade has strange payouts. The "no deal" case (putatively) creates massive loss for us, but also for the aliens - they lose the benefit of using the substrate for their own purposes AND they don't gain our world. The "deal" case disposes of all humanity and gains the aliens a tiny slice of elbow room.

If you do care about what happens to simulations, a better trade would be to let us use the substrate to support a population in the trillions, in exchange for the use of our smaller world.

Me, I'd rather be alive in a simulation (barring torture) than dead in the real world.

Comment author: 19 May 2011 05:29:27PM *  19 points [-]

It seems you can set up a similar scenario without simulations: have the aliens physically kidnap 90% of the human race, then try to blackmail the other 10% into killing themselves. That would make for an interesting moral dilemma too.

Comment author: 19 May 2011 05:48:29PM *  8 points [-]

Doubtful. It's likely that losing 90% of earth's population would result in collapse of the world's cultural and economic institutions, including nation states, resulting in a very different test. Maybe if they had clone-farms and produced physical humans to torture offworld?

Which brings to mind we should expend effort on "rescue" attempts, such as hacking the little spheres or stopping the aliens through other means.

Comment author: 19 May 2011 05:56:32PM 4 points [-]

It's also likely that an advanced alien species with an interest in getting rid of humans would do it by creating some nasty nanotech or redirecting a few asteroids, rather than going through this elaborate and unreliable blackmail scenario. It's not a dilemma if you parse out all the practical considerations, so don't.

Comment author: 19 May 2011 05:59:46PM *  5 points [-]

Suspension of disbelief is important for what-if scenarios. I find little black spheres to be weirdly possible given a particular world-state ("crazy aliens"); thinking of 90% of humans being kidnapped evokes a very different possibility-view of the result. I can't stop thinking about post-apocalypse movies in the latter case.

Comment author: 19 May 2011 06:03:27PM 19 points [-]

And you can set up a scenario without dragging in torture and extinction. Aliens from Ganymede are about to ink a contract to trade us tons of Niobium in exchange for tons of Cobalt. But then the aliens reveal that they have billions of cloned humans working as an indentured proletariat in the mines of the Trojan asteroids. These humans are generally well treated, but the aliens offer to treat them even better - feed them ice cream - if we send the Cobalt without requiring payment in Niobium.

The central problem in all of these thought experiments is the crazy notion that we should give a shit about the welfare of other minds simply because they exist and experience things analogously to the way we experience things.

Comment author: 19 May 2011 06:26:39PM 40 points [-]

Is there a standard name for the logical fallacy where you attempt a reductio ad absurdum but fail to notice that you're deriving the absurdity from more than one assumption? Why conclude that it's the caring about far-away strangers that is crazy, as opposed to the decision algorithm that says you should give in to extortions like this?

Comment author: 19 May 2011 06:35:01PM 5 points [-]

I'm not sure words like "crazy" and "absurd" are even meaningful in this context. It's pretty easy to come up with internally consistent arguments generating both results, and the scenario's outlandish enough that it's not clear which one has more practical vulnerabilities; essentially we're dealing with dueling intuitions.

Comment author: 19 May 2011 08:54:20PM *  4 points [-]

Is there a standard name for the logical fallacy where you attempt a reductio ad absurdum but fail to notice that you're deriving the absurdity from more than one assumption?

Good catch. Yes, I was deriving the absurdity from more than one assumption.

Why conclude that it's the caring about far-away strangers that is crazy, as opposed to the decision algorithm that says you should give in to extortions like this?

Maybe with the right decision algorithm you wouldn't give in to extortions like this. However, this extortion attempt cost the aliens approximately nothing, so unless correctly inferring our decision algorithm cost them less than approximately nothing, the rational step for the aliens is to try the extortion regardless. Thus having a different decision algorithm probably wouldn't prevent the extortion attempt.

Comment author: 19 May 2011 09:17:42PM *  11 points [-]

But then changing your values to not care about simulated torture won't prevent the extortion attempt either (since the aliens will think there's a small chance you haven't actually changed your values and it costs them nothing to try). Unless you already really just don't care about simulated torture, it seems like you'd want to have a decision algorithm that makes you go to war against such extortionists (and not just ignore them).

Comment author: 20 May 2011 09:22:12AM *  2 points [-]

But then changing your values to not care about simulated torture won't prevent the extortion attempt either (since the aliens will think there's a small chance you haven't actually changed your values and it costs them nothing to try).

That 'costs them nothing' part makes a potentially big difference. That the aliens must pay to make their attempt is what gives your decision leverage. The war that you suggest is another way of ensuring that there is a cost. Even though you may actually lose the war and be exterminated.

(Obviously there are whole other scenarios where becoming a 'protectorate' and tithing rather than going to war constitutes a mutually beneficial cooperation. When their BATNA is just to wipe you out but it is slightly better for them to just let you pay them.)

Comment author: 19 May 2011 09:53:31PM 2 points [-]

... changing your values to not care about simulated torture ... prevent the extortion attempt ...

Wait, is this a variant on Newcomb's problem?

(Am I just slow today? Nobody else seems to have mentioned it outright, at least.)

Comment author: 20 May 2011 07:26:57AM 3 points [-]

This sort of thing is really the motivating example behind Newcomb's problem.

Comment author: 20 May 2011 03:49:13PM 3 points [-]

This sort of thing is really the motivating example behind Newcomb's problem.

I'm not seeing the analogy. Can you explain?

The extortion attempt cost the aliens almost nothing, and would have given them a vacant solar system to move into if someone like Fred was in power, so it's rational for them to make the attempt almost regardless of the odds of succeeding. Nobody is reading anybody else's mind here, except the idiots who read their own minds and uploaded them to the Internet, and they don't seem to be making any of the choices.

Comment author: 20 May 2011 08:25:37PM 3 points [-]

This case looks most like the 'transparent boxes' version of the problem, which I haven't read much about.

In Newcomb's problem, Omega offers a larger amount of utility if you will predictably do something that intuitively would give a smaller amount of utility.

In this situation, being less open to blackmail probably gives you less disutility in the long run (fewer instances of people trying to blackmail you) than acceding to the blackmail, even though acceding intuitively gives you less disutility.

The other interesting part of this particular scenario is how to define 'blackmail' and differentiate it from, say, someone accidentally doing something that's harmful to you and asking you to help fix it. We've approached that issue, too, but I'm not sure if it's been given a thorough treatment yet.

Comment author: 20 May 2011 06:11:08PM *  1 point [-]

They had other choices though. It would have been similarly inexpensive to offer to simulate happy people.

Even limiting the spheres to a single proof-of-concept would have been a start.

Comment author: 19 May 2011 09:28:46PM *  -1 points [-]

I really don't care about simulated torture, certainly not enough to prefer war over self-modification if simulated torture becomes an issue. War is very expensive and caring about simulated torture appears to be cost without benefit.

The story is consistent with this. Fred has problems because he cares about simulated torture, and Thud doesn't care and doesn't have problems.

Hmm, perhaps we agree that the story has only one source of absurdity now? No big deal either way.

(UDT is still worth my time to understand. I owe you that, and I didn't get to it yet.)

Comment author: 19 May 2011 09:45:52PM 10 points [-]

Err, the point of having a decision theory that makes you go to war against extortionists is not to have war, but to have no extortionists. Of course you only want to do that against potential extortionists who can be "dissuaded". Suffice it to say that the problem is not entirely solved, but the point is that it's too early to say "let's not care about simulated torture because otherwise we'll have to give in to extortion" given that we seem to have decision theory approaches that still show promise of solving such problems without having to change our values.

Comment author: 20 May 2011 07:13:48PM 8 points [-]

If Fred cared about the aliens exterminating China, and Thud didn't care; then if the aliens instead threatened to exterminate China, Fred would again have problems and Thud again wouldn't have.

A rock doesn't care about anything, and therefore it has no problems at all.

This topic isn't really about simulation, it's about the fact that caring about anything permits you to possibly sacrifice something else for it. Anything that isn't our highest value may end up traded away, sure.

Comment author: 20 May 2011 07:45:25PM *  -2 points [-]

If Fred cared about the aliens exterminating China, and Thud didn't care; then if the aliens instead threatened to exterminate China, Fred would again have problems and Thud again wouldn't have.

You can travel from here to China and back. Therefore, caring about China has at least a potential instrumental consequence on the rest of my life. You can't travel from here to the aliens' simulation and back, so caring about what happens there imposes costs on the rest of my life but no benefits. The analogy is not valid.

Now, if the black spheres had decent I/O capabilities and you could outsource human intellectual labor tasks to the simulations, I suppose it would make sense to care about what happens there. People can't do useful work while they're being tortured, so that wasn't the scenario in the story.

Comment author: 20 May 2011 08:06:00PM *  12 points [-]

You can travel from here to China and back. Therefore, caring about China has at least a potential instrumental consequence on the rest of my life.

That's the only sane reason you believe can exist for caring about distant people at all? That you can potentially travel to them?

So if you're a paraplegic , who doesn't want to travel anywhere, can't travel anywhere, and know you'll die in two weeks anyway. You get a choice to push a button or not push it. If you push it you get 1 dollar right now, but 1 billion Chinese people will die horrible deaths in two weeks, after your own death.

Are you saying that the ONLY "sane" choice is to push the button, because you can use the dollar to buy bubblegum or something, while there'll never be a consequence on you for having a billion Chinese die horrible deaths after your own death?

If so, your definition of sanity isn't the definition most people have. You're talking about the concept commonly called "selfishness", not "sanity".

Comment author: 20 May 2011 08:13:51PM *  5 points [-]

If so, your definition of sanity isn't the definition most people have. You're talking about the concept commonly called "selfishness", not "sanity".

Fine. Explain to me why Fred shouldn't exterminate his species, or tell me that he should.

The extortion aspect isn't essential. Fred could have been manipulated by true claims about making simulated people super happy.

ETA: At one point this comment had downvotes but no reply, but when I complained that that wasn't a rational discussion, someone actually replied. LessWrong is doing what it's supposed to do. Thanks people for making it and participating in it.

Comment author: 21 May 2011 12:48:03AM 1 point [-]

If so, your definition of sanity isn't the definition most people have.

Um, I never tried to define sanity. What are you responding to?

Comment author: 20 May 2011 11:49:04AM *  1 point [-]

caring about simulated torture appears to be cost without benefit.

Generally the benefit of caring about about any bad thing is that if you care about it there will be less of it because you will work to stop it.

Comment author: 20 May 2011 03:40:08PM *  2 points [-]

caring about simulated torture appears to be cost without benefit.

Generally the benefit of caring about about any bad thing is that if you care about it there will be less of it because you will work to stop it.

Well, Fred cared, and his reaction was to propose exterminating humanity. I assume you think his is a wrong decision. Can you say why?

If you care about simulated torture (or simulated pleasure), and you're willing to shut up and multiply, then anybody with a big enough computer can get you to do anything even when that computer has no inputs or outputs and makes absolutely no difference to the real world. I think it's better to adjust oneself so one does not care. It's not like it's a well-tested human value that my ancestors on the savannah acted upon repeatedly.

Comment author: 20 May 2011 07:03:57PM 7 points [-]

If you care about simulated torture (or simulated pleasure), and you're willing to shut up and multiply, then anybody with a big enough computer can get you to do anything even when that computer has no inputs or outputs and makes absolutely no difference to the real world.

Do your calculations and preferred choices change if instead of "simulations", we're talking about trillions of flesh-and-blood copies of human beings who are endlessly tortured to death and then revived to be tortured again? Even if they're locked in rooms without entrances or exists, and it makes absolutely no difference to the outside world?

If you care about them, then anybody with a big enough copier-of-humans, and enough torture chambers "can get you to do anything", as you say. So it's not really an issue that depends on caring for simulations. I wish the concept of "simulations" wasn't needlessly added where it has no necessity to be entered.

General Thud would possibly not care if it was the whole real-life population of China that got collected by the aliens, in exchange for a single village of Thud's own nation.

The issue of how-to-deal-with-extortion is a hard one, but it's just made fuzzier by adding the concept of simulations into the mix.

Comment author: 20 May 2011 07:50:36PM 2 points [-]

The issue of how-to-deal-with-extortion is a hard one, but it's just made fuzzier by adding the concept of simulations into the mix.

I agree that it's a fuzzy mix, but not the one you have in mind. I intended to talk about the practical issues around simulations, not about extortion.

Given that the aliens' extortion attempt cost them almost nothing, there's not much hope of gaming things to prevent it. Properly constructed, the black spheres would not have an audit trail leading back to the aliens' home, so a competent extortionist could prevent any counterattack. Extortion is not an interesting part of this situation.

If you care about them, then anybody with a big enough copier-of-humans, and enough torture chambers "can get you to do anything", as you say. So it's not really an issue that depends on caring for simulations. I wish the concept of "simulations" wasn't needlessly added where it has no necessity to be entered.

Right. It's an issue about caring about things that are provably irrelevant to your day-to-day activities.

Comment author: 21 May 2011 06:10:02AM 1 point [-]

That sounds like a flaw in the decision theory. What kind of broken decision theory achieves its values better by optimizing for different values?

Comment author: 20 May 2011 06:17:01PM 0 points [-]

What do you mean by "the real world"? Why does it matter if it's "real"?

Comment author: 20 May 2011 06:47:47PM 2 points [-]

Why does it matter if it's "real"?

The real world generally doesn't get turned off. Simulations generally do. That's why it matters.

If there were a simulation that one might reasonably expect to run forever, it might make sense to debate the issue.

Comment author: [deleted] 21 May 2011 05:50:01PM 2 points [-]

To a degree, arguing about extortion is arguing about definitions. In the context of the heuristic "don't give in to extortion", we would like to know exactly what the heuristic shouldn't give in to, though, and why.

In my opinion, the main problem is that the extortionist is making a no-downside trade: the thing it is trading is "not torturing simulated humans" or "not killing hostages" or whatever, which probably wasn't worth anything to the extortionist anyway.

A lot of no-downside trades are obviously unfair, so a useful heuristic is not to agree to no-downside trades in general. In fact, extremely unfair trades in general are metaphorically labeled "extortion" (for instance, I'm sure I've heard the term applied to the price of a diamond ring).

Comment author: 19 May 2011 06:49:02PM 1 point [-]

How is offering to supply ice cream characterized as "extortion"?

In any case, I was not using the scenario as a reductio against universal unreciprocated altruism. That notion fails under its own weight, due to complete absence of support.

Comment author: 19 May 2011 07:06:16PM *  5 points [-]

Sorry, I misread your comment and thought it was an extortion scenario similar to the OP. Now that I've read it more carefully, it's not clear to me that we shouldn't give up the Niobium in order to provide those humans workers with ice cream. (ETA: why did you characterize those humans as indentured workers? It would have worked as well if they were just ordinary salaried workers.)

That notion fails under its own weight, due to complete absence of support.

Altruists certainly claim to have support for their stated preferences. Or one could argue that preferences don't need to have support. What kind of support do you have for liking ice cream, for example?

Comment author: 19 May 2011 07:12:09PM 3 points [-]

Sorry, I misread your comment and thought it was an extortion scenario similar to the OP.

Your reading wasn't far off: "in all of these thought experiments" makes your reply remain relevant.

Comment author: 19 May 2011 07:25:37PM 5 points [-]

True enough. My main objection to calling my ice cream negotiating tactic 'extortion' is that I really don't like the "just say 'No' to extortion" heuristic. I see no way of definitionally distinguishing extortion from other, less objectionable negotiating stances. Nash's 1953 cooperative game theory model suggests that it is rational to yield to credible threats. I.e. saying 'no' to extortion doesn't win! An AI that begins with the "just say no" heuristic will self-modify to one that dispenses with that heuristic.

Comment author: 19 May 2011 11:11:26PM *  6 points [-]

I don't think anybody is suggesting building an explicit "just say 'No' to extortion" heuristic into an AI. (I agree we do not have a good definition of "extortion" so when I use the word I use it in an intuitive sense.) We're trying to find a general decision theory that naturally ends up saying no to extortion (when it makes sense to).

Here's an argument that "saying 'no' to extortion doesn't win" can't be the full picture. Some people are more credibly resistant to extortion than others and as a result are less likely to be extorted. We want an AI that is credibly resistant to extortion, if such credibility is possible. Now if other players in the picture are intelligent enough, to the extent of being able to deduce our AI's decision algorithm, then isn't being "credibly resistant to extortion" the same as having a decision algorithm that actually says no to extortion?

ETA: Of course the concept of "credibility" breaks down a bit when all agents are reasoning this way. Which is why the problem is still unsolved!

Comment author: 20 May 2011 05:15:27PM 1 point [-]

Of course the concept of "credibility" breaks down a bit when all agents are reasoning this way.

It does what? How so?

Comment author: 20 May 2011 05:06:16AM *  0 points [-]

I don't think anybody is suggesting building an explicit "just say 'No' to extortion" heuristic into an AI. (I agree we do not have a good definition of "extortion" so when I use the word I use it in an intuitive sense.) We're trying to find a general decision theory that naturally ends up saying no to extortion (when it makes sense to).

That is pretty incoherent. If you are trying to come up with a general decision theory that wins and also says no to extortion, then you have overdetermined the problem (or will overdetermine it once you supply your definition). If you are predicting that a decision theory that wins will say no to extortion, then it is a rather pointless claim until you supply a definition. Perhaps what you really intend to do is to define 'extortion' as 'that which a winning decision theory says no to'. In which case, Nash has defined 'extortion' for you - as a threat which is not credible, in his technical sense.

ETA: Of course the [informal] concept of "credibility" breaks down a bit when all agents are reasoning this way. Which is why the problem is still unsolved!

Why do you say the problem is still unsolved? What issues do you feel were not addressed by Nash in 1953? Where is the flaw in his argument?

Part of the difficulty of discussing this here is that you have now started to use the word "credible" informally, when it also has a technical meaning in this context.

Comment author: 19 May 2011 11:34:49PM 0 points [-]

"Commit to just saying 'no' and proving that when just committing to just saying 'no' and proving that wins."

Perhaps something like that.

Comment author: 19 May 2011 07:48:02PM *  5 points [-]

I really don't like the "just say 'No' to extortion" heuristic.

Well you don't want to signal that you give in to extortion. That would just increase the chances of people attempting extortion against you. Better to signal that you are on a vendetta to stamp out extortion - at your personal expense!!!

Comment author: 19 May 2011 10:03:45PM 0 points [-]

There is an idea, surprisingly prevalent on a rationality website, that costless signaling is an effective way to influence the behavior of rational agents. Or in other words, that it is rational to take signalling at face value. I personally doubt that this idea is correct. In any case, I reiterate that I suggest yielding only to credible threats. My own announcements do not change the credibility of any threats available to agents seeking to exploit me.

Comment author: 19 May 2011 11:38:00PM 0 points [-]

Perhaps what is really being expressed is the belief that social costs are real, and that mere pseudonymous posting has costs.

Comment author: 19 May 2011 10:14:08PM *  0 points [-]

My own announcements do not change the credibility of any threats available to agents seeking to exploit me.

They inflluence the liklihood of them being made in the first place - by influencing the attacker's expected payoffs. Especially if it appears as though you were being sincere. Your comment didn't look much like signalling. I mean, it doesn't seem terribly likely that someone would deliberately publicly signal that they are more likely than unnamed others to capitulate if threatened with an attempt at extortion.

Credibly signalling resistance to extortion is non-trivial. Most compelling would be some kind of authenticated public track record of active resistance.

Comment author: 20 May 2011 05:21:32PM 2 points [-]

I see no way of definitionally distinguishing extortion from other, less objectionable negotiating stances.

Well, a simple way would be to use the legal definition of extortion. That should at least help prevent house fires, kidnapping, broken windows and violence.

...but a better definition should not be too difficult - for instance: the set of "offers" which you would rather not be presented with.

Comment author: 19 May 2011 07:43:59PM 3 points [-]

My main objection to calling my ice cream negotiating tactic 'extortion

My objection to calling the ice cream negotiation tactic 'extortion' is it just totally isn't. It's an offer of a trade.

Nash's 1953 cooperative game theory model suggests that it is rational to yield to credible threats. I.e. saying 'no' to extortion doesn't win!

Then it's a good thing we've made developments in our models in the last six decades!

Comment author: 19 May 2011 10:08:49PM 2 points [-]

Then it's a good thing we've made developments in our models in the last six decades!

Cute. But perhaps you should provide a link to what you think is the relevant development.

Comment author: 20 May 2011 05:25:59PM *  3 points [-]

Well, the key concept underlying strong resistance to extortion is reputation management. Once you understand the long-term costs of becoming identified as a vulnerable "mark" by those in the criminal underground, giving in to extortion can start to look a lot less attractive.

Comment author: 19 May 2011 09:53:31PM -1 points [-]

My objection to calling the ice cream negotiation tactic 'extortion' is it just totally isn't. It's an offer of a trade.

To elaborate a bit:

I'll give you utility if you give me utility is a trade.

I won't cause you disutility if you give me utility is extortion.

Comment author: 20 May 2011 06:45:08PM *  7 points [-]

I'll give you utility if you give me utility is a trade.

I won't cause you disutility if you give me utility is extortion.

I don't think that's exactly the right distinction. Let's say you go to your neighbour because he's being noisy.

Scenario A: He says "I didn't mean to disturb you, I just love my music loud. But give me 10 dollars, and sure, I'll turn the volume down." I'd call that a trade, though it's still about him not giving you disutility.

Scenario B: He says "Yeah, I do that on purpose, so that I can make people pay me to turn the volume down. It'll be 10 bucks. " I'd call that an extortion.

The difference isn't between the results of the offer if you accept or reject -- the outcomes and their utility for you is the same in each (loud music, silence - 10 dollars).

The difference is that in Scenario B, you wish the other person had never decided to make this offer. It's not the utility of your options that are to be compared with each other, but the utility of the timeline where the trade can be made vs the utility of the timeline where the trade can't be made...

In the Trade scenarios, if you can't make a trade with the person, he's still being noisy, and utility minimizes. In the Extortion scenarios, if you can't make a trade with the person, he has no reason to be noisy, and utility maximizes.

I'll probably let someone else to transform the above description into equations containing utility functions.

Comment author: 19 May 2011 10:06:37PM 1 point [-]

And what is the distinction between giving utility and not giving disutility? As consequentialists, I thought we were committed to the understanding that they are the same thing.

Comment author: 19 May 2011 07:15:12PM 0 points [-]

What kind of support do you have for liking ice cream, for example?

None at all. But then I don't claim that it is a universal moral imperative that will be revealed to be 'my own imperative' once my brain is scanned, the results of the scan are extrapolated, and the results are weighted in accordance with how "muddled" my preferences are judged to be.

Comment author: 19 May 2011 07:30:36PM 3 points [-]

I see, so you're saying that universal unreciprocated altruism fails as a universal moral imperative, not necessarily as a morality that some people might have. Given that you used the word "crazy" earlier I thought you were claiming that nobody should have that morality.

Comment author: 19 May 2011 07:42:51PM 3 points [-]

I think it is easily possible to imagine naturalists describing some kinds of maladaptive behaviour as being "crazy". The implication would be that the behaviour was being caused by some kind of psychological problem interfering with their brain's normal operation.

Comment author: 20 May 2011 04:39:33PM 0 points [-]

I thought you were claiming that nobody should have that morality.

I do claim that. In two flavors.

1. Someone operating under that moral maxim will tend to dispense with that maxim as they approach reflective equilibrium.

2. Someone operating under that 'moral' maxim is acting immorally - this operationally means that good people should (i.e. are under a moral obligation to) shun such a moral idiot and make no agreements with him (since he proclaims that he cannot be trusted to keep his commitments).

Part of the confusion between us is that you seem to want the word 'morality' to encompass all preferences - whether a preference for chocolate over vanilla, or a preference for telling the truth over lying, or a preference for altruism over selfishness. It is the primary business of metaethics to make the distinction between moral opinions (i.e. opinions about moral issues) and mere personal preferences.

Comment author: 20 May 2011 07:18:05PM *  2 points [-]

Part of the confusion between us is that you seem to want the word 'morality' to encompass all preferences - whether a preference for chocolate over vanilla, or a preference for telling the truth over lying, or a preference for altruism over selfishness.

No, I don't want that. In fact I do not currently have a metaethical position beyond finding all existing metaethical theories (that I'm aware of) to be inadequate. In my earlier comment I offered two possible lines of defense for altruism, because I didn't know which metaethics you prefer:

Altruists certainly claim to have support for their stated preferences. Or one could argue that preferences don't need to have support.

In your reply to that comment you chose to respond to only the second sentence, hence the "confusion".

Anyway, why don't you make a post detailing your metaethics, as well as your arguments against "universal unreciprocated altruism"? It's not clear to me what you're trying to accomplish by calling people who believe such things (many of whom are very smart and have already seriously reflected on these issues) "crazy" without backing up your claims.

Comment author: 20 May 2011 09:17:24PM *  0 points [-]

It's not clear to me what you're trying to accomplish by calling people who believe such things (many of whom are very smart and have already seriously reflected on these issues) "crazy" without backing up your claims.

I'm not sure why you think I have called anyone crazy. What I said above is that a particular moral notion is crazy.

Perhaps you instead meant to complain that (in the grandparent) I had referred to the persons in question as "moral idiots". I'm afraid I must plead guilty to that bit of hyperbole.

Anyway, why don't you make a post detailing your metaethics, as well as your arguments against "universal unreciprocated altruism"?

I am gradually coming to think that there is little agreement here as to what the word metaethics even means. My current understanding is that metaethics is what you do to prepare the linguistic ground so that people operating under different ethical theories and doctrines can talk to each other. Meta-ethics strives to be neutral and non-normative. There are no meta-ethical facts about the world - only definitions that permit discourse and disputation about the facts.

Given this interpretation of "meta-ethics", it would seem that what you mean to suggest is that I make a post detailing my normative ethics, which would include an argument against "universal unreciprocated altruism" (which I take to be a competing theory of normative ethics).

Luke and/or Eliezer and/or any trained philosopher here: I would appreciate feedback as to whether I finally have the correct understanding of the scope and purpose of meta-ethics.

Comment author: 19 May 2011 11:33:03PM 0 points [-]

Is there a standard name for the logical fallacy

Hidden assumptions play a role similar to the auxiliary hypotheses which undermine naive Popperianism. The fallacy of ignoring auxiliary assumptions seems like a special case of the fallacy of presenting an argument from ignorance.

Comment author: 21 May 2011 06:14:55PM *  9 points [-]

No, I think the central "problem" is that having preferences that others can thwart with little effort is risky because it makes you more vulnerable to extortion.

For example, if you have a preference against non-prime heaps of pebbles existing, the aliens can try to extort you by building huge numbers of non-prime heaps on their home planet and sending you pictures of them, and therefore, the argument goes, it's crazy and stupid to care about non-prime heaps.

The argument also yields a heuristic that the farther away a thing is from you, the more stupid and crazy it is to care about it.

Comment author: 22 May 2011 04:19:23AM 1 point [-]

Right. What you are saying is related to the notion of "credible threats". If other agents can give you disutility with little disutility for themselves, then they have a credible threat against you. And unless you either change your utility function, or find a way of making it much more difficult and costly for them to harm you, the rational course is to give in to the extortion.

One way to make it costly for others to harm you is to join a large coalition which threatens massive retaliation against anyone practicing extortion against coalition members. But notice that if you join such a coalition, you must be willing to bear your share of the burden should such retaliation be necessary.

The alternative I suggested in the grandparent was to change your utility function so as to make you less vulnerable - only care about things you have control over. Unfortunately, this is advice that may be impossible to carry out. Preferences, as several commentators here have pointed out, tend to be incorrigible.

Comment author: 22 May 2011 06:51:28AM *  4 points [-]

The alternative I suggested in the grandparent was to change your utility function so as to make you less vulnerable - only care about things you have control over. Unfortunately, this is advice that may be impossible to carry out. Preferences, as several commentators here have pointed out, tend to be incorrigible.

I took the obvious solution to that difficulty. I self modified to an agent that behaves exactly as if he had self modified to be an agent with preferences that make him less vulnerable. This is a coherent configuration for my atoms to be in terms of physics and is also one that benefits me.

Comment author: 19 May 2011 06:16:37PM *  2 points [-]

Your variation is better than mine! Not sure about your solution though, it looks a little hurried.

Comment author: 19 May 2011 06:39:08PM 2 points [-]

Your variation is better than mine!

However it is a different problem. An interesting problem in its own right but one for which many people's coherent preferences will produce a different answer for slightly different reasons.

Comment author: 22 May 2011 11:13:44PM 0 points [-]

The central problem in all of these thought experiments is the crazy notion that we should give a shit about the welfare of other minds simply because they exist and experience things analogously to the way we experience things.

Well, I see the central problem in the notion that we should care about something that happens to other people if we're not the ones doing it to them. Clearly, the aliens are sentient; they are morally responsible for what happens to these humans. While we certainly should pursue possible avenues to end the suffering, we shouldn't act as if we were.

Comment author: 23 May 2011 01:29:46AM 0 points [-]

Interesting. Though in the scenario I suggested there is no suffering. Only an opportunity to deploy pleasure (ice cream).

I'm curious as to your reasons why you hold the aliens morally responsible for the human clones - I can imagine several reasons, but wonder what yours are. Also, I am curious as to whether you think that the existence of someone with greater moral responsibility than our own acts to decrease or eliminate the small amount of moral responsibility that we Earthlings have in this case.

Comment author: 23 May 2011 09:11:01AM *  0 points [-]

Why would I not hold them responsible? They are the ones who are trying to make us responsible by giving us an opportunity to act, but their opportunities are much more direct - after all, they created the situation that exerts the pressure on us. This line of thought is mainly meant to be argued in Fred's terms, who has a problem with feeling responsible for this suffering (or non-pleasure) - it offers him an out of the conundrum without relinquishing his compassion for humanity (i.e. I feel the ending as written is illogical, and I certainly think "Michael" is acting very unprofessionally for a psychoanalyst). ["Relinquish the compassion" is also the conclusion you seem to have drawn, thus my response here.]

Of course, the alien strategy might not be directed at our sense of responsibility, but at some sort of game theoretic utility function that proposes the greater good for the greater number - these utility functions are always sort of arbitrary (most of them on lesswrong center around money, with no indication why money should be valuable), and the arbitrariness in this case consists in including the alien simulations, but not the aliens themselves. If the aliens are "rational agents", then not rewarding their behaviour will make them stop it if it has a cost, while rewarding it will make them continue. (Haven't you ever wondered how many non-rational entities are trying to pose conundrums to rational agents on here? ;)

I don't have a theory of quantifyable responsibility, and I don't have a definite answer for you. Let's just say there is only a limited amount of stuff we can do in the time that we have, so we have to make choices what to do with our lives. I hope that Fred comes to feel that he can accomplish more with his life than to indirectly die for a tortured simulation that serves alien interests.

Comment author: 19 May 2011 06:01:52PM *  2 points [-]

But that would require actually doing something. Sending in a bunch of otherwise inert computational nodes is much cheaper. Hell, they don't even need to send them in. If they can convince people remotely that they're torturing simulations, they can commit genocide by telecommuting, assuming people with Fred's preferences are in power.

Comment author: 20 May 2011 06:18:32PM 0 points [-]

He's saying that Fred would end up with essentially the same dilemma, not that the aliens should have done it that way.

Comment author: 19 May 2011 06:48:18PM *  13 points [-]

Thud yawned. "Fred, you're fired. Get out of my house." As Fred left, Thud closed his curtains and tried to get back to sleep.

I loved this line. It gave me the same warm glow inside that I get when I read about a guy heroically saving the entire world and Thud pulls it off by going to sleep. That's a whole new level above Chuck Norris. Even a level beyond The Doctor. Tennant went to all the trouble of challenging the alien leader to a duel to the death when the Sycorax pulled this stunt!

Comment author: 19 May 2011 11:31:50PM *  4 points [-]

Don't give in to blackmail, duh.

This would be more interesting if they were offering some sort of simulated reward instead. For sufficiently large, otherwise unreachable rewards giving in might even be the correct answer.

Comment author: 20 May 2011 01:59:48AM *  1 point [-]

What's "blackmail", especially given that these are aliens, and there is no understanding of their motive?

Comment author: 20 May 2011 02:07:09AM 1 point [-]

Understanding is not zero.

If someone expends resources in a way that has a good chance of influencing a person similar to you to do X, you can conclude that there is a good chance they want you to do X.

If this expenditure also harms you, then it may make sense to not do X.

Comment author: 20 May 2011 02:02:18AM 1 point [-]
Comment author: 19 May 2011 06:24:33PM 4 points [-]

This story made me smile because it was an enjoyable, well-written piece of fiction :)

Comment author: 20 May 2011 01:39:14AM 2 points [-]

This has been said below but in two parts after a fashion.

Thud is in a position of protecting the people of earth. Thus it is his job to do whatever is best for that group.

If Thud makes a policy of giving in to simulated torture, it seems likely (although our experience with inhuman intelligence is a limiting factor here) that aliens or whatnot would be more likely to simulate and torture people.

So someone in Thud's position has a specific obligation to stop people like Fred from making the universe see earth as a pushover planet that will give up their resources to simulators.

Comment author: 20 May 2011 02:02:09AM 1 point [-]

What about the reverse, then?

The aliens promise that, if we give them all our stuff, they will simulate a bunch of super happy copies of us.

Do we give them stuff?

Comment author: 20 May 2011 02:15:45AM 2 points [-]

The aliens promise that, if we give them all our stuff, they will simulate a bunch of super happy copies of us.

Do we give them stuff?

If it were up to Thud, he'd say "no", since he doesn't care about simulations. However, he's a soldier, not a politician, so that decision wouldn't be part of his job.

In retrospect, perhaps I should have done the story with that premise and "President Thud" instead. It would have allowed me to show an example about inappropriate care about simulations, but without the confusing issue of how to deal with extortion situations.

Comment author: 20 May 2011 02:05:49AM 2 points [-]

already suggested that. Depends on how many super happy copies, how super happy, whether we expect to be able to do that ourselves, and possibly whose copies and how representative/diverse they are.

Depending on those answers possibly yes.

Comment author: 20 May 2011 10:44:23PM 2 points [-]

This. Basically this becomes an economic transaction; we are buying simulations. If it's a good price, buy them. If not, don't.

Comment author: 19 May 2011 09:59:20PM 3 points [-]

What happens if we surrender? Do they start treating the tortured simulations well? What kind of torture do they subject us to?

Comment author: 19 May 2011 11:36:04PM 0 points [-]

What happens if we surrender?

I assumed it would be obvious that the non-simulated humans would be killed if they were marched into the death camps, although I didn't actually say so.

Do they start treating the tortured simulations well?

The torture is worse than death, so the simplest detail that makes the story work is that the aliens delete the simulated humans in that case.

Comment author: 20 May 2011 01:15:12AM *  2 points [-]

The terms of the ultimatum weren't clear from the story. These are aliens, but we have to fill in the blanks using anthropomorphic intuitions.

Comment author: 19 May 2011 05:57:44PM 3 points [-]

Why would we be willing to believe that they will stop the torture if we killed ourselves? Further, if we killed ourselves why wouldn't they delete or kill the people being tortured?

They gave us the diagrams of themselves if I understand correctly and from the story we have the understanding to tell they tell the truth. What is to stop us from building massive super computers that can simulate some way of torturing trillions of them unless they stop simulating the torture of us? By communicating this possibility to the aliens and letting them know that if they don't stop by the time the torture computers are built we will immediately begin construction on yet another torture computer to double the amount of torture then, if the aliens have the morality that they are assuming of us, they will immediately stop torturing the simulations.

Comment author: 19 May 2011 08:48:48PM 0 points [-]

Why would we be willing to believe that they will stop the torture if we killed ourselves?

Destructive analysis of a few of the small spheres.

Further, if we killed ourselves why wouldn't they delete or kill the people being tortured?

Ah, but the torture is worse than death! Deletion would be an improvement.

What is to stop us from building massive super computers that can simulate some way of torturing trillions of them unless they stop simulating the torture of us? By communicating this possibility to the aliens and letting them know that if they don't stop by the time the torture computers are built we will immediately begin construction on yet another torture computer to double the amount of torture then, if the aliens have the morality that they are assuming of us, they will immediately stop torturing the simulations.

If I were going to continue the story in that vein, I'd give the aliens the same morality as Thud, and have them make an uninformed guess that someone like Fred might have power.

Comment author: 19 May 2011 06:59:45PM *  0 points [-]

These aliens seem temptingly fragile. They would make perfect targets to aim at when cane toad whacking. There is something appealing about treating that kind of threat with complete and utter contempt. Take them out with projectile vermin!

Since there are only a couple of thousand aliens around using such an inaccurate cane toad launching mechanism will just serve to make the fun last longer. But if more of the aliens start floating around upgrading the targetting system could be viable.

Comment author: 17 November 2011 05:07:10AM 1 point [-]

I think we have to take this as motivation for war with the aliens, rather than giving into or ignoring their demands. Assume that the torturers try this sort of thing with more than one set of victims. If any of the victims successfully defeats the torturers, that spares the would-be subsequent victims from the dilemma. Assume that there are multiple species of aliens that would try this torture technique, and some will concoct more persuasive versions of their threat. If none are stopped, we receive a series of alien torture threats that continues until we run into one with a threat so horrible we decide to kill ourselves. So we need to stop them, to the best of our ability, because we need other threatened aliens to try to stop the torturers they encounter, so the torturers who would successfully threaten us into extinction get killed off before they encounter us.

Comment author: 20 May 2011 06:25:08PM 1 point [-]

Why exactly is Fred changing his preferences? If he's trying to make it so his future preferences are fulfilled, he should change them to preferring the universe to be exactly the way it is. If he's trying to make it so his current preferences are fulfilled, he probably shouldn't change them.

And what's he changing them to? Don't care about cheaply made people?

Comment author: 20 May 2011 09:54:48PM 2 points [-]

Why exactly is Fred changing his preferences?

As he said, his concerns about the simulated people interfere with his sleep. More abstractly, he has preferences that attract his attention even though he cannot act upon them, so he is thinking about getting rid of them.

Comment author: 20 May 2011 10:33:56PM 0 points [-]

He can try to raise awareness. He can try to do something to keep the remaining spheres from breaking.

Also, I got the impression that you were implying we should change our preferences like that. I haven't lost any sleep to this issue, so it wouldn't apply to me.

Comment author: 20 May 2011 06:34:38PM *  1 point [-]

It seems like people often change their preferences in the real world. Do you think these instances fit your model, or do you disagree that it occurs (for example, are we using different definitions of 'preference')?