"General Thud! General Thud! Wake up! The aliens have landed. We must surrender!" General Thud's assistant Fred turned on the lights and opened the curtains to help Thud wake up and confront the situation. Thud was groggy because he had stayed up late supervising an ultimately successful mission carried out by remotely piloted vehicles in some small country on the other side of the world. Thud mumbled, "Aliens? How many? Where are they? What are they doing?" General Thud looked out the window, expecting to see giant tripods walking around and destroying buildings with death rays. He saw his lawn, a bright blue sky, and hummingbirds hovering near his bird feeder.
Fred was trying to bring Thud up to speed as quickly as possible. "Thousands of them, General! 2376, to be precise. They gave us a map; we know where they all are. They aren't doing anything overt, but the problem is their computation! I have one here, if you'd like to look." Fred removed a black sphere two inches in diameter from his pocket and gave it to Thud.
Thud sat on his bed holding the small sphere and staring at it dumbfounded. "Okay, you think we should surrender to a few thousand small spheres. Why is that, exactly?" The sphere seemed a little flexible in Thud's hand. As he experimented a few seconds to see just how flexible, it collapsed in his hand, converting itself into a loose clump of alien sand that landed in his lap and started to dribble onto his bed and the floor. Thud stood up and brushed the rest of the sand off of his pyjamas and bed, and thought for a moment about where he left his vacuum cleaner bags. He was not impressed with these aliens.
Fred said "I don't think you wanted to do that, sir. Their ultimatum states that for every alien we destroy, they'll manufacture two in the outer reaches of the Solar System where we'll never find them!"
Thud said, "Okay, so now you think we should surrender to 2375 small spheres, and two or more small spheres that are out of the battlefield for the moment. Why is that?"
Fred said "Well, you remember a few years back when some people copied their brain state into a computer and posted it to the Internet? Apparently somebody copied the data across an unencrypted wireless link, the aliens picked it up with their radio telescopes, and now they are simulating those poor people in these black spheres and torturing the simulations! They sent us videos!" Fred held up his cell phone, pushed a button, and showed the video to Thud.
Thud looked at the video for a moment and said, "Yep, that's torture. Do these people know anything potentially useful to the aliens?"
Fred said, "Well, they know how to break into a laboratory that has brain scanning tools and push some buttons. That was apparently the high point of their lives. But none of that matters, the aliens don't seem to be torturing them for information anyway."
Thud was still suffering from morning brain fog. He rubbed his eyes. "And why should we surrender?"
Fred said, "The aliens have made a trillion copies of these poor people and will run the torture simulations on the little black spheres until we march all of our citizens into the death camps they demand we build! We have analyzed these black spheres and the engineering diagrams the aliens gave us, and we know this to be true. We only have ten billion citizens, and this simulated torture is much worse than simulated death, so the total utility is much greater if we surrender!"
Thud yawned. "Fred, you're fired. Get out of my house." As Fred left, Thud closed his curtains and tried to get back to sleep.
Michael said "So I take it you no longer assist Thud. What are you doing now?"
Fred reclined comfortably on the analyst's couch. "I help out at the cafeteria as a short order cook. But I'm not worried about my career right now. I have nightmares about all these simulated people being tortured in the flimsy alien spheres."
"Thud surely knows the simulations are being tortured too. Do you think he has nightmares about this?"
"No, he doesn't seem to care."
"Have you always cared about the well-being of simulations?"
"No, when I was a teenager I was self-centered and conceited and didn't care about anybody else, including simulated people."
"So at some point you self-modified to care about simulations. If it helps you, you could self-modify again."
"But I don't want to!"
"Did you want to self-modify to care about simulations in the first place?"
"No, it just sort of happened as I grew up."
"Is there any logical inconsistency in Thud's position?"
Fred thought for a bit. "Not that I can see. The value one assigns to simulations seems to be an arbitrary choice. Ignoring the alien invasion certainly hasn't harmed his career."
"Concern about simulations seems to give the aliens more influence over you than Thud would prefer. What would you prefer?"
"Well, I'd also prefer the aliens not to be able to jerk me around. I really don't have room in my life for it now. In the grand scheme of things, it seems just wrong -- they shouldn't be able to genocide a species with a few thousand stupid spheres that just sit there converting sunlight to heat."
Michael passed Fred a piece of paper with a short list of bulleted items. "This is the procedure I teach my clients who want to change their preferences. After you've learned it, you can decide whether and how you want to use it..."