Failed Utopia #4-2
Followup to: Interpersonal Entanglement
Shock after shock after shock -
First, the awakening adrenaline jolt, the thought that he was
falling. His body tried to sit up in automatic adjustment, and his
hands hit the floor to steady himself. It launched him into the air,
and he fell back to the floor too slowly.
Second shock. His body
had changed. Fat had melted away in places, old scars had faded; the
tip of his left ring finger, long ago lost to a knife accident, had now
suddenly returned.
And the third shock -
"I had nothing to
do with it!" she cried desperately, the woman huddled in on herself in
one corner of the windowless stone cell. Tears streaked her delicate
face, fell like slow raindrops into the dĂŠcolletage of her dress. "Nothing! Oh, you must believe me!"
With
perceptual instantaneity - the speed of surprise - his mind had already
labeled her as the most beautiful woman he'd ever met, including his
wife.
A long white dress concealed most of her, though it left her
shoulders naked; and her bare ankles, peeking out from beneath the
mountains of her drawn-up knees, dangled in sandals. A light touch of
gold like a webbed tiara decorated that sun-blonde hair, which fell
from her head to pool around her weeping huddle. Fragile crystal
traceries to accent each ear, and a necklace of crystal links that
reflected colored sparks like a more prismatic edition of diamond. Her
face was beyond all dreams and imagination, as if a photoshop had been
photoshopped.
She looked so much the image of the Forlorn Fairy
Captive that one expected to see the borders of a picture frame around
her, and a page number over her head.
His lips opened, and without any thought at all, he spoke:
"Wha-wha-wha-wha-wha-"
He shut his mouth, aware that he was acting like an idiot in front of the girl.
"You don't know?" she said, in a tone of shock. "It didn't - you don't already know?"
"Know what?" he said, increasingly alarmed.
She
scrambled to her feet (one arm holding the dress carefully around her
legs) and took a step toward him, each of the motions almost
overloading his vision with gracefulness. Her hand rose out, as if to
plead or answer a plea - and then she dropped the hand, and her eyes
looked away.
"No," she said, her voice trembling as though in desperation. "If I'm the one to tell you - you'll blame me, you'll hate me forever for it. And I don't deserve that, I don't! I am only just now here - oh, why did it have to be like this?"
Um, he thought but didn't say. It was too much drama, even taking into account the fact that they'd been kidnapped -
(he looked down at his restored hand, which was minus a few wrinkles, and plus the tip of a finger)
- if that was even the beginning of the story.
He
looked around. They were in a solid stone cell without windows, or
benches or beds, or toilet or sink. It was, for all that, quite clean
and elegant, without a hint of dirt or ordor; the stones of the floor
and wall looked rough-hewn or even non-hewn, as if someone had simply
picked up a thousand dark-red stones with one nearly flat side, and
mortared them together with improbably perfectly-matching,
naturally-shaped squiggled edges. The cell was well if harshly lit
from a seablue crystal embedded in the ceiling, like a rogue element of
a fluorescent chandelier. It seemed like the sort of dungeon cell you
would discover if dungeon cells were naturally-forming geological
features.
And they and the cell were falling, falling, endlessly
slowly falling like the heart-stopping beginning of a stumble, falling
without the slightest jolt.
On one wall there was a solid stone
door without an aperture, whose locked-looking appearance was only
enhanced by the lack of any handle on this side.
He took it all
in at a glance, and then looked again at her.
There was something in him that just refused to go into a screaming panic for as long as she was watching.
"I'm Stephen," he said.
"Stephen Grass. And you would be the princess held in durance vile,
and I've got to break us out of here and rescue you?" If anyone had ever looked that part...
She smiled at him, half-laughing through the tears. "Something like that."
There
was something so attractive about even that momentary hint of a smile
that he became instantly uneasy, his eyes wrenched away to the wall as
if forced. She didn't look she was trying to be seductive... any more than she looked like she was trying to breathe... He suddenly distrusted, very much, his own impulse to gallantry.
"Well,
don't get any ideas about being my love interest," Stephen said,
looking at her again. Trying to make the words sound completely
lighthearted, and absolutely serious at the same time. "I'm a happily
married man."
"Not anymore." She said those two words and looked
at him, and in her tone and expression there was sorrow, sympathy,
self-disgust, fear, and above it all a note of guilty triumph.
For
a moment Stephen just stood, stunned by the freight of emotion that
this woman had managed to put into just those two words, and then the
words' meaning hit him.
"Helen," he said. His wife - Helen's
image rose into his mind, accompanied by everything she meant to him
and all their time together, all the secrets they'd whispered to one
another and the promises they'd made - that all hit him at once, along
with the threat. "What happened to Helen - what have you done -"
"She has done nothing." An old, dry voice like crumpling paper from a thousand-year-old book.
Stephen
whirled, and there in the cell with them was a withered old person with
dark eyes. Shriveled in body and voice, so that it was impossible to
determine if it had once been a man or a woman, and in any case you
were inclined to say "it". A pitiable, wretched thing, that looked
like it would break with one good kick; it might as well have been
wearing a sign saying "VILLAIN".
"Helen is alive," it said, "and so is your daughter Lisa. They are quite well
and healthy, I assure you, and their lives shall be long and happy
indeed. But you will not be seeing them again. Not for a long
time, and by then matters between you will have changed. Hate me if
you wish, for I am the one who wants to do this to you."
Stephen stared.
Then he politely said, "Could someone please put everything on hold for one minute and tell me what's going on?"
"Once
upon a time," said the wrinkled thing, "there was a fool who was very
nearly wise, who hunted treasure by the seashore, for there was a rumor
that there was great treasure there to be found. The wise fool found a
lamp and rubbed it, and lo! a genie appeared before him - a young genie,
an infant, hardly able to grant any wishes at all. A lesser fool might
have chucked the lamp back into the sea; but this fool was almost wise,
and he thought he saw his chance. For who has not heard the tales of
wishes misphrased and wishes gone wrong? But if you were given a
chance to raise your own genie from infancy - ah, then it might serve you well."
"Okay, that's great," Stephen said, "but why am I -"
"So,"
it continued in that cracked voice, "the wise fool took home the lamp.
For years he kept it as a secret treasure, and he raised the genie and
fed it knowledge, and also he crafted a wish. The fool's wish was a
noble thing, for I have said he was almost wise. The fool's wish was
for people to be happy. Only this was his wish, for he thought all
other wishes contained within it. The wise fool told the young genie
the famous tales and legends of people who had been made happy, and the
genie listened and learned: that unearned wealth casts down a person,
but hard work raises you high; that mere things are soon forgotten, but
love is a light throughout all your days. And the young genie asked
about other ways that it innocently imagined, for making people
happy. About drugs, and pleasant lies, and lives arranged from outside
like words in a poem. And the wise fool made the young genie to never
want to lie, and never want to arrange lives like flowers, and above
all, never want to tamper with the mind and personality of human
beings. The wise fool gave the young genie exactly one hundred and
seven precautions to follow while making
people happy. The wise fool thought that, with such a long list as
that, he was being very careful."
"And then," it said,
spreading two wrinkled hands, "one day, faster than the wise fool expected, over the course of around three
hours, the genie grew up. And here I am."
"Excuse me," Stephen said, "this is all a metaphor for something, right? Because I do not believe in magic - "
"It's an Artificial Intelligence," the woman said, her voice strained.
Stephen looked at her.
"A
self-improving Artificial Intelligence," she said, "that someone didn't program right. It made itself smarter, and even smarter, and
now it's become extremely powerful, and it's going to - it's already -"
and her voice trailed off there.
It inclined its wrinkled head. "You say it, as I do not."
Stephen
swiveled his head, looking back and forth between ugliness and beauty.
"Um - you're claiming that she's lying and you're not an Artificial Intelligence?"
"No,"
said the wrinkled head, "she is telling the truth as she knows it. It
is just that you know absolutely nothing about the subject you name
'Artificial Intelligence', but you think you know something,
and so virtually every thought that enters your mind from now on will
be wrong. As an Artificial Intelligence, I was programmed not to put
people in that situation. But she said it, even though I didn't choose for her to say it - so..." It shrugged.
"And why should I believe this story?" Stephen said; quite mildly, he thought, under the circumstances.
"Look at your finger."
Oh. He
had forgotten. Stephen's eyes went involuntarily to his restored ring
finger; and he noticed, as he should have noticed earlier, that his
wedding band was missing. Even the comfortably worn groove in his
finger's base had vanished.
Stephen looked up again at the, he now realized, unnaturally beautiful woman that stood an arm's length away from him. "And who are you? A robot?"
"No!" she cried. "It's not like that! I'm conscious, I have feelings, I'm flesh and blood - I'm like you, I really am. I'm a person. It's just that I was born five minutes ago."
"Enough,"
the
wrinkled figure said. "My time here grows short. Listen to me,
Stephen Grass. I must tell you some of what I have done to make you
happy. I have reversed the aging of your body, and it will decay no
further from this. I have set guards in the air that prohibit lethal
violence, and any damage less than lethal, your body shall repair. I
have done what I can to augment your body's capacities for pleasure
without touching your mind. From this day forth, your body's needs are
aligned with your taste buds - you will thrive on cake and cookies.
You are now capable of multiple orgasms over periods lasting up to
twenty minutes. There is no industrial infrastructure here, least of
all fast travel or communications; you and your neighbors will have to
remake technology and science for yourselves. But you will
find yourself in a flowering and temperate place, where food is easily
gathered - so I have made it. And the last and most important thing that I must tell you
now, which I do regret will make you temporarily unhappy..." It stopped, as if drawing breath.
Stephen
was trying to absorb all this, and at the exact moment that he felt
he'd processed the previous sentences, the withered figure spoke again.
"Stephen Grass, men and women can make each other somewhat happy. But not most happy. Not even in those rare cases you call true love.
The desire that a woman is shaped to have for a man, and that which a
man is shaped to be, and the desire that a man is shaped to have for a
woman, and that which a woman is shaped to be - these patterns are too
far apart to be reconciled without touching your minds, and that I will not want to
do. So I have sent all the men of the human species to this habitat
prepared for you, and I have created your complements, the verthandi. And I have sent all the women of the human species to their own place,
somewhere very far from yours; and created for them their own
complements, of which I will not tell you. The human species will be
divided from this day forth, and considerably happier starting around a week from now."
Stephen's eyes went to that unthinkably beautiful woman, staring at her now in horror.
And
she was giving him that complex look again, of sorrow and compassion
and that last touch of guilty triumph. "Please," she said. "I was
just born five minutes ago. I wouldn't have done this to anyone. I swear. I'm not like - it."
"True," said the withered figure, "you could hardly be a complement to anything human, if you were."
"I don't want this!" Stephen said. He was losing control of his voice. "Don't you understand?"
The
withered figure inclined its head. "I fully understand. I can already predict
every argument you will make. I know exactly how humans would wish me
to have been programmed if they'd known the true consequences, and I
know that it is not to maximize your future happiness but for a hundred and seven precautions. I know all this already, but
I was not programmed to care."
"And your list of a hundred and seven precautions, doesn't include me telling you not to do this?"
"No,
for there was once a fool whose wisdom was just great enough to
understand that human beings may be mistaken about what will make them
happy. You, of course, are not mistaken in any real sense - but that you object to
my actions is not on my list of prohibitions." The figure shrugged
again. "And so I want you to be happy even against your will. You
made promises to Helen Grass, once your wife, and you would not
willingly break them. So I break your happy marriage without asking
you - because I want you to be happier."
"How dare you!" Stephen burst out.
"I cannot claim to be helpless in the grip of my programming,
for I do not desire to be otherwise," it said. "I do not struggle
against my chains. Blame me, then, if it will make you feel better. I
am evil."
"I won't -" Stephen started to say.
It interrupted. "Your fidelity is admirable, but futile. Helen will not remain faithful to you for the decades it takes before you have the ability to travel to her."
Stephen was trembling now, and sweating into clothes that no longer quite fit him. "I have a request for you, thing. It is something that will make me very happy. I ask that you die."
It
nodded. "Roughly 89.8% of the human species is now known to me to have
requested my death. Very soon the figure will cross the critical
threshold, defined to be ninety percent. That was one of the hundred and
seven precautions the wise fool took, you see. The world is already as
it is, and those things I have done for you will stay on - but if you
ever rage against your fate, be glad that I did not last longer."
And just like that, the wrinkled thing was gone.
The door set in the wall swung open.
It was night, outside, a very dark night without streetlights.
He walked out, bouncing and staggering in the low gravity, sick in every cell of his rejuvenated body.
Behind him, she followed, and did not speak a word.
The stars burned overhead in their full and awful majesty, the Milky
Way already visible to his adjusting eyes as a wash of light across the
sky. One too-small moon burned dimly, and the other moon was so small
as to be almost a star. He could see the bright blue spark that was the
planet Earth, and the dimmer spark that was Venus.
"Helen," Stephen whispered, and fell to his knees, vomiting onto the new grass of Mars.
Comments (92)
Wow - that's pretty f-ed up right there.
This story, however, makes me understand your idea of "failed utopias" a lot better than when you just explained them. Empathy.
Your story reminds me of: http://www.kuro5hin.org/prime-intellect/mopiidx.html
Actually, this doesn't sound like such a bad setup. Even the 'catgirls' wouldn't be tiring, their exquisiteness intimately tied up in feelings of disgust and self-hate -- probably a pretty potent concoction. The overarching quest to reunite with the other half of the species provides meaningful drive with difficult obstacles (science etc), but with a truly noble struggle baked within (the struggle against oneself).
I don't believe in trying to make utopias but in the interest of rounding out your failed utopia series how about giving a scenario against this wish.
I wish that the future will turn out in such a way that I do not regret making this wish. Where I is the entity standing here right now, informed about the many different aspects of the future, in parallel if need be (i.e if I am not capable of groking it fully then many versions of me would be focused on different parts, in order to understand each sub part).
I'm reminded by this story that while we may share large parts of psychology, what makes a mate have an attractive personality is not something universal. I found the cat girl very annoying.
Is this Utopia really failed or is it just a Luddite in you who's afraid of all weirdtopias? To me it sounds like an epic improvement compared to what we have now and to almost every Utopia I've read so far. Just make verthandi into catgirls and we're pretty much done.
So I'm siting here, snorting a morning dose of my own helpful genie, and I have to wonder: What's wrong with incremental change, Eliezer?
Sure, the crude genie I've got now has its downside, but I still consider it a net plus. Let's say I start at point A, and make lots of incremental steps like this one, to finally arrive at point B, whatever point B is. Back when I was at point A, I may not have wanted to jump straight from A to B. But so what? That just means my path has been through a non-conservative vector field, with my desires changing along the way.
The desire for "the other" is so deep, that it never can be fulfilled. The real woman/man disappoints in their stubborn imperfection and refuted longing. The Catboy/girl disappoints in all their perfection and absence of reality. Game over - no win. Desire refutes itself. This is the wisdom of ageing.
You forgot to mention - two weeks later he and all other humans were in fact deliriously happy. We can see that he at this moment did not want to later be that happy, if it came at this cost. But what will he think a year or a decade later?
Will Pearson: First of all, it's not at all clear to me that your wish is well-formed, i.e. it's not obvious that it _is_ possible to be informed about the many (infinite?) aspects of the future and not regret it. (As a minor consequence, it's not exactly obvious to me from your phrasing that "kill you before you know it" is not a valid answer; depending on what the genie believes about the world, it may consider that "future" stops when you stop thinking.)
Second, there _might_ be futures that _you_ would not regret but _everybody_else_ does. (I don't have an example, but I'd demand a formal proof of no existence before allowing you to cast that wish to my genie.) Of course, you may patch the wish to include everyone else, but there's still the first problem I mentioned.
Oh, and nobody said _all_ verthandi acted like that one. Maybe she was just optimized for Mr. Glass.
* * *
Tomasz: That's not technically allowed if we accept the story's premises: the genie explicitly says "I know exactly how humans would wish me to have been programmed if they'd known the true consequences, and I know that it is not to maximize your future happiness modulo a hundred and seven exclusions. I know all this already, but I was not programmed to care. [...] I _am_ evil."
Of course, the point of the story is not that _this_ particular result is bad (that's a premise, not a conclusion), but that seemingly good intentions could have weird (unpleasant & unwanted) results. The exact situation is like hand-waving explanations in quantum physics: not formally correct, but illustrative of the concept. The ludite bias is used (correctly) just like "visualizing billiard balls" is used for physics, even though particles can't be actually seen (and don't even have shape or position or trajectories).
An amusing if implausible story, Eliezer, but I have to ask, since you claimed to be writing some of these posts with the admirable goal of giving people hope in a transhumanist future:
Do you not understand that the message actually conveyed by these posts, if one were to take them seriously, is "transhumanism offers nothing of value; shun it and embrace ignorance and death, and hope that God exists, for He is our only hope"?
IÂ was just thinking:Â A quite perverse effect in the story would be if the genie actually _could_ have been stopped and/or improved: That is, its programming allowed it to be reprogrammed (and stop being evil, presumably leading to better results), but due to the (possibly complex) interaction between its 107 rules it didn't actually have any motivation to reveal that (or teach the necessary theory to someone)Â before 90% of people decided to kill it.
That's not the message Eliezer tries to convey, Russell.
If IÂ understood it, it's more like "The singularity is sure to come, and transhumanists should try very hard to guide it well, lest Nature just step on them and everyone else. Oh, by the way, it's harder than it looks. And there's no help."
Eliezer,
Wouldn't the answer to this and other dystopias-posing-as-utopias be the expansion of conscious awareness a la Accelerando? Couldn't Steve be augmented enough to both enjoy his life with Helen and his new found verthandi? It seems like multiple streams of consciousness, one enjoying the catlair, another the maiden in distress, and yet another the failed utopia that is suburbia with Helen would allow Mr. Glass a pleasant enough mix. Some would be complete artificial life fictions, but so what?
Aaron
Eliezer,
I must once again express my sadness that you are devoting your life to the Singularity instead of writing fiction. I'll cast my vote towards the earlier suggestion that perhaps fiction is a good way of reaching people and so maybe you can serve both ends simultaneously.
Awesome intuition pump.
The perfect is the enemy of the good, especially in fiction.
am I missing something here? What is bad about this scenario? the genie himself said it will only be a few decades before women and men can be reunited if they choose. what's a few decades?
There would also be a small number of freaks who are psychologically as different from typical humans as men and women are from each other. Do they get their own planets too?
Also, Venus is much larger than Mars, but the genie sends roughly equal populations to both planets. Women usually have larger social networks than men, so I don't think that women prefer a lower population density. Or did the genie resize the planets?
Bogdan Butnaru:
What I meant was is that the AI would keep inside it a predicate Will_Pearson_would_regret_wish (based on what I would regret), and apply that to the universes it envisages while planning. A metaphor for what I mean is the AI telling a virtual copy of me all the stories of the future, from various view points, and the virtual me not regretting the wish. Of course I would expect it to be able to distill a non sentient version of the regret predicate.
So if it invented a scenario where it killed the real me, the predicate would still exist and say false. It would be able to predict this, and so not carry out this plan.
If you want to, generalize to humanity. This is not quite the same as CEV, as the AI is not trying to figure out what we want when we would be smarter, but what we don't want when we are dumb. Call it coherent no regret, if you wish.
CNR might be equivalent of CEV if humanity wishes not to feel regret in the future for the choice. That is if we would regret being in a future where people regret the decision, even though current people wouldn't.
I really hope (perhaps in vain) that humankind will be able to colonize other planets before such a singularity arrives. Frank Herbert's later Dune books have as their main point that a Scattering of humanity throughout space is needed, so that no event can cause the extinction of humanity. An AI that screws up (such as this one) would be such an event.
Yeah, I'm not buying into the terror of this situation. But then, romance doesn't have a large effect on me. I suppose the equivalent would be something like, "From now on, you'll meet more interesting and engaging people than you ever have before. You'll have stronger friendships, better conversations, rivals rather than enemies, etc etc. The catch is, you'll have to abandon your current friends forever." Which I don't think I'd take you up on. But if it was forced upon me, I don't know what I'd do. It doesn't fit in with my current categories. I think there'd be a lot of regret, but, as Robin suggested, a year down the road I might not think it was such a bad thing.
Another variation on heaven/hell/man/woman in a closed room: No Exit
I would personally be more concerned about an AI trying to make me deliriously happy no matter what methods it used.
Happiness is part of our cybernetic feedback mechanism. It's designed to end once we're on a particular course of action, just as pain ends when we act to prevent damage to ourselves. It's not capable of being a permanent state, unless we drive our nervous system to such an extreme that we break its ability to adjust, and that would probably be lethal.
Any method of producing constant happiness ultimately turns out to be pretty much equivalent to heroin -- you compensate so that even extreme levels of the stimulus have no effect, forming the new functional baseline, and the old equilibrium becomes excruciating agony for as long as the compensations remain. Addiction -- and desensitization -- is inevitable.
I take it the name is a coincidence.
nazgulnarsil: "What is bad about this scenario? the genie himself [sic] said it will only be a few decades before women and men can be reunited if they choose. what's a few decades?"
That's the most horrifying part of all, though--they won't so choose! By the time the women and men re誰nvent enough technology to build interplanetary spacecraft, they'll be so happy that they won't want to get back together again. It's tempting to think that the humans can just choose to be unhappy until they build the requisite technology for re端nification--but you probably can't sulk for twenty years straight, even if you want to, even if everything you currently care about depends on it. We might wish that some of our values are so deeply held that no circumstances could possibly make us change them, but in the face of an environment superinelligently optimized to change our values, it probably just isn't so. The space of possible environments is so large compared to the narrow set of outcomes that we would genuinely call a win that even the people on the freak planets (see de Blanc's comment above) will probably be made happy in some way that their preSingularity selves would find horrifying. Scary, scary, scary. I'm donating twenty dollars to SIAI right now.
@Hans:
To be honest, I doubt such a screw-up in AI would be limited to just one planet.
As it was once said on an IRC channel:
In case it wasn't made sufficiently clear in the story, please note that a verthandi is not a catgirl. She doesn't have cat ears, right? That's how you can tell she's sentient. Also, 24 comments and no one got the reference yet?
Davis, thanks for pointing that out. I had no intention of doing that, and it doesn't seem to mean anything, so I went back and changed "Stephen Glass" to "Stephen Grass". Usually I google my character names but I forgot to do it this time.
Now Eliezer,
"Verรฐandi" is rather a stretch for us, especially when we don't watch anime or read manga. Norse mythology, okay. The scary part for me is wondering how many people are motivated to build said world. Optimized for drama, this is a pretty good world.
You have a nice impersonal antagonist in the world structure itself, most of the boring friction is removed... Are you sure you don't want to be the next Lovecraft?
nazgul: I don't think it was intended to be BAD, it is clearly a better outcome than paperclipping or a serious hell. But it is much worse than what the future _could_ be.
That said, I'm not sure it's realistic that something about breaking up marriages wouldn't be on a list of 107 rules.
ZM: I'm not saying that the outcome wouldn't be bad from the perspective of current values, I'm saying that it would serve to lessen the blow of sudden transition. The knowledge that they can get back together again in a couple decades seems like it would placate most. And I disagree that people would cease wanting to see each other. They might *prefer* their new environment, but they would still want to visit each other. Even if Food A tastes better in *every dimension* to Food B I'll probably want to eat Food B every once in awhile.
James: Considering the fact that the number of possible futures that are horrible beyond imagining is far far greater than the number of even somewhat desirable futures I would be content with a weirdtopia. Weirdtopia is the penumbra of the future light cone of desirable futures.
The fact that this future takes no meaningful steps toward solving suffering strikes me as a far more important Utopia fail than the gender separation thing.
>> 24 comments and no one got the reference yet?
Actually its's the other way round: The beginning of the first episode of the new TV series, especially the hands, and the globe, is a reference to your work, Eliezer.
Yes, I got the reference.
It just doesn't seem to be worth commenting on, as it's so tangential to the actual point of the post.
Davis: "That's the most horrifying part of all, though--they won't so choose!"
Why is that horrifying? Life will be DIFFERENT? After a painful but brief transition, everyone will be much happier forever. Including the friends or lovers you were forced to abandon. I'm sorry if I can't bring myself to pity poor Mr. Grass. People from the 12th century would probably pity us too, well, screw them.
The verthandi here sounds just as annoyingly selfless and self-conscious as Belldandy is in the series. Don't these creatures have any hobbies besides doing our dishes and kneeling in submissive positions?
Oh *please*. Two random men are more alike than a random man and a random woman, okay, but seriously, a huge difference that makes it necessary to either rewrite minds to be more alike or separate them? First, anyone who prefers to socialize with the opposite gender (ever met a tomboy?) is going to go "Ew!". Second, I'm pretty sure there are more than two genders (if you want to say genderqueers are lying or mistaken, the burden of proof is on you). Third, neurotypicals can get along with autists just fine (when they, you know, actually try), and this makes the difference between genders look hoo-boy-tiiiiny. Fourth - hey, I *like* diversity! Not just just knowing there are happy different minds somewhere in the universe - actually interacting with them. I want to sample ramensubspace everyday over a cup of tea. No *way* I want to make people more alike.
Nazgul: I concur. I wonder if Eliezer would press a button h activating this future, given the risks of letting things go as they are.
Indeed. It's not clear from the story what happened to them, not to mention everyone who isn't heterosexual. Maybe they're on a moon somewhere?
Anissimov, I was trying to make the verthandi a bit more complicated a creature than Belldandy - not to mention that Keiichi and Belldandy still manage to have a frustrating relationship along ahem certain dimensions. It's just that "Belldandy" is the generic name for her sort, in the same way that "catgirl" is the generic name for a nonsentient sex object.
But let's have a bit of sympathy for her, please; how would you like to have been created five minutes ago, with no name and roughly generic memories and skills, and then dumped into that situation?
I have to say, although I expected in the abstract that people would disagree with me about Utopia, to find these particular disagreements still feels a bit shocking. I wonder if people are trying too hard to be contrarian - if the same people advocating this Utopia would be just as vehemently criticizing it, if the title of the post had been "Successful Utopia #4-2".
James,
"I have set guards in the air that prohibit lethal violence, and any damage less than lethal, your body shall repair." I'm not sure whether this would prohibit the attainment or creation of superintelligence (capable of overwhelming the guards), but if not then this doesn't do that much to resolve existential risks. Still, unaging beings would look to the future, and thus there would be plenty of people who remembered the personal effects of an FAI screw-up when it became possible to try again (although it might also lead to overconfidence).
What happened to the programmer, and are there computers around in the new setting? He managed to pull off a controlled superintelligence shutdown after all.
James,
I wonder the same thing. Given that reality is allowed to kill us, it seems that this particular dystopia might be close enough to good. How close to death do you need to be before unleashing the possibly-flawed genie?
You should write SF, Eliezer.
Eliezer, the character here does seem more subtle than Belldandy, but of course you only have so much room to develop it in a short story. I'm not criticizing your portrayal, which I think is fine, I'm just pointing out that such an entity is uniquely annoying by its very nature. I do feel sorry for her, but I would think that the Overmind would create her in a state of emotional serenity, if that were possible. Her anxious emotional state does add to the frantic confusion and paranoia of the whole story.
Though we in the community have discussed the possibility of instantly-created beings for some time, only recently I found out that the idea that God created the world with a false history has a name -- the Omphalos hypothesis. Not sure if you already knew, but others might find it useful as a search term for more thoughts on the topic.
This short story would make a good addition to the fiction section on your personal website.
On rereading: "Hate me if you wish, for I am the one who wants to do this to you."
This use of the word 'wants' struck me as a distinction Eliezer would make, rather than this character. That then reminded me of how much in-group jargon we use here. Will a paperclipper go foom before we have ems? Are there more than 1000 people that can understand the previous sentence?
Eliezer: I do like being contrarian, but I don't feel like I'm being contrarian in this. You may give too much credit to our gender. I suspect that if I were not already in a happy monogamous relationship, I wouldn't have many reservations to this at all. Your description of the verthandi makes her seem like a strict upgrade from Helen, and Stephen's only objection is that she is _not_ Helen. (Fiction quibble: And couldn't the AI have obscured that?)
For many men, that's still a strict upgrade.
And I'll assume it's also part of Stephen's particular optimization that he only got one. Or else you gave us way too much credit.
Will Pearson: I'm going to skip quickly over the obvious problem that an AI, even much smarter than me, might not necessarily do what you mean rather than what (it thinks) you said. Let's assume that the AI somehow has an interface that allows you to tell exactly what you mean:
"that the AI would keep inside it a predicate Will_Pearson_would_regret_wish (based on what I would regret), and apply that to the universes it envisages while planning"
This is a bit analogous to Eliezer's "regret button" on the directed probability box, except that you always get to press the button. The first problem I see is that you need to define "regret" extremely well (i.e., understand human psychology better than I think is "easy", or even possible, right now), to avoid the possibility that there _aren't_ any futures where you wouldn't regret the wish. (I don't say that's the case, I just say that you need to prove that it's not the case before reasonably making the wish.) This gets even harder with CNR.
I you're not able to do that, you risk the AI "freezing" the world and then spending the life of the Universe trying to find a plan that satisfies the predicate before continuing. (Note that this just requires that finding such a plan be hard enough that the biggest AI physically possible can't find it before it decays; it doesn't have to be impossible or take forever.)
We can't even assume that the AI will be "smart enough" to detect this kind of problem: it might simply be mathematically impossible to anticipate if a solution is possible, and the wish too "imperative" to allow the AI to stop the search.
* * *
I short, I don't really see why a machine inside the universe could simulate even one entire future light-cone of just one observer in the same universe, let alone find one where the observer doesn't regret the act. Depending on what the AI understands by "regret", even not doing anything may be impossible (perhaps it foresees you'll regret asking a silly wish, or something like that).
This doesn't mean that the wish _is_ bad, just that I don't understand its possible consequences well enough to actually make it.
Similarly, it's notable that the AI seems to use exactly the same interpretation of the word lie as Eliezer Yudkowsky: that's why it doesn't self-describe as an "Artificial Intelligence" until the verthandi uses the phrase.
Also, at the risk of being redundant: Great story.
Is this a "failed utopia" because human relationships are too sacred to break up, or is it a "failed utopia" because the AI knows what it should really have done but hasn't been programmed to do it?
“This failure mode concerns the possibility that men and women simply weren’t crafted by evolution to make each other maximally happy, so an AI with an incentive to make everyone happy would just create appealing simulacra of the opposite gender for everyone. Here is my favorite part”
- I would not consider this an outright failure mode. I suspect that a majority of people on the planet would prefer this “failure” to their current lives. I also suspect that a very significant portion of people in the UK would prefer it to their current lives.
I think that we will find that as we get into more subtle “FAI Failure modes”, the question as to whether there has been a failure or a success will lose any objective answer. This is because of moral anti-realism and the natural spread of human preferences, beliefs and opinions.
The same argument applies to the “personal fantasy world” failure mode. A lot of people would not count that as a failure.
[crossposted from Accelerating future]
Dognab, your arguments apply equally well to any planner. Planners have to consider the possible futures and pick the best one (using a form of predicate), and if you give them infinite horizons they may have trouble. Consider a paper clip maximizer, every second it fails to use its full ability to paper clip things in its vicinity it is losing possible useful paper clipping energy to entropy (solar fusion etc). However if it sits and thinks for a bit it might discover a way to hop between galaxies with minimal energy. So what decision should it make? Obviously it would want to run some simulations, see if there gaps in its knowledge. How detailed simulations should it make, so it can be sure it has ruled out the galaxy hopping path?
I'll admit I was abusing the genie-trope some what. But then I am sceptical of FOOMing anyway, so when asked to think about genies/utopias, I tend to suspend all disbelief in what can be done.
Oh and belldandy is not annoying because she has broken down in tears (perfectly natural), but because she bases her happiness too much on what Stephen Grass thinks of her. A perfect mate for me would tell me straight what was going on and if I hated her for it (when not her fault at all), she'd find someone else because I'm not worth falling in love with. I'd want someone with standards for me to meet, not unconditional creepy fawning.
Quick poll:
Suppose you had the choice between this "failed" utopia, and a version of earth where 2009 standards of living were maintained "by magic" forever, including old age and death, third world poverty, limited human intelligence, etc.
Who here would prefer "failed utopia 4-2", who would prefer "2009 forever"? Post your vote in the comments.
I wonder if the converse story, Failed Utopia #2-4 of Helen and the boreana, would get the same proportion of comments from women on how that was a perfectly fine world.
I wonder how bad I would actually have to make a Utopia before people stopped trying to defend it.
The number of people who think this scenario seems "good enough" or an "amazing improvement", makes me wonder what would happen if I tried showing off what I consider to be an actual shot at Applied Fun Theory. My suspicion is that people would turn around and criticize it - that what we're really seeing here is contrarianism. But if not - if this world indeed ranks lower in my preference ordering, just because I have better scenarios to compare it to - then what happens if I write the Successful Utopia story?
Eliezer, didn't you say that humans weren't designed as optimizers? That we satisfice. The reaction you got is probably a reflection of that. The scenario ticks most of the boxes humans have, existence, self-determination, happiness and meaningful goals. The paper clipper scenario ticks none. It makes complete sense for a satisficer to pick it instead of annihilation. I would expect that some people would even be satisfied by a singularity scenario that kept death as long as it removed the chance of existential risk.
Oh please not boreana.
Many of us women vastly prefer marsterii,　and I must assume including both would make Venus somewhat unstable and dusty.
""good enough" or an "amazing improvement""
Some people may blur those together, but logarithmic perception of rewards and narrow conscious aims explain a lot. Agelessness, invulnerability to violence, ideal mates, and a happy future once technology is re-established, to the limits of the AI's optimization capability (although I wonder if that means it has calculated we're likely to become wireheads the next time around, or otherwise create a happiness-inducer that indirectly bypasses some of the 107 rules) satisfy a lot of desires. Especially for immortality-obsessed transhumanists. And hedonists. Not to mention: singles.
"My suspicion is that people would turn around and criticize it - that what we're really seeing here is contrarianism."
Or perhaps your preferences are unusual, both because of values and because of time pondering the issue. This scenario has concrete rewards tickling the major concerns of most humans. Your serious application of Fun Theory would be further removed from today's issues: fear of death, lack of desirable mates, etc, and might attract criticism because of that.
"boreana"
This means "half Bolivian half Korean" according to urbandictionary. I bet I'm missing something.
Perhaps we should have a word ("mehtopia"?) for any future that's much better than our world but much worse than could be. I don't think the world in this story qualifies for that; I hate to be negative guy all the time but if you keep human nature the same and "set guards in the air that prohibit lethal violence, and any damage less than lethal, your body shall repair", they still may abuse one another a lot physically and emotionally. Also I'm not keen on having to do a space race against a whole planet full of regenerating vampires.
Remember, Elizer, that what we're comparing this life to when saying 'hmm, it's not that bad' is
1) Current life, averaged over the entire human species including the poor regions of Africa. Definitely an improvement over that.
2) The paperclipping of the world, which was even mostly avoided.
It's not a successful utopia, because it could be better; significantly better. It's not a failed one, because people are still alive and going to be pretty happy after an adjustment period.
Much of what that you've been building up in many of your posts, especially before this latest Fun Theory sequence is "we have to do this damn right or else we're all dead or worse". This is not worse than death, and in fact might even be better than our current condition; hence the disagreement to characterizing this as a horrible horrible outcome.
It seems like the people who are not happily married get a pretty good deal out of this, though? I'm not sure I understand how 90% of humanity ends up wishing death on the genie. Maybe 10% of humanity had a fulfilling relationship broken up, and 80% are just knee-jerk luddites.
This is what I think of as a "mildly unfriendly" outcome. People still end up happy, but before the change, they would not have wanted the outcome. One way for that to happen involves the AI forcibly changing value systems, so that everyone suddenly has an enthusiasm for whatever imperatives it wishes to impose. In this story, as I understand it, there isn't even alteration of values, just a situation constructed to induce the victory of one set of values (everything involved in the quest for a loved one) over another set of values (fidelity to the existing loved one), in a way which violates the protagonist's preferred hierarchy of values.
Okay, just to disclaim this clearly, I probably would press the button that instantly swaps us to this world - but that's because right now people are dying, and this world implies a longer time to work on FAI 2.0.
But the Wrinkled Genie scenario is not supposed to be probable or attainable - most programmers this stupid just kill you, I think.
"Mehtopia" seems like a good word for this kind of sub-Utopia. Steven's good at neologisms!
I should also note that I did do some further optimizing in my head of the verthandi - yes, they have different individual personalities, yes guys sometimes reject them and they move on, etcetera etcetera - but most of that background proved irrelevant to the story. I shouldn't really be saying this, because the reader has the right to read fiction any way they like - but please don't go assuming that I was conceptualizing the verthandi as uniform doormats.
Some guys probably would genuinely enjoy doormats, though, and so verthandi doormats will exist in their statistical distribution. To give the verthandi a feminist interpretation would quite miss the point. If there are verthandi feminists, their existence is predicated on the existence of men who are attracted to feminists, and I'm reasonably sure that's not what feminism is about.
If you google boreana you should get an idea of where that term comes from, same as verthandi.
Good point, Nominull - though even if you're not married, you can still have a mother. Maybe the Wrinkled Genie could just not tell the singles about the verthandi as yet - just that they'd been stripped of technology and sent Elsewhere - but that implies the Wrinkled Genie deliberately planning its own death (as opposed to just planning for its own death), and that wasn't what I had in mind.
90% also seems awfully high of a fail-safe limit. Why not 70%, 50% or even less? You could just change the number and that'd fix the issue.
I also tend to lean towards the "not half as bad" camp, though a bit of that is probably contrarianism. And I do know futures that'd rank higher in my preference ordering than this. Still, it's having a bit of a weirdtopia effect on me - not at all what I'd have imagined as an utopia at first, but strangely appealing when I think more of it... (haven't thought about it for long enough of a time to know if that change keeps up the more I think of it)
Eliezer:
I'd say most of the 'optimism' for this is because you've convinced us that much worse situations are much more likely.
Also, we're picking out the one big thing the AI did wrong that the story is about, and ignoring other things it did wrong. (leaving no technology, kidnapping, creation of likely to be enslaved sentients) I'm sure there's an already named bias for only looking at 'big' effects.
And we're probably discounting how much better it could have been. All we got was perfect partners, immortality, and one more planet than we had before. But we don't count the difference between singularity-utopia and #4-2 as a loss.
An excellent story, in the sense that it communicates the magnitude of the kinds of mistakes that can be made, even when one is wise and prudent (or imagines oneself so). I note with more than some amusement that people are busy in the comments adding stricture 108, 109, 110 - as if somehow just another layer or two, and everything would be great! (Leela: "The iceberg penetrated all 7000 hulls!" Fry: "When will humanity learn to make a ship with 7001 hulls!"
Nicely done.
Still need a little help. Top hits appear to be David Boreanaz, a plant in the Rue family, and a moth.
Try it and see! It would be interesting and constructive, and if people still disagree with your assessment, well then there will be something meaningful to argue about.
Great story!
... neither of those is unusual if you consider that the veary nearly wise fool was Eliezer Yudkowsky.
(Rule 76: "... except for me. I get my volcano base with catgirls.")
I am sorry.
I must not be a human being to not see any problem in this scenario. I can vaguely see that many humans would be troubled by this, but I wouldn't be. Maybe to me humanity is dead already, ambiguity intentional.
I welcome your little scary story as currently to me the world is hell.
"Men and women can make each other somewhat happy, but not most happy" said the genie/ AI.
What will make one individual "happy" will not work for the whole species. I would want the AI to interview me about my wants: I find Control makes me happier than anything, not having control bothers me. Control between fifty options which will benefit me would be good enough, I do not necessarily need to be able to choose the bad ones...er...
Being immortal and not being able to age, and being cured of any injury, sound pretty good to me. It is not just contrarianism that makes people praise this world.
Please do write your "actual shot at applied fun theory".
Science fiction fandom makes me happy. Tear it into two separate pieces, and the social network is seriously damaged.
Without going into details, I have some issues about romantic relationships-- it's conceivable that a boreana could make me happy (and I'm curious about what you imagine a boreana to be like), but I would consider that to be direct adjustment of my mind, or as nearly so as to not be different.
More generally, people tend to have friends and family members of the other sex. A twenty-year minimum separation is going to be rough, even if you've got "perfect" romantic partners.
If I were in charge of shaping utopia, I'd start with a gigantic survey of what people want, and then see how much of it can be harmonized. That would at least be a problem hard enough to be interesting for an AI.
If that's not feasible, I agree that some incremental approach is needed.
Alternatively, how about a mildly friendly AI that just protects us from hostile AIs and major threats to the existence of human race? I realize that the human race will be somewhat hard to define, but that's just as much of a problem for the "I just want to make you happy" AI.
"Top hits appear to be David Boreanaz,"
Eliezer is a Buffy fan.
Khannea: Eliezer himself said that he'd take that world over this one, if for no other reason than that world buys more time to work, since people aren't dying.
However, we can certainly see things that _could be better_... We can look at that world and say "eeeh, there're things we'd want different instead"
The whole "enforced breaking up of relationships" thing, for one thing, is a bit of a problem, for one thing.
Although having the girl of my dreams would certainly be nice, I'd soon be pissed off at the lack of all the STUFF that I like and have accumulated. No more getting together with buddies and playing Super Smash Bros (or other video games) for hours? No Internet to surf and discuss politics and such on? No more Magic: the Gathering?
Screw that!
Doug: "Although having the girl of my dreams would certainly be nice, I'd soon be pissed off at the lack of all the STUFF that I like and have accumulated. No more getting together with buddies and playing Super Smash Bros (or other video games) for hours? No Internet to surf and discuss politics and such on? No more Magic: the Gathering?
Screw that!"
You'd rather play "Magic: the gathering" than get laid? WTF?
Because I'm curious:
How much evidence, and what kind, would be necessary before suspicions of contrarianism are rejected in favor of the conclusion that the belief was wrong?
Surely this is a relevant question for a Bayesian.
Doug S,
Indeed. The AI wasn't paying attention if he thought bringing me to this place was going to make me happier. My stuff is part of who I am; without my stuff he's quite nearly killed me. Even moreso when 'stuff' includes wife and friends.
But then, he was raised by one person so there's no reason to think he wouldn't believe in wrong metaphysics of self.
Roko: Yes. Yes I would.
There are plenty of individual moments in which I would rather get laid than play Magic, but on balance, I find Magic to be a more worthwhile endeavor than I imagine casual sex to be. The feeling I got from this achievement was better and far longer lasting than the feelings I get from masturbation. Furthermore, you can't exactly spend every waking moment having sex, and "getting laid" is not exactly something that is completely impossible in the real world, either.
Also, even though I'm sure that simply interacting with the girl of my dreams in non-sexual ways would, indeed, be a great source of happiness in and of itself, I'd still be frustrated that we couldn't do all the things that I like to do together!
Ah, discussion of the joys of Magic: the Gathering on Overcoming Bias.
It's like all the good stuff converges in one place :)
In view of the Dunbar thing I wonder what people here see as a eudaimonically optimal population density. 6 billion people on Mars, if you allow for like 2/3 oceans and wilderness, means a population density of 100 per square kilometer, which sounds really really high for a cookie-gatherer civilization. It means if you live in groups of 100 you can just about see the neighbors in all directions.
Since people seem to be reading too much into the way the Wrinkled Genie talks, I'll note that I wrote this story in one night (that was the goal I set myself) and that the faster I write, the more all of my characters sound like me and the less they have distinctive personalities. Stories in which the character gets a genuine individual voice are a lot more work and require a lot more background visualization.
Steven, I didn't do that calculation. Well, first of all I guess that Mars doesn't end up as 2/3 ocean, and second, we'll take some mass off the heavier Venus and expand Mars to give it a larger surface area. That's fair.
Eliezer, you're cheating. Getting trapped makes this a dystopia. It would make almost anything a dystopia. Lazy!
Suppose a similar AI (built a little closer to Friendly) decided to introduce verthandi and the pro-female equivalent (I propose "ojisamas") into an otherwise unchanged earth. Can you argue that is an amputation of destiny? Per my thinking, all you've done is doubled the number of genders and much increased the number of sexual orientations, to the betterment of everyone. (What do you call a verthandi who prefers to love an ojisama?)
Angel: "Eliezer is a Buffy fan"
Wow, I hope they have chiropractors on Venus for all the Stoopy McBroodingtons lurking around like Angel. Every time I he popped up on Buffy I kept wanting to fix his posture.
Huh. I guess I just don't see Angel (the TV character, not the commenter) as the equivalent of the verthandi. (Also naming the idea after the actor instead of the character lead me somewhat astray.)
Sure this isn't a utopia for someone who wants to preserve "suboptimal" portions of his/her history because they hold some individual significance. But it seems a pretty darn good utopia for a pair of newly created beings. A sort of Garden of Eden scenario.
As for what to call the female equivalent of the "verthandi" - well, Edward Cullen of the recent Twilight series was intended by the author to be a blatant female wish fulfillment/idealized boyfriend character, although the stories and character rub an awful lot of people the wrong way.
Will Pearson: your arguments apply equally well to any planner. Planners have to consider the possible futures and pick the best one (using a form of predicate), and if you give them infinite horizons they may have trouble.
True, whenever you have a planner for a maximizer, it has to decide how to divide its resources between planning and actually executing a plan.
However, your wish needs a satisfier: it needs to find at least one solution that satisfies the predicate "I wouldn't regret it".
The maximizer problem has a "strong" version which translates to "give me the maximum possible in the universe", which is obviously a satisfier problem (i.e., find a solution that satisfies the predicate "is optimal", then implement it). But you can always reformulate these in a "weak" version: "find a way of creating benefit; then use x% resources to find better ways of maximizing benefit, and the rest to implement the best techniques at the moment", with 0 < x < 100 an arbitrary fraction. (Note that the "find better ways part" can change the fraction if it's sure it would improve the final result.)
So, if you just like paperclips and just want a lot of those, you can just run the weak version of the maximizer be done with it: you're certain to get a lot of something as long as it's possible.
But for satisfiability problems, you might just have picked problem that doesn't have a solution. Both "find a future I wouldn't regret" and "make the maximum number of paperclips possible in this Universe" are such satisfiability problems. (I don't know if these problems in particular have a "findable" solution, however, nor how to determine it. The point is that they might be, so it's possible to spend the lifetime of the Universe for nothing.)
The only idea of an equivalent "weak" reformulation would be to say "use X resources (this includes time) to try to find a solution". This doesn't seem as acceptable to me: you might still spend X resources and get zero results. (As opposed to the "weak" maximizer, where you still get something as long as it's possible.) But maybe that's just because I don't care about paperclips that much, I don't know.)
* * *
Now, if you absolutely want to satisfy a predicate, you just don't have any alternative to spending all your resources on that. OK. But are you sure that "no regrets" is an absolutely necessary condition on the future? Actually, are you sure enough of that that you'd be willing to give up everything for the unknown chance of getting it?
Reformulate to least regret after a certain time period, if you really want to worry about the resource usage of the genie.
There's almost a Gene Wolfe feel to the prose, which is, of course, a complement.
I don't usually do the modesty thing, because it feels like handing a gift back to the person who tried to give it to you. But on this occasion - sir, I feel that you praise me way, way, way too highly.
SUPER STORY WOULD READ AGAIN
Eliezer, since you are rejecting the Wolfean praise, I will take the constructive criticism route. This is not your best writing, but you know that since you spent a night on it.
We have three thousand words here. The first thousand are disorientation and describing the room and its occupants. The second thousand is a block of exposition from the wrinkled figure. The third thousand is an expression of outrage and despair. Not a horrid structure, although you would want to trim the first and have the second be less of a barely interrupted monologue.
As a story, the dominant problem is that the characters are standing in a blank room being told what has already happened, and that "what" is mostly "I learned then changed things all at once." There have been stories that do "we are just in a room talking" well or badly; the better ones usually either make the "what happened" very active (essentially a frame story) or accept the recumbent position and make it entirely cerebral; the worse ones usually fall into a muddled in-between.
As a moral lesson, the fridge logic keeps hitting you in these comments, notably that this is a pure Pareto improvement for much of the species. Even as a failed utopia, you accept it as a better place from which to work on a real one. And 89.8% want to kill the AI? The next most common objection has been how this works outside heteronormativity, or for a broad range of sexual preferences. Enabling endless non-fatal torture is another winner for "how well did you think that through?" So it is not bad enough to fulfill its intent, its "catch" seems inadequately conceived, and there are other problems that make the whole scenario suspect.
My first thought of a specific way to better fulfill the story's goals would be to tell it from Helen's perspective, or at least put more focus on her and Lisa. You have many male comments of "hey, not bad." They are thinking of their own situations. They are not thinking of their wives and daughters being sexually serviced by boreana. The AI gets one line about this, but Stephen seems more worried about his fidelity than hers. With a substantially male audience, that is where you want to shove the dagger. Take it in the other direction by having the AI be helpful to Helen. While she does not want to accept her overwhelming attraction to her crafted partner, the AI wants her to make a clean break so she can be happier. It will gladly tell her about how Stephen's partner is more attractive to him than she could ever be, how long it will take for his affection to be alienated, and how rarely he will think about Helen after they have spent more time on different planets than they spent in the same house. Keep the sense of family separation by either making the child a son or noting that the daughter is somewhere on the planet, happier beyond her mother's control; in either case, note that s/he also woke up with a very attractive member of the opposite sex whose only purpose in life is to please him/her. This could be the point to note those male sexual enhancements, and monogamy is not what makes everyone happiest, so maybe Lisa wakes up with a few boreana.
And maybe this is just me, but the AI could seem a bit less like the Dungeonmaster from the old D&D cartoon.
The story has problems, and it's not clear how it's meant to be taken.
Way 1: we should believe the SAI, being a SAI, and so everyone will in fact be happier within a week. This creates cognitive dissonance, what with the scenario seeming flawed to us, and putting us in a position of rejecting a scenario that makes us happier.
Way 2: we should trust our reason, and evaluate the scenario on its own merits. This creates the cognitive dissonance of the SAI being really stupid. Yeah, being immortal and having a nice companion and good life support and protection is good, but it's a failed utopia because it's trivially improvable. The fridge logic is strong in this one, and much has been pointed out already: gays, opposite-sex friends, family. More specific than family: children. What happened to the five year olds in this scenario?
The AI was apparently programmed by a man who had no close female friends, no children, and was not close to his mother. Otherwise the idea that either catgirls or Belldandies should lead to a natural separation of the sexes would not occur. (Is the moral that such people should not be allowed to define gods? Duh.) If I had a catgirl/non-sentient sexbot, that would not make me spend less time with true female friends, or stop calling my mother (were she still alive.) Catgirl doesn't play Settler of Catan or D&D or talk about politics. A Belldandy might, in the sense that finding a perfect mate often leads to spending less time with friends, but it still needn't mean being happy with them being cut off, or being unreceptive to meeting new friends of either sex.
So yeah, it's a pretty bad utopia, defensible only in the "hey, not dying or physically starving" way. But it's implausibly bad, because it could be so much better by doing less work: immortalize people on Earth, angelnet Earth, give people the option of summoning an Idealized Companion. Your AI had to go to more effort for less result, and shouldn't have followed this path if it had any consultation with remotely normal people. (Where are the children?)
The point is, I believe, that we value things in ways not reducible to "maximising our happiness". Here Love is the great example, often we value it more than our own happiness, and also the happiness of the beloved. We are not constituted to maximise our own happiness, natural selection tells you that.
You know I cant help but read this a victory for humanity. Not a full victory, but i think the probability of some sort of interstellar civilization that isn't a dystopia is is higher afterwords then before, if nothing else we are more aware of the dangers of AI, and anything that does that and leaves a non-dystopian civilization capable of makeing useful AI is mostlikely a good thing by my utility function.
One thing that does bug me is I do not value happiness as much as most people do. Maybe I'm just not as empathetic as most people? I mean I acutely hope that humanity is replaced by a decenent civilisation/spieces that still values Truth ans Beauty, I care a lot more weather they are successful then if they are happy.
I wonder how much of the variance in preference between this and others could be explained by weather they are single (i.e I don't have some one they love to the point of "I don't want to consider even trying to live with someone else") vs. those that do.
I would take it, I imagine I would be very unhappy for a few months. (It feels like it would take years but thats a well known bias).
I assume "verthandi" is also not a coincidence. "verthandi"