I wonder how long before an insurance company decides to test cryonics as an excuse. "We respect his belief that he is not dead, but rather in suspended animation."

Not totally IT, but I tried it on Eliezer's "The 5-Second Level". Highlits include:

I won't socially kill you

Hope to reflect on consequentialist grounds

Say, what a vanilla ice cream, and not-indignation, and from green?

Associate to persuade anyone of how you were making the dreadful personal habit displays itself in a concrete example.

Rather you can't bear the 5-second level?

To develop methods of teaching rationality skills, you need more practice to get lost in verbal mazes; we will tend to have our feet on the other person.

Be sufficiently averse to the fire department and see if that suggests anything.

I assume you just copy and pasted characters into the comment box from another source?

I got bored of doing that and just put my text through

def sc(x): return unicode(x).translate(dict([(ord('a') + i, c) for i, c in enumerate(u"ᴀʙᴄᴅᴇꜰɢʜɪᴊᴋʟᴍɴᴏᴘ-ʀꜱᴛᴜᴠᴡxʏᴢ")]))

Different people get upset about all sorts of stimuli, from squirting blood to scraping nails to clicking computers to a microcosm inhabited by adorable inhuman sentients to whom no one gives proper moral consideration .

Actually, if I recall correctly, in the original Friendship is Optimal, once they were constructed, the non-uploaded people received the same moral consideration as those originally human. They were designed to fit into a preconceived world but they weren't slaves. I'm not quite sure whether that feels bad because it's actually bad or because it's so very different from our current methods of manufacturing minds (roll genetic dice, expose to local memes, hope for the best).

Well, I don't really remember the exact boundary between Friend is Optimal vs Caelum est Conterrens, but...

• People who don't want to upload are eternally harassed as society crumbles around them until they give in or die.
• People are lied to and manipulated constantly.
• Everything non-human (puppies, trees, stars, aliens capable of radio communication) is destroyed.
• The uploading process seemed to be destructive only for convenience's sake.

Consider the epistemic state of someone who knows that they have the attention of a vastly greater intelligence than themselves, but doesn't know whether that intelligence is Friendly. An even-slightly-wrong CAI will modify your utility function, and there's nothing you can do but watch it happen.

So you think you can guess that character's desire more accurately than a godlike AI with full access to her mind could?

I personally didn't find the actual experience at Equestria itself terrifying at all. It was a little disturbing at first, but almost all of that was sheer physical disgust or a knee-jerk sour grapes reaction. But it seems to avoid almost all of the pitfalls of failed Utopias everywhere:

• You interact with real, sentient creatures who are independent and have their own values and desires. Thunder is capable of getting hurt, angry, and frustrated with his wife. Limeade is capable of feeling envious of her friend. They are in no way less than any complete human mind, and are given the same moral weight. They satisfy Lavender's values, but only as an effect of satisfying their own values, not as their primary directive. The love and friendship are real.
• You're not isolated from other uploaded humans. There are only very few shards of Equestria that grow around only one upload; most interact with others from their Earthly lives.
• It's not stagnant - jryy, guvf vf n ovg qrongnoyr, V xabj, orpnhfr bs gur Ybbc Vzzbegnyf, but there are always new things to learn and discover and opportunities for growth and enlightenment if you so choose.
• It's not devoid of pain or sadness; it's only devoid of arbitrary pain or sadness. It recognizes that to live a fully human life, you need sadness and frustration sometimes; it just makes sure that the pain is, as Paul Graham said, the pain of running a marathon, not the pain of stepping on a nail. Not everything is perfect.

That said, there were moments of genuine horror, mainly stuff people have pointed out before:

• Perhaps trillions and trillions of sentient alien species were wiped out to expand Celestia's empire.
• The people left behind, who didn't upload, are living in a post-apocalyptic wasteland. Celestia was no doubt capable of arranging for functional societies and amenities for those who chose not to upload, but her primary directive was to satisfy values through friendship and ponies, and making life hell for those who held out made them more likely to upload quickly.
• Fridge Logic: One of Siofra's coworkers said his version of the PonyPad game was like God of War; he brutally slaughtered and tortured ponies as part of Celestia's palace guard. Well, what would happen to his shard of Equestria when he uploaded? Would he be massacring living minds? Presumably the ponies in Horndog Dan's version of Equestria truly desired him and satisfied their own values by having sex with him, but the ones who were killed to satisfy the other colleague's desire for heroism? Presumably his values don't involve killing ponies who are essentially automata who exist only to be killed; he wants to kill genuinely evil enemy minds, not drones. Also, how does Celestia manage to satisfy the values of sociopaths with "friendship and ponies"?

I wonder if a Catholic priest is theologically allowed to kill sinners so long as they never say why

I don't think they are, any more than they are allowed to kill anyone else.

Incidentally, it is currently possible to achieve total happiness, or perhaps a close approximation. A carefully implanted electrode to the right part of the brain, will be more desirable than food to a starving rat, for example. While this part of the brain is called the "pleasure center", it might rather be about desire and reward instead. Nevertheless, pleasure and happiness are by necessity mental states, and it should be possible to artificially create these.

Why should a man who is perfectly content, bother to get up to eat, or perhaps achieve something? He may starve to death, but would be happy to do so. And such a man will be content with his current state, which of course is contentment, and not at all resent his current state. Even a less invasive case, where a man is given almost everything he wants, yet not so much so that he does not eventually become dissatisfied with the amount of food in his belly and decide to put more in, even so there will be higher level motivations this man will lose.

While I consider myself a utilitarian, and believe the best choices are those that maximize the values of everyone, I cannot agree with the above situation. For now, this is no problem because people in their current state would not choose to artificially fulfill their desires via electrode implants, nor is it yet possible to actually fulfill everyone's desires in the real world. I shall now go and rethink why I choose a certain path, if I cannot abide reaching the destination.