I think he got the point across well, for some of the common responses I kind of wish there was a much much shorter and easier to digest version of The Hidden Complexity of Wishes which could fit into a reddit post.

I know that people don't like to read long texts, the longer the text the more likely people are to skip it so I've tried to edit it down to a cliff-notes version that doesn't go into probability pumps etc.

The Hidden Complexity of Wishes also uses too many less-wrong specific terms.

http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent.

Suppose your aged mother is trapped in a burning building, and it so happens that you're in a wheelchair; you can't rush in yourself. You could cry, "Get my mother out of that building!" but there would be no one to hear. Luckily you have, in your pocket, a magic lamp with a genie. Unfortunately it does not have Robin Williams voice and doesn't have a human mind or human morality and is constrained to fulfilling wishes through reasonably physically plausible means.

BOOM! With a thundering roar, the gas main under the building explodes. As the structure comes apart, in what seems like slow motion, you glimpse your mother's shattered body being hurled high into the air.

This is a genie of the second class. No wish is safe. It's smart but smart is not the same thing as sharing all your values and you get what you wish for, not what you want.

You are human. If someone asked you to get their poor aged mother out of a burning building, you might help, or you might pretend not to hear. But it wouldn't even occur to you to explode the building. "Get my mother out of the building" sounds like a much safer wish than it really is, because you don't even consider the plans that you assign extreme negative values.

Perhaps you just have to cover all the bases....

"I wish to move my mother (defined as the woman who shares half my genes and gave birth to me) to outside the boundaries of the building currently closest to me which is on fire; but not by exploding the building; nor by causing the walls to crumble so that the building no longer has boundaries; nor by waiting until after the building finishes burning down for a rescue worker to take out the body..."

All these special cases, the seemingly unlimited number of required patches should hint to you that this is not a good approach. Miss one of the thousands of special cases and you're likely to end up with a horrible outcome.

If your mother's foot is crushed by a burning beam, is it worthwhile to extract the rest of her? What if her head is crushed, leaving her body? What if her body is crushed, leaving only her head? Is Terry Schiavo a person? How much is a chimpanzee worth?

We value many things, and no they are not reducible to valuing happiness or valuing reproductive fitness.

The only safe genie is a genie that shares all your judgment criteria, and at that point, you can just say "I wish for you to do what I should wish for."

To be a safe fulfiller of a wish, a genie must share the same values that led you to make the wish or it may fail to exclude horrible side effects that would lead you to not even consider a plan in the first place.

Giving a goal to an advanced AI which has the potential ability to improve itself is like making a wish and we don't yet know a safe way of giving an AI the instruction "I wish for you to do what I should wish for."