You're right, in that I can't refute the core statement of a system of ethics.
Perhaps genies should grant wishes- but developing a system that creates a moral imperative for genies to grant wishes doesn't make genies or grant wishes. Even if you believe that it is morally right and proper to build perpetual motion machines, you don't actually get to build perpetual motion machines.
Okay. Now take step two—try to show that, in fact, a 'should' really cannot be derived from an 'is'.
"Perpetual motion machines cannot be built" can be demonstrated to be true based on empirically-observable facts. If "'Should' statements cannot be made from 'is' statements" is a true 'is' statement, it will also be possible to show it is true based entirely on empirically-observable facts, right?
The usual mistake people make at this point is to claim that various "shoulds" contradict what "is". But what people thi...
Today's post, Moral Error and Moral Disagreement was originally published on 10 August 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Sorting Pebbles Into Correct Heaps, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.