Do you believe in an objective morality capable of being scientifically investigated (a la Sam Harris *or others*), or are you a moral nihilist/relativist? There seems to be some division on this point. I would have thought Less Wrong to be well in the former camp.
Edit: There seems to be some confusion - when I say "an objective morality capable of being scientifically investigated (a la Sam Harris *or others*)" - I do NOT mean something like a "one true, universal, metaphysical morality for all mind-designs" like the Socratic/Platonic Form of Good or any such nonsense. I just mean something in reality that's mind-independent - in the sense that it is hard-wired, e.g. by evolution, and thus independent/prior to any later knowledge or cognitive content - and thus can be investigated scientifically. It is a definite "is" from which we can make true "ought" statements relative to that "is". See drethelin's comment and my analysis of Clippy.
Hank, why would Clippy believe that maximizing paperclips is based on something external to its own mind? Clippy could just as easily be programmed to desire staples, and Clippy is probably intelligent enough to know that.
That said, I think Jack's general point about the relationship between ethics and meta-ethics is probably right.
Presumably Clippy has a hard-coded utility function sitting in his source code somewhere. It's a real set of 0s and 1s sitting on a disk somewhere, and we could open the source file and investigate the code.
Clippy's value system is a specific objective, pre-programmed utility function that's inherent in his design and independent/prior to any knowledge or other cognitive content Clippy eventually gains or invents.
And yes, it could have been easily changed (until such a time as Clippy is all grown up and would prevent such change) to make him desire staples. But then that's a different design and we'd probably better call him Stapley at that point.