Do you believe in an objective morality capable of being scientifically investigated (a la Sam Harris *or others*), or are you a moral nihilist/relativist? There seems to be some division on this point. I would have thought Less Wrong to be well in the former camp.
Edit: There seems to be some confusion - when I say "an objective morality capable of being scientifically investigated (a la Sam Harris *or others*)" - I do NOT mean something like a "one true, universal, metaphysical morality for all mind-designs" like the Socratic/Platonic Form of Good or any such nonsense. I just mean something in reality that's mind-independent - in the sense that it is hard-wired, e.g. by evolution, and thus independent/prior to any later knowledge or cognitive content - and thus can be investigated scientifically. It is a definite "is" from which we can make true "ought" statements relative to that "is". See drethelin's comment and my analysis of Clippy.
In summary of my own current position (and which I keep wanting to make a fuller post thereof):
If factual reality F can represent a function F(M) -> M from moral instructions to moral instructions (e.g. given the fact that burning people hurts them, F("it's wrong to hurt people")-> "It's wrong to burn people"), then there may exist universal moral attractors for our given reality -- these would represent objective moralities that are true for a vast set of different moral starting positions. Much like you reach the Sierpinski Triangle no matter the starting shape.
This would however still not be able to motivate an agent that starts with an empty set of moral instructions.
That sounds likely to me.
The other things that sound possible to me are that we could ultimately determine
We ought not build dangerously uncontrolled (AI, nuclear reactors, chemical reactors) or tolerate those who do. We ought not get sentiences to do things for us through promises of physical pain or other disutility (threats and coercion)j
We may have a somewhat different understanding of what ought means by that point, just as we have a diff... (read more)