Your explanation is simple and fits the facts!

I like it :)

I do think that there is a way of understanding 'right' and 'wrong' independently of preferences...Morality teaches you when to punish and reward (and when to expect punishment and reward). It is a second-order concept, and hence not directly tied to preferences.

Sociology? Psychology? Game theory? Mathematics? What does moral philosophy add to the sciences that is useful, that helps us to dissolve confusion and understand the nature of reality?

imagine it as a doctrine teaching you how to judge the behavior of others (and to a lesser extent, yourself).

Which metrics do I use to judge others?

There has been some confusion over the word "preference" in the thread, so perhaps I should use "subjective value". Would you agree that the only tools I have for judging others are subjective values? (This includes me placing value on other people reaching a state of subjective high value)

Or do you think there's a set of metrics for judging people which has some spooky, metaphysical property that makes it "better"?

Don't think of morality as a doctrine guiding you as to how to behave. Instead, imagine it as a doctrine teaching you how to judge the behavior of others (and to a lesser extent, yourself).

Yes! That's a point that I've repeated so often to so many different people [not on LW, though] that I'd more-or-less "given up" - it began to seem as futile as swatting flies in summer. Maybe I'll resume swatting now I know I'm not alone.

Based upon my experiences, physical truths appear to be concrete and independent of beliefs and opinions. I see no cases where "right" has a meaning outside of an agent's preferences. I don't know how one would go about discovering the "rightness" of something, as one would a physical truth.

It is a poor analogy.

Edit: Seriously? I'm not trying to be obstinate here. Would people prefer I go away?

New edit: Thanks wedrifid. I was very confused.

Example? I prefer not to stay up late, but here I am doing it. It's not that I'm acting against my preferences, because my current preference is to continue typing this sentence. It's simply that English doesn't differentiate very well between "current preferences"= "my preferences right this moment" and "current preferences"= "preferences I have generally these days."

Seinfeld said it best.

... I'm not down-voting the comments I disagree with.

I down-voted a couple of snide comments from Peter earlier.

there aren't just people fulfilling their preferences.

You missed a word in my original. I said that there were agents trying to fulfill their preferences. Now, per my comment at the end of your subthread with Amanojack, I realize that the word "preferences" may be unhelpful. Let me try to taboo it:

There are intelligent agents who assign higher values to some futures than others. I observe them generally making an effort to actualize those futures, but sometimes failing due to various immediate circumstances, which we could call cognitive overrides. What I mean by that is that these agents have biases and heuristics which lead them to poorly evaluate the consequences of actions.

Even if a human sleeping on the edge of a cliff knows that the cliff edge is right next to him, he will jolt if startled by noise or movement. He may not want to fall off the cliff, but the jolt reaction occurs before he is able to analyze it. Similarly, under conditions of sufficient hunger, thirst, fear, or pain, the analytical parts of the agent's mind give way to evolved heuristics.

definition of morality (that doesn't involve the word preferences) is that set of habits that are most likely to bring long-term happiness to oneself and those around one.

If that's how you would like to define it, that's fine. Would you agree then, that the contents of that set of habits is contingent upon what makes you and those around you happy?

Morality is about having the right preferences, as rationality is about having true beliefs.

I understand what it would mean to have a true belief, as truth is noticeably independent of belief. I can be surprised, and I can anticipate. I have an understanding of a physical world of which I am part, and which generates my experiences.

It does not make any sense for there to be some "correct" preferences. Unlike belief, where there is an actual territory to map, preferences are merely a byproduct of the physical processes of intelligence. They have no higher or divine purpose which demands certain preferences be held. Evolution selects for those which aid survival, and it doesn't matter if survival means aggression or cooperation. The universe doesn't care.

I think you and other objective moralists in this thread suffer from extremely anthropocentric thinking. If you rewind the universe to a time before there are humans, in a time of early expansion and the first formation of galaxies, does there exist then the "correct" preferences that any agent must strive to discover? Do they exist independent of what kinds of life evolve in what conditions?

If you are able to zoom out of your skull, and view yourself and the world around you as interesting molecules going about their business, you'll see how absurd this is. Play through the evolution of life on a planetary scale in your mind. Be aware of the molecular forces at work. Run it on fast forward. Stop and notice the points where intelligence is selected for. Watch social animals survive or die based on certain behaviors. See the origin of your own preferences, and why they are so different from some other humans.

Objective morality is a fantasy of self-importance, and a hold-over from ignorant quasi-religious philosophy which has now cloaked itself in scientific terms and hides in university philosophy departments. Physics is going to continue to play out. The only agents who can ever possibly care what you do are other physical intelligences in your light cone.

Clarity can be difficult. What do you mean by "truth"?

What is the difference between:

"Killing innocent people is wrong barring extenuating circumstances"

and

"Killing innocent people is right barring extenuating circumstances"

How do you determine which one is accurate? What observable consequences does each one predict? What do they lead you to anticipate?

I'm not sure it's possible for my example to be wrong anymore then its possible for 2+2 to equal 3.

What would it take to convince you your example is wrong?

Note how "2+2=4" has observable consequences:

Suppose I got up one morning, and took out two earplugs, and set them down next to two other earplugs on my nighttable, and noticed that there were now three earplugs, without any earplugs having appeared or disappeared - in contrast to my stored memory that 2 + 2 was supposed to equal 4. Moreover, when I visualized the process in my own mind, it seemed that making XX and XX come out to XXXX required an extra X to appear from nowhere, and was, moreover, inconsistent with other arithmetic I visualized, since subtracting XX from XXX left XX, but subtracting XX from XXXX left XXX. This would conflict with my stored memory that 3 - 2 = 1, but memory would be absurd in the face of physical and mental confirmation that XXX - XX = XX.

Does your example (or another you care to come up with) have observable consequences?

What is weasel-like with "near the surface of the earth"?

Your criticisms of "truth" are not so far off, but you're essentially saying that parts of science are wrong so you can be wrong, too. No actually, you think it is OK to flounder around in the field when you're just starting out. Sure, but not when you don't even know what it is you're supposed to be studying - if anything! This is not analogous to physics, where the general goal was clear from the very beginning: figure out what physical mechanisms underly macro-scale phenomena, such as the hardness of metal, conductivity, magnetic attraction, gravity, etc.

You're just running around to whatever you can grab onto to avoid the main point that there is nothing close to a semblance of delineation of what this "field" is actually about, and it is getting tiresome.

Morality is not just social signalling, because it makes sense to say some social signals ("I am higher status than you because I have more slaves") are morally wrong.

That conclusion does not follow. Saying you have slaves is a signal about morality and, depending on the audience, often a bad signal.

This gets silly.

"Do you believe in woojits?" Well, no, I don't.

"Ah, well, if you disbelieve in woojits, then you must know what woojits are! So, what are woojits?" I have no idea.

"But how is that possible? If you don't have a definition for woojits, on what basis do you reject belief in them?" Having a well-defined notion of something is a prerequisite for belief in it; I don't have a well-defined notion of woojits; therefore I don't believe in woojits.

"No, no. You're confused. All woojit-disbelievers have to adopt a broad definition of woojits in order to disbelieve in them; otherwise they would merely disbelieve in a specific woojit." (shrug) OK, if you like, I have a broad definition of woojit... so broad, in fact, that it is effectively identical to my definition of all the other concepts I don't believe in and haven't thought about, which is the overwhelming majority of all possible concepts. For my part, I consider this equivalent to not having a definition of woojit at all.

As I say, this gets silly. It's just arguing about definitions of words.

Now, I would agree that atheists who grow up in theist cultures do have a definition of God, though I disagree with you that it's necessarily broad: I know at least one atheist who was raised Roman Catholic, for example, and the god he disbelieves in is the Roman Catholic god of his youth, and the idea that "God" might conceivably refer to anything else just doesn't have a lot of meaning to him.

I probably don't understand what you mean.

I think that it's easy to be an atheist -- i.e. one doesn't have to make any difficult definitions or arguments to arrive at atheism, and those easy definitions and arguments are correct. If you think it's harder than I do, that would be interesting and could explain why we have such different opinions here.