I am making a distinction here between subjectivity as you define it, and subjectivity as it is commonly used, i.e. "just a matter of opinion". I think (though could be mistaken) that the test described subjectivism as it just being a matter of opinion, which I would not agree with:

I don't think these two are really different. An "opinion", a "belief", and a "preference" are fundamentally similar; the word used indicates how attached the person is to that state, and how malleable it appears to be. There exist different underlying mechanisms, but these words don't clearly differentiate between them, they don't cut reality at its joints.

Morality depends on individual preferences, but only in the sense that healthcare depends on an individual's health.

How is that different from beliefs or normative statements about the world, which depend on what opinions an individual holds? "Holding an opinion" seems to cash out in either believing something, or having a preference for something, or advocating some action, or making a statement of group allegiance ("my sports team is the best, but that's just my opinion").

Maybe you use the phrase "just an opinion" to signal something people don't actually care about, or don't really believe in, just say but never act on, change far too easily, etc.. That's true of a lot of opinions that people hold. But it's also true of a lot of morals.

It does not preclude a science of morality.

You can always make a science of other people's subjective attributes. You can make a science of people's "just an" opinions, and it's been done - about as well as making a science of morality.

If I take that literally, then moral realism would hold the correct answer, as everything regarding morality concerns empirical fact (As the article you link to tried to explain).

That makes no sense to me. How is it different from saying nothing at all is subjective? This seems to just ignore the definition of "subjective", which is "an attribute of a person, such that you don't know that attribute's value without knowing who the person is". Or, more simply, a "subjective X" is a function from a person to X.

All this is disregarding the empirical question of to what extend our preferences actually overlap - and to what extend we value each other's utility functions an sich. If the overlap/altruism is large enough, we could still end up with de facto objective morality, depending. Has Eliezer ever tried answering this? Would be interesting.

I believe that's where the whole CEV story comes into play. That is, Eliezer believes or believed that while today the shared preferences of all humans form a tiny, mostly useless set - we can't even agree on which of us should be killed! - that something useful and coherent could be "extrapolated" from them. However, as far as I know, he never gave an actual argument for why such a thing could be extrapolated, or why all humans could agree on an extrapolation procedure, and I don't believe it myself.

Subjectivism seems to imply that one person's judgment is no better than another's (which is false)

Subjectivism implies that morals are two-place concepts, just like preferences. Murder isn't moral or immoral, it can only be Sophronius!moral or Sophronius!immoral. This means Sophronius is probably best equipped to judge what is Sophronius!moral, so other people's judgements clearly aren't as good in that sense. But if you and I disagree about what's moral, we may be just confused about words because you're thinking of Sophronius!moral and I'm thinking of DanArmak!moral and these are similar but different things.

Is that what you meant?

As before, I found the question on metaethics (31) to be a tossup because I agree with several of the options given. I'd be interested in hearing from people who agree with some but not all of these answers:

• Non-cognitivism: Moral statements don't express propositions and can neither be true nor false. "Murder is wrong" means something like "Boo murder!".
• Error theory: Moral statements have a truth-value, but attempt to describe features of the world that don't exist. "Murder is wrong" and "Murder is right" are both false statements because moral rightness and wrongness aren't features that exist.
• Subjectivism: Some moral statements are true, but not universally, and the truth of a moral statement is determined by non-universal opinions or prescriptions, and there is no non-attitudinal determinant of rightness and wrongness. "Murder is wrong" means something like "My culture has judged murder to be wrong" or "I've judged murder to be wrong".

I'm a subjectivist: I understand that when someone says "murder is wrong", she's expressing a personal judgement - others can judge differently. But I also know that most people are moral realists, so they wrongly think they are describing features of the world that don't in fact exist; thus, I believe in error theory. And what does it mean to proclaim that something "is wrong", other than to boo it, i.e. to call for people not to do it and to shun those who do? Thus, I also agree with non-cognitivism.

[Survey Taken Thread]

By ancient tradition, if you take the survey you may comment saying you have done so here, and people will upvote you and you will get karma.

Let's make these comments a reply to this post. That way we continue the tradition, but keep the discussion a bit cleaner.

Btw, my ability to get answers correct is apparently better than 99% of the other test takers.

So, your biases are similar to Stefan's, congratulations.

Cool! I've been desperate to see a rationality test and so make improvements in rationality measurable (I think the Less Wrong movement really really needs this) so it's fantastic to see people working on this. I haven't checked the methodology yet but the basic principle of measuring bias seems sound.

last paragraph teaches false lesson on cleverness

What exactly do you believe the false lesson to be and why do you think it's false?

I interpreted it as meaning one should take into account your prior for whether someone with a gambling machine is telling the truth about how the machine works.

The point is that you don't ignore countless people saying the same thing just because you can think of a reason to dismiss them. Even if you are right and that's all it is, you'll still have sinned for not considering it.

Doesn't the very fact that I have a reason imply that I must have considered it?

And at any rate, how is "They got their ideas about rationality from popular fiction" a failure to consider? Things are not always said by countless people because they have merit. And in this case, there's a very well known, fairly obvious, reason why countless people would say such a thing. You may as well ask why countless people think that crashed cars explode.

I translate "I might have my own truth, but other people have their truth as well" as "You might have your perspective, but other people have their own perspectives. No one has the complete truth (territory), so don't state your mere perspective as if it's the complete truth."

Another translation" "You may be certain you're right, but the people you're arguing with are just as certain that they are right."