You can argue that having values other than hedonistic utility is mistaken in certain cases. But that doesn't imply that it's mistaken in all cases.

But the logic that Eliezer is using is exactly the same logic that drives somebody who's dying of a horrible disease to refuse antibiotics, because she wants to keep her body natural. And this choice is — well, it isn't wrong, choices can't be "wrong" — but it reflects a very fundamental sort of human bias. It's misguided.

Very well, let's back up Eliezer's argument with some hard evidence. Fortunately, lukeprog has already written a brief review of the neuroscience on this topic. The verdict? Eliezer is right. People value things other than happiness and pleasure. The idea that pleasant feelings are the sole good is an illusion created by the fact that the signals for wanting something and getting pleasure from it are comingled on the same neurons.

So no, Eliezer is not misguided. On the contrary, the evidence is on his side. People really do value more things than just happiness. If you want more evidence consider this thought experiment Alonso Fyfe cooked up:

Assume that you and somebody you care about (e.g., your child) are kidnapped by a mad scientist. This scientist gives you two options:

Option 1: Your child will be taken away and tortured. However, you will be made to believe that your child is living a happy and healthy life. You will receive regular reports and even correspondence explaining how great your child’s life is. Except, they will all be fake. In fact, we will take your child to another location and spend every day peeling off his skin while soaking him in a vat of salt water, among other things.

Option 2: Your child will be taken away, provided with paid medical insurance, an endowment to complete an education, will be hired into a good job, and will be caused to live a healthy and happy life. However, you will be made to believe that your child is suffering excruciating torture. You will be able to hear what you think are your child’s screams coming down the hallway. We will show you video of the torture. It will all be fake, of course, but you will be convinced it is real.

Of course, after you make your choice, we will make you forget that you even had these options presented to you.

What do you choose?

Now, we are not going to kidnap people and make them choose. However, both theories need to explain the fact that the vast majority of parents, for example, report that, in such a situation, they would choose Option 2.

Happiness theory seems to suggest that the agent should choose Option 1. After all, the agent will be happier receiving news (that she believes) that says that her child is living a happy and healthy life. So, if happiness is what she is after, and Option1 delivers more happiness, then Option 1 is the rational choice.

Why do people choose Option 2?

Because happiness theory is wrong. In fact, people do not choose happiness. They choose “making or keeping true the propositions that are the objects of our desires.” In this case, the desire in question is the desire that one’s child be healthy and happy. A person with a desire that “my child is healthy and happy” will select that option that will make or keep the proposition, “my child is healthy and happy” true. That is Option 2.

Choices can be wrong, and that one is. The hippy is simply mistaken about the kinds of differences that exist between "natural" and "non-natural" things, and about how much she would care about those differences if she knew more chemistry and physics. And presumably if she was less mistaken in expectations of what happens "after you die".

As for relating this to Eliezer's argument, a few examples of wrong non-subjective-happiness values is no demonstration that subjective happiness is the only human terminal value. Especially given the introspective and experimental evidence that people care about certain things that aren't subjective happiness.

But the logic that Eliezer is using is exactly the same logic that drives somebody who's dying of a horrible disease to refuse antibiotics, because she wants to keep her body natural.

I see absolutely no reason that people shouldn't be allowed to decide this. (Where I firmly draw the line is people making decisions for other people on this kind of basis.)

I surmise from your comments that you may not be aware that Eliezer's written quite a bit on this matter; http://wiki.lesswrong.com/wiki/Complexityofvalue is a good summary/index ( is one of my favorites). There's a lot of stuff in there that is relevant to your points.

However, you asked me what I think, so here it is...

The wording of your first post in this thread seems telling. You say that "Refusing to become orgasmium is a hedonistic utilitarian mistake, full stop."

Do you want to become orgasmium?

Perhaps you do. In that case, I direct the question to myself, and my answer is no: I don't want to become orgasmium.

That having been established, what could it mean to say that my judgment is a "mistake"? That seems to be a category error. One can't be mistaken in wanting something. One can be mistaken about wanting something ("I thought I wanted X, but upon reflection and consideration of my mental state, it turns out I actually don't want X"), or one can be mistaken about some property of the thing in question, which affects the preference ("I thought I wanted X, but then I found out more about X, and now I don't want X"); but if you're aware of all relevant facts about the way the world is, and you're not mistaken about what your own mental states are, and you still want something... labeling that a "mistake" seems simply meaningless.

On to your analogy:

If someone wants to "keep her body natural", then conditional on that even being a coherent desire[1], what's wrong with it? If it harms other people somehow, then that's a problem... otherwise, I see no issue. I don't think it makes this person "kind of dumb" unless you mean that she's actually got other values that are being harmed by this value, or is being irrational in some other ways; but values in and of themselves cannot be irrational.

[Eliezer] can't stand the idea that scientific discovery is only an instrument to increase happiness, so he makes it a terminal value just because he can.

This construal is incorrect. Say rather: Eliezer does not agree that scientific discovery is only an instrument to increase happiness. Eliezer isn't making scientific discovery a terminal value, it is a terminal value for him. Terminal values are given.

In this discussion we are, of course, ignoring external effects altogether.

Why are we doing that...? If it's only about happiness, then external effects should be irrelevant. You shouldn't need to ignore them; they shouldn't affect your point.

[1]Coherence matters: the difference between your hypothetical hippie and Eliezer the potential-scientific-discoverer is that the hippie, upon reflection, would realize (or so we would like to hope) that "natural" is not a very meaningful category, that her body is almost certainly already "not natural" in at least some important sense, and that "keeping her body natural" is just not a state of affairs that can be described in any consistent and intuitively correct way, much less one that can be implemented. That, if anything, is what makes her preference "dumb". There's no analogous failures of reasoning behind Eliezer's preference to actually discover things instead of just pretend-discovering, or my preference to not become orgasmium.