The author is far from alone in his view that both a complete rightness criterion and a consistent decision method must be required of all serious moral theories.

Among hedonistic utilitarians it's quite normal to demand both completeness, to include all (human) situations, and consistency, to avoid contradictions. The author simply describes what's normal among consequentialists, who, after all, are more or less the rational ones. ;-) There's one interesting exception though! The demand to include all situations, including the non-human ones, is radical, and quite hard a challenge for hedonistic utilitarians, who do have problems with the bloodthirsty predators of the jungle.

The question needs to cover how one should act in all situations, simply because we want to answer the question. Otherwise we’re left without guidance and with uncertainty.

Well first, we normally don't think of questions like which clothes to wear as being moral. Secondly, we're not left without guidance when morality leaves these issues alone: we have pragmatic reasons, for instance. Thirdly, we will always have to deal with uncertainty due to empirical uncertainty, so it must be acceptable anyway.

There is one additional issue I would like to highlight, an issue which rarely is mentioned or discussed. Commonly, normative ethics only concerns itself with human actions. The subspecies homo sapiens sapiens has understandably had a special place in philosophical discussions, but the question is not inherently only about one subspecies in the universe. The completeness criterion covers all situations in which somebody should perform an action, even if this “somebody” isn’t a human being. Human successors, alien life in other solar systems, and other species on Earth shouldn’t be arbitrarily excluded.

I'd agree, but accounts of normativity which are mind- or society-dependent, such as constructivism would have reason to make accounts of ethics for humanity different from accounts of ethics for nonhumans.

It seems like an impossible task for any moral theory based on virtue or deontology to ever be able to fulfil the criteria of completeness and consistency

I'm not sure I agree there. Usually these theories don't because the people who construct them disagree with some of the criteria, especially #1. But it doesn't seem difficult to make a complete and demanding form of virtue ethics or deontology.

I've been meditating lately on a possibility of an advanced artificial intelligence modifying its value function, even writing some excrepts about this topic.

Is it theoretically possible? Has anyone of note written anything about this -- or anyone at all? This question is so, so interesting for me.

My thoughts led me to believe that it is theoretically possible to modify it for sure, but I could not come to any conclusion about whether it would want to do it. I seriously lack a good definition of value function and understanding about how it is enforced on the agent. I really want to tackle this problem from human-centric point, but i don't really know if anthropomorphization will work here.

It seems like hedging is the sort of thing which tends to make the writer sound more educated and intelligent, if possibly more pretentious.

I'm not sure in what way it's unjustified for me to have an intuition that qualia are different from physical structures

It's unjustified in the same way that vilalism was an unjustified explanation of life: it's purely a product of our ignorance. Our perception of subjective experience/first-hand knowledge is no more proof of accuracy than our perception that water breaks pencils.

Intuition pumps supporting the accuracy of said perception either beg the question or multiply entities unnecessarily (as detailed below).

Nothing you said indicates that p-zombies are inconceivable or even impossible.

I disagree. You've said that epiphenominalists hold that having first-hand knowledge is not causally related to our conception and discussion of first-hand knowledge. This premise has no firm justification.

Denying it yields my original argument of inconceivability via the p-zombie world. Accepting it requires multiplying entities unnecessarily, for if such knowledge is not causally efficacious, then it serves no more purpose than vital in vitalism and will inevitably be discarded given a proper scientific account of consciousness, somewhat like this one.

I previously asked for any example of knowledge that was not a permutation of properties previously observed. If you can provide one such an example, this would undermine my position.

Also, regarding the personal things here, I am not surprised that you find it hard to understand me, for two reasons. First, as I have said, I haven't been trying to lay out an entire position anyway, because it is not something that would fit into a few comments on Less Wrong. Second, you are deeply confused about a large number of things.

Of course, you suppose that I am the one who is confused. This is normal for disagreements. But I have good evidence that it is you who are confused, rather than me. You admit that you do not understand what I am saying, calling it "vague hand-waving." In contrast, I understand both what I am saying, and what you are saying. I understand your position quite well, and all of its reasons, along with the ways that you are mistaken. This is a difference that gives me a reason to think that you are the one who is confused, not me.

I agree that it would not be productive to continue a discussion along those lines, of course.

And yet it seems really quite easy to conceive of a p zombie. Merely claiming that consciousness is emergent doesn't change our ability to imagine the presence or absence of the phenomenon.

Not too long ago, it would also have been quite easy to conceive of a world in which heat and motion were two separate things. Today, this is no longer conceivable. If something seems conceivable to you now, that might just be because you don't yet understand how it's actually impossible. To make the jump from "conceivability" (a fact about your bounded mind) to "logically possible" (a fact about reality) is a misstep, and a rather enormous one at that.

But clearly we do have such a reason: that we are conscious, and know this fact through direct experience of consciousness.

By stipulation, you would have typed the above sentence regardless of whether or not you were actually conscious, and hence your statement does not provide evidence either for or against the existence of consciousness. If we accept the Zombie World as a logical possibility, our priors remain unaltered by the quoted sentence, and continue to be heavily weighted toward the Zombie World. (Again, we can easily get out of this conundrum by refusing to accept the logical possibility of the Zombie World, but this seems to be something you refuse to do.)

The claim is that we know consciousness is real due to our experience of it.

This exact statement could have been emitted by a p-zombie. Without direct access to your qualia, I have no way of distinguishing the difference based on anything you say or do, and as such this sentence provides just as much evidence that you are conscious as the earlier quoted statement does--that is to say, no evidence at all.

The fact that this knowledge is causally inefficacious does not change its epistemic value.

Oh, but it does. In particular, for a piece of knowledge to have epistemic value to me (or anyone else, for that matter), I need to have some way of acquiring that knowledge. For me to acquire that knowledge, I must causally interact with it in some manner. If that knowledge is "causally inefficacious", as you put it, by definition I have no way of knowing about it, and it can hardly be called "knowledge" at all, much less have any epistemic value.

Allow me to spell things out for you. Your claims, interpreted literally, would imply the following statements:

1. There exists a mysterious substance called "consciousness" that does not causally interact with anything in the physical universe.
2. Since this substance does not causally interact with anything in the physical universe, and you are part of the physical universe, said substance does not causally interact with you.
3. This means, among other things, that when you use your physical fingers to type on your physical keyboard the words, "we are conscious, and know this fact through direct experience of consciousness", the cause of that series of physical actions cannot be the mysterious substance called "consciousness", since (again) that substance is causally inactive. Instead, some other mysterious process in your physical brain is occurring and causing you to type those words, operating completely independently of this mysterious substance. Moreover, this physical process would occur and cause you to type those same words regardless of whether the mysterious epiphenomenal substance called "consciousness" was actually present.
4. Nevertheless, for some reason you appear to expect me to treat the words you type as evidence of this mysterious, causally inactive substance's existence. This, despite the fact that those words and that substance are, by stipulation, completely uncorrelated.

...Yeah, no. Not buying it, sorry. If you can't seeing the massive improbabilities you're incurring here, there's really not much left for me to say.

I claim that it is "conceivable" for there to be a universe whose psychophysical laws are such that only the collection of physical states comprising my brainstates are conscious, and the rest of you are all p-zombies. Note that this argument is exactly as plausible as the standard Zombie World argument (which is to say, not very) since it relies on the exact same logic; as such, if you accept the standard Zombie World argument, you must accept mine as well. Now then: I claim that by sheer miraculous coincidence, this universe that we are living in possesses the exact psychophysical laws described above (even though there is no way for my body typing this right now to know that), and hence I am the only one in the universe who actually experiences qualia. Also, I would say this even if we didn't live in such a universe.

Prove me wrong.

While I disagree with Eliezer's post

Where? How?

I also disagree with the Zombie world argument as you have presented it.

Well, I disagree with the Zombie World argument, period, so it's possible I may have misrepresented it somehow (though naturally, I don't believe I did). Is there something you specifically disagree with about my phrasing of the Zombie World argument, i.e. some objection that applies to my phrasing, but not to (what you consider) the original?

That said, it is not true that your argument is completely analogous with it. One difference is in number 7.

Okay, so it seems like this is the meat of your objection. This being the case, I'm going to devote a rather larger amount of effort to answering this objection than to what you wrote above. If you feel I didn't focus enough on what you wrote above, again, please feel free to expand on any objections you may have there.

In the first argument, we believe we are living in a world where everyone is conscious for inductive reasons. The fact that other human beings have similar bodies and actions with mine, gives me reason to think that others are conscious just as I am. In your argument, there is simply no reason to accept your #7, since there is no analogy that would lead you to that conclusion.

Well, first off, I personally think the Zombie World is logically impossible, since I treat consciousness as an emergent phenomenon rather than a mysterious epiphenomenal substance; in other words, I reject the argument's premise: that the Zombie World's existence is "conceivable". (That's why I believe every human on the planet is conscious--given the structure of their brains, there's no way for them not to be.)

That being said, if you do accept the Zombie World argument, then there's no reason to believe we live in a universe with any conscious beings. The Zombie World (the one that has no consciousness in it, period) is far simpler than both (1) a universe in which I'm the only conscious one, and (2) a universe in which everyone is conscious. In both of the latter cases, you're saying that there's a mysterious epiphenomenal substance called consciousness that isn't there by necessity; it just happens to be there in order to make all the philosophers of consciousness (and dxu-2) right. Let's repeat that for emphasis: there is literally no reason for consciousness to exist in our universe other than to make David Chalmers right when he writes about consciousness.

If you accept that the Zombie World is conceivable, in other words, the next logical step is not to conclude that by sheer luck, we somehow ended up in a universe with consciousness--no, the next logical step would be to conclude that we ourselves are actually living in the Zombie World. There's no reason to believe that you're conscious, or that I'm conscious, or that anyone is conscious; the Zombie World (assuming it's possible) is strictly simpler than all of those cases.

Remember how, in both arguments, step 7 contained the phrase "just trust me on this one"? That wasn't by accident. In order to accept that we live in a universe with any consciousness at all, you need an absolutely tremendous of faith. True, a universe in which I'm the only conscious being might be slightly more complicated that one where everyone is conscious, but that slight increase in complexity is nothing compared with the huge complexity penalty both hypotheses receive compared with the Zombie World hypothesis (assuming, once again, that you admit the Zombie World hypothesis as a valid hypothesis).

Quoting the last part of your comment once more:

In your argument, there is simply no reason to accept your #7, since there is no analogy that would lead you to that conclusion.

If you reject step 7 of my argument because you feel it is unjustified ("there is no analogy that would lead you to that conclusion"), then you must reject step 7 of (my phrasing of) the original Zombie World argument as well, because compared to the Zombie World itself, the latter claim is virtually just as unjustified as the former. Your objection is acknowledged, but it plays no role in determining the conclusion of the original discussion: you must either accept both arguments as I presented them, or accept neither.

TL;DR: I concede that the final steps of each argument were not exactly analogous. However, this does not change the fact that if you accept one argument, you must accept the other, and hence, my original contention remains unchallenged.

Okay, here's the Zombie World argument, paraphrased:

1. It is "conceivable" (whatever that means) for there to be a universe with physical laws exactly identical to ours, but without the "bridging psychophysical laws" that cause certain physical configurations of atoms to produce subjective awareness, i.e. "consciousness".
2. By assumption, the universe described above is physically identical to ours, right down to the last quark. As a result, there is a planet called "Earth" in this universe, and this planet is populated by humans identical to ourselves; each of us has a counterpart in this other universe. Moreover, each of those counterparts behaves exactly like you or I would, talking to each other, laughing at jokes, and even falling in love.
3. However, since this hypothetical "conceivable" universe lacks the "bridging psychophysical laws" that are necessary for true consciousness to exist, each of those people in that universe, despite acting exactly like you'd expect a conscious being to act, aren't actually conscious, i.e. they don't experience qualia or possess any sense of self-awareness at all. They are, for all intents and purposes, automatons.
4. Since by definition, there is no physical experiment you can perform to distinguish our universe from the Zombie Universe, any observer would have be told, as a separate and independent fact, that "yes, this universe is not the Zombie World--there is actually consciousness in this universe". This is then taken as proof that consciousness must be extra-physical, i.e. epiphenomenal.
5. In both the Zombie World and our universe, people write philosophy papers about consciousness, since (again) the Zombie World and our universe are stipulated to be physically identical, and the act of writing a philosophy paper is a physical act. Incidentally, by the way, this means that the philosophers in the Zombie World are being absolutely crazy, since they're talking about a phenomenon that they have no way of knowing exists, by definition.
6. However, it turns out that our universe's philosophers (whose beliefs about consciousness are no more justified than the Zombie World's philosopher's beliefs) actually are correct about consciousness, because by sheer miraculous coincidence, they happen to be living in a universe with the correct "psychophysical laws" that produce consciousness. They are correct, not because of any logical reasoning on their part (indeed, the reasoning they used must be flawed, since they somehow deduced the existence of a phenomenon they literally have no way of knowing about), but because they just happen to be living in a universe where their statements are true. Yay for them (and us)!
7. Oh, and by the way, we really are living in a universe with consciousness, not the Zombie World. I know that there's literally no way for me to prove this to you (in fact, there's no way for me to know this myself), but just trust me on this one.

And now here's my argument, paraphrased:

1. It is "conceivable" (whatever that means) for there to be a universe with physical laws exactly identical to ours, but whose "bridging psychophysical laws" are such that only those physical configurations of atoms corresponding to my (dxu's) brainstates produce consciousness; nothing else is or can ever be conscious.
2. By assumption, the universe described above is physically identical to ours, right down to the last quark. As a result, there is a planet called "Earth" in this universe, and this planet is populated by humans identical to ourselves; each of us has a counterpart in this other universe. Moreover, each of those counterparts behaves exactly like you or I would, talking to each other, laughing at jokes, and even falling in love. One of those people is a counterpart to me; we'll call him "dxu-2".
3. However, since this hypothetical "conceivable" universe has a different set of "bridging psychophysical laws", each of those people in that universe (with one exception), despite acting exactly like you'd expect a conscious being to act, aren't actually conscious, i.e. they don't experience qualia or possess any sense of self-awareness at all. They are, for all intents and purposes, automatons. Of course, I said there was one exception, and that exception should be obvious: dxu-2 is the only person in this universe who possess consciousness.
4. Since by definition, there is no physical experiment you can perform to distinguish our universe from the Modified Zombie Universe, any observer would have be told, as a separate and independent fact, that "yes, this universe is not the Modified Zombie World--everyone here is conscious, not just dxu-2". This is then taken as proof that consciousness must be extra-physical, i.e. epiphenomenal.
5. In both the Modified Zombie World and our universe, people write philosophy papers about consciousness, since (again) the Modified Zombie World and our universe are stipulated to be physically identical, and the act of writing a philosophy paper is a physical act. Incidentally, by the way, this means that the philosophers in the Modified Zombie World are being absolutely crazy, since they're talking about a phenomenon that they have no way of knowing exists, by definition.
6. Dxu-2, by the way, isn't a professional philosopher, but he's fond of making comments on the Internet that assert he's conscious and that no one else is. Of course, when he makes these comments, his physical self is being exactly as crazy as the other philosophers in the Modified Zombie World, but luckily for dxu-2, the drivel that his physical self types just happens to be exactly right, because by sheer miraculous coincidence, he lives in a universe with the correct "psychophysical laws" that cause him to be conscous.
7. Oh, and by the way, the Modified Zombie World is our universe, and "dxu-2" is actually me. I know I can't prove this to you, but just trust me on this one.

If you accept the Zombie World argument, you have to accept my argument; the two are exactly analogous. Of course, the contrapositive of the above statement is also true: if you reject my argument, you must reject the Zombie World argument. In effect, my argument is a reductio ad absurdum of the Zombie World argument; it shows that given the right motivation, you can twist the Zombie World argument to include/exclude anything you want as conscious. Just say [insert-universe-here] is "conceivable" (whatever that means), and the rest of the logic plays out identically.

P. S. One last thing--this part of your comment here?

No [one] thinks that the real world is a zombie world.

If the Zombie World exists (which I don't believe it does--but if it did), all of the people in that universe (who don't think their world is a zombie world) are dead wrong.