Why is that ?

As far as I understand, if anything like objective morality existed, it would be a property of our physical reality, similar to fluid dynamics or the electromagnetic spectrum or the inverse square law that governs many physical interactions. The same laws of physics that will not allow you to fly to Mars on a balloon will not allow you to perform certain immoral actions (at least, not without suffering some severe and mathematically predictable consequences).

This is pretty much the only way I could imagine anything like an "objective morality" existing at all, and I personally find it very unlikely that it does, in fact, exist.

But our inability to suspend our human values when making those observations doesn't prevent us from acquiring that knowledge.

Not this specific knowledge, no. But it does prevent us (or, at the very least, hinder us) from acquiring knowledge about our values. I never claimed that suspension of values is required to gain any knowledge at all; such a claim would be far too strong.

just the capacity to recognize the necessary structures and carry out its task.

And how would it know which structures are necessary, and how to carry out its task upon them ?

We can imagine the consequences of not having our core values...

Can we really ? I'm not sure I can. Sure, I can talk about Pebblesorters or Babyeaters or whatever, but these fictional entities are still very similar to us, and therefore relateable. Even when I think about Clippy, I'm not really imagining an agent who only values paperclips; instead, I am imagining an agent who values paperclips as much as I value the things that I personally value. Sure, I can talk about Clippy in the abstract, but I can't imagine what it would like to be Clippy.

If you could remove your core values, as in the thought experiment above, would you want to?

It's a good question; I honestly don't know. However, if I did have an ability to instantiate a copy of me with the altered core values, and step through it in a debugger, I'd probably do it.

Not programmed to, or programmed not to? If you can code up a solution to value drift, lets see it. Otherwise, note that Life programmes can update to implement glider generators without being "programmed to".

...with extremely low probability. It's far more likely that the Life field will stabilize around some relatively boring state, empty or with a few simple stable patterns. Similarly, a system subject to value drift seems likely to converge on boring attractors in value space (like wireheading, which indeed has turned out to be a problem with even weak self-modifying AI) rather than stable complex value systems. Paperclippism is not a boring attractor in this context, and a working fully reflective Clippy would need a solution to value drift, but humanlike values are not obviously so, either.

emergent

The Futility of Emergence

By hypothesis, clippers have certain functionalities walled off from update.

A paperclipper no more has a wall stopping it from updating into morality than my laptop has a wall stopping it from talking to me. My laptop doesn't talk to me because I didn't program it to. You do not update into pushing pebbles into prime-numbered heaps because you're not programmed to do so.

Does a stone roll uphill on a whim?

Perhaps you should study Reductionism first.

a totally neutral agent might be able to say that behaviors are less rational than others given the values of the agents trying to execute them, although it wouldn't care as such. But it wouldn't be able to discriminate between the value of end goals.

Agreed, but that goes back to my point about objective morality. If it exists at all (which I doubt), then attempting to perform objectively immoral actions would make as much sense as attempting to fly to Mars in a hot air balloon -- though perhaps with less in the way of immediate feedback.

Why would it take a third person neutral perspective and give that perspective the power to change its goals?

For the same reason anthropologists study human societies different from their own, or why biologists study the behavior of dogs, or whatever. They do this in order to acquire general knowledge, which, as I argued before, is generally a beneficial thing to acquire regardless of one's terminal goals (as long as these goals involve the rest of the Universe of some way, that is). In addition:

A human could use the machine to rewrite his or her morality into supporting human suffering and death, but why would they?

I actually don't see why they necessarily wouldn't; I am willing to bet that at least some humans would do exactly this. You say,

Similarly, Clippy has no need to implement a third-person perspective which doesn't share its values in order to judge how to self-modify...

But in your thought experiment above, you postulated creating machines with exactly this kind of a perspective as applied to humans. The machine which removes my need to sleep (something I personally would gladly sign up for, assuming no negative side-effects) doesn't need to implement my exact values, it just needs to remove my need to sleep without harming me. In fact, trying to give it my values would only make it less efficient. However, a perfect sleep-remover would need to have some degree of intelligence, since every person's brain is different. And if Clippy is already intelligent, and can already act as its own sleep-remover due to its introspective capabilities, then why wouldn't it go ahead and do that ?

I think people at Less Wrong mostly accept that our value system is arbitrary in the same sense, but it hasn't compelled us to try and replace our values.

I think there are two reasons for this: 1). We lack any capability to actually replace our core values, and 2). We cannot truly imagine what it would be like not to have our core values.

That is a good point, I did not think of it this way. I'm not sure if I agree or not, though. For example, couldn't we at least say that un-achievable goals, such as "fly to Mars in a hot air balloon", are sillier than achievable ones ?

Well, a totally neutral agent might be able to say that behaviors are less rational than others given the values of the agents trying to execute them, although it wouldn't care as such. But it wouldn't be able to discriminate between the value of end goals.

But, speaking more generally, is there any reason to believe that an agent who could not only change its own code at will, but also adopt a sort of third-person perspective at will, would have stable goals at all ? If it is true what you say, and all goals will look equally arbitrary, what prevents the agent from choosing one at random ? You might answer, "it will pick whichever goal helps it make more paperclips", but at the point when it's making the decision, it doesn't technically care about paperclips.

Why would it take a third person neutral perspective and give that perspective the power to change its goals?

Changing one's code doesn't demand a third person perspective. Suppose that we decipher the mechanisms of the human brain, and develop the technology to alter it. If you wanted to redesign yourself so that you wouldn't have a sex drive, or could go without sleep, etc, then you could have those alterations made mechanically (assuming for the sake of an argument that it's feasible to do this sort of thing mechanically.) The machines that do the alterations exert no judgment whatsoever, they're just performing the tasks assigned to them by the humans who make them. A human could use the machine to rewrite his or her morality into supporting human suffering and death, but why would they?

Similarly, Clippy has no need to implement a third-person perspective which doesn't share its values in order to judge how to self-modify, and no reason to do so in ways that defy its current values.

My point is that, during the course of its research, it will inevitable stumble upon the fact that its value system is totally arbitrary (unless an absolute morality exists, of course).

I think people at Less Wrong mostly accept that our value system is arbitrary in the same sense, but it hasn't compelled us to try and replace our values. They're still our values, however we came by them. Why would it matter to Clippy?

"better according to some shared, motivating standard or procedure of evaluation",

I broadly agree. My thinking ties shoulds and musts to rules and payoffs. Wherever you are operating a set of rules (which might be as localised as playing chess), you have certain localised "musts".

It seems to me that different people (including different humans) can have different motivating standards and procedures of evaluation, and apparent disagreements about "should' sentences can arise from having different standards/procedures, or disagreement about whether something is better according to a shared standard/procedure.

I'm very resitant to the idea, promoted by EY in the thread you refenced, that the meaning of should changes. Does he think chess players have a different concept of "rule" to poker players?

What about alien "ought"s?

There are no alien oughts, though there are alien desires and alien would-wants. They don't see morality differently from us; the criterion by which they choose is simply not that which we name morality.

There's a human morality in about the same sense as there's a human height.

This is a wonderful epigram, though it might be too optimistic. The far more pessimistic version would be "There's a human morality in about the same sense as there's a human language." (This is what Greene seems to believe and it's a dispute of fact.)

Arbitrary and Bias are not defined properties in formal logic. The bare assertion that they are properties of rationality assumes the conclusion.

Keep in mind that "rationality" has a multitude of meanings, and this community's usage of rationality is idiosyncratic.

Non contradictoriness probably isn't a sufficient condition for truth.

Sure, but the discussion is partially a search for other criteria to evaluate of the truth of moral propositions. Arbitrary is not such a criteria. If you were to taboo arbitrary, I strongly suspect you'd find moral propositions that are inconsistent with being values-neutral.

It's uncontrovesial that rational agents need to update, and that AIs need to self-modify. The claim that values are in either case insulated from updates is the extraordinary one.

I never claimed that it was controversial, nor that AIs didn't need to self-modify, nor that values are exempt.

I'm claiming that updates and self modification do not imply a change of behavior towards behavior desired by humans.

I can build a small toy program to illustrate, if that would help.

Other key problem:

But a supersmart, uper-rational clipper has to be able to update.

has to be able to update

"update"

Please unpack this and describe precisely, in algorithmic terms that I could read and write as a computer program given unlimited time and effort, this "ability to update" which you are referring to.

I suspect that you are attributing Magical Powers From The Beyond to the word "update", and forgetting to consider that the ability to self-modify does not imply active actions to self-modify in any one particular way that unrelated data bits say would be "better", unless the action code explicitly looks for said data bits.