TheOtherDave comments on Welcome to Less Wrong! (5th thread, March 2013) - Less Wrong

27 Post author: orthonormal 01 April 2013 04:19PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (1750)

You are viewing a single comment's thread. Show more comments above.

Comment author: Bugmaster 24 April 2013 11:40:07PM 0 points [-]

When I try to imagine this, I conclude that I would not use the word "morality" to refer to the thing that we're talking about...

Yes, you are probably right -- but as I said, this is the only coherent meaning I can attribute to the term "objective morality". Laws of physics are objective; people generally aren't.

Comment author: TheOtherDave 24 April 2013 11:53:53PM 3 points [-]

I generally understand the phrase "objective morality" to refer to a privileged moral reference frame.

It's not an incoherent idea... it might turn out, for example, that all value systems other than M turn out to be incoherent under sufficiently insightful reflection, or destructive to minds that operate under them, or for various other reasons not in-practice implementable by any sufficiently powerful optimizer. In such a world, I would agree that M was a privileged moral reference frame, and would not oppose calling it "objective morality", though I would understand that to be something of a term of art.

That said, I'd be very surprised to discover I live in such a world.

Comment author: Bugmaster 25 April 2013 12:34:09AM 0 points [-]

it might turn out, for example, that all value systems other than M turn out to be incoherent under sufficiently insightful reflection, or destructive to minds that operate under them...

I suppose that depends on what you mean by "destructive"; after all, "continue living" is a goal like any other.

That said, if there was indeed a law like the one you describe, then IMO it would be no different than a law that says, "in the absence of any other forces, physical objects will move toward their common center of mass over time" -- that is, it would be a law of nature.

I should probably mention explicitly that I'm assuming that minds are part of nature -- like everything else, such as rocks or whatnot.

Comment author: TheOtherDave 25 April 2013 01:31:48AM 1 point [-]

Sure. But just as there can be laws governing mechanical systems which are distinct from the laws governing electromagnetic systems (despite both being physical laws), there can be laws governing the behavior of value-optimizing systems which are distinct from the other laws of nature.

And what I mean by "destructive" is that they tend to destroy. Yes, presumably "continue living" would be part of M in this hypothetical. (Though I could construct a contrived hypothetical where it wasn't)

Comment author: Bugmaster 25 April 2013 01:58:11AM 1 point [-]

But just as there can be laws governing mechanical systems ... there can be laws governing the behavior of value-optimizing systems which are distinct from the other laws of nature.

Agreed. But then, I believe that my main point still stands: trying to build a value system other than M that does not result in its host mind being destroyed, would be as futile as trying to build a hot air balloon that goes to Mars.

And what I mean by "destructive" is that they tend to destroy.

Well, yes, but what if "destroy oneself as soon as possible" is a core value in one particular value system ?

Comment author: TheOtherDave 25 April 2013 04:33:32AM 2 points [-]

what if "destroy oneself as soon as possible" is a core value in one particular value system ?

We ought not expect to find any significantly powerful optimizers implementing that value system.

Comment author: PrawnOfFate 25 April 2013 11:28:54AM -2 points [-]

Isn't the idea of moral progress based on one reference frame being better than another?

Comment author: TheOtherDave 25 April 2013 01:04:47PM 0 points [-]

Yes, as typically understood the idea of moral progress is based on treating some reference frames as better than others.

Comment author: PrawnOfFate 25 April 2013 01:09:26PM -2 points [-]

And is that valid or not? If you can validly decide some systems are better than others, you are some of the way to deciding which is best.

Comment author: TheOtherDave 25 April 2013 01:43:49PM 0 points [-]

Can you say more about what "valid" means here?

Just to make things crisper, let's move to a more concrete case for a moment... if I decide that this hammer is better than that hammer because it's blue, is that valid in the sense you mean it? How could I tell?

Comment author: MugaSofer 25 April 2013 12:27:03PM -1 points [-]

No, because "better" is defined within a reference frame.