I can understand such usage. However, we have things like: "I'm trying to cure a particular kind of confusion here". The confusion he is apparently talking about is the conventional view of "ought" and "should" - and it doesn't need "curing".

In fact, it helps us to understand the moral customs of other cultures - rather than labeling them as being full of "bad" heathens - who need to be brought into the light.

If you tell me that "should" has a usage which is unrelated to "right", "good", and "ought", then that usage could be adapted for aliens.

One of the standard usages is "doing this will most enhance your utility". As in "you should kill that motherf@#$%". This is distinct from 'right' and 'good' although 'ought' is used in the same way, albeit less frequently. It is advice, rather than exhortation.

It's counterintuitive to me - and I'm not the only one - if you look at the other comments here.

Aliens could have the "right", "good", "ought" and "should" concept cluster - just as some other social animals can, or other tribes, or humans at other times.

Basically, there are a whole bunch of possible and actual moral frameworks - and these words normally operate relative to the framework under consideration.

There are some people who think that "right" and "wrong" have some kind of universal moral meaning. However most of those people are religious, and think morality comes straight from god - or some such nonsense.

The fact that it is the intuitive use - that only humans ever think of what they should do in the ordinary sense, while aliens do what is babyeating; that looking at a paperclipper's actions conveys no more information about what we should do than looking at evolution or a rockslide - is counterintuitive.

It is the claims along the lines of 'truth value' that are most counterintuitive. The universality that you attribute to 'Right' also requires some translation.

Re: You seem confused (or, perhaps, hysterical).

Uh, thanks :-(

A powerful corporate agent not bound by the law might well choose to assassinate its potential competitors - if it thought it could get away with it. Its competitors are likely to be among those best placed to prevent it from meeting its goals.

Its competitors don't have to want to destroy all humankind for it to want to eliminate them! The tiniest divergence between its goals and theirs could potentially be enough.

Here's my guess at one type of situation Eliezer might be thinking of when calling proposition B false: It is rational (let us stipulate) for a paperclip maximizer to turn all the matter in the solar system into computronium in order to compute ways to maximize paperclips, but "should" does not apply to paperclip maximizers.

Eliezer, I think your proposed semantics of "ought" is confusing, and doesn't match up very well with ordinary usage. May I suggest the following alternative?

ought<X> refer's to X's would-wants if X is an individual. If X is a group, then ought<X> is the overlap between the oughts of its members.

In ordinary conversation, when people use "ought" without an explicit subscript or possessive, the implicit X is the speaker plus the intended audience (not humanity as a whole).

ETA: The reason we use "ought" is to convince the audience to do or not do something, right? Why would we want to refer to ought<humanity>, when ought<speaker+audience> would work just fine for that purpose, and ought<speaker+audience> covers a lot more ground than ought<humanity>?

"There's a human morality in about the same sense as there's a human language." (This is what Greene seems to believe and it's a dispute of fact.)

That seems to hit close to the mark. Human language contains all sorts of features that are more or less universal to humans due to their hardware while also being significantly determined by cultural influences. It also shares the feature that certain types of language (and 'ought' systems) are more useful in different cultures or subcultures.

This is a wonderful epigram, though it might be too optimistic. The far more pessimistic version would be

I'm not sure I follow this. Neither seem particularly pessimistic to me and I'm not sure how one could be worse than the other.

"There are no alien oughts" and "They don't see morality differently from us" - these seem like more bizarre-sounding views on the subject of morality - and it seems especially curious to hear them from the author of the "Baby-Eating Aliens" story.

Jumping recklessly in at the middle: even granting your premises regarding the scope of 'ought', it is not wholly clear that an alien "ought" is impossible. As timtyler pointed out, the Babyeaters in "Three Worlds Collide" probably had a would-want structure within the "ought" cluster in thingspace, and systems of behaviors have been observed in some nonhuman animals which resemble human morality.

I'm not saying it's likely, though, so this probably constitutes nitpicking.

the criterion by which they choose is simply not that which we name morality.

Even if "morality" means "criterion for choosing.."? Their criterion might have a different referent, but that does not imply a different sense. cf. "This planet". Out of the two, sense has more to do with meaning, since it doesn't change with changes of place and time.

I vote no, it wouldn't be

That is plausible.

This is a good framing for explaining the problem - you would not, in fact, try to build the same FAI for Clippies and humans, and then pass it humans as a parameter.

I expect you would build the same FAI for paperclipping (although we don't have any Clippies to pass it as parameter), so I'd appreciate it if you did explain the problem given you believe there is one, since it's a direction that I'm currently working.

Humans are stuff, just like any other feature of the world, that FAI would optimize, and on stuff-level it makes no difference that people prefer to be "free to optimize". You are "free to optimize" in a deterministic universe, it's the way this stuff is (being) arranged that makes the difference, and it's the content of human preference that says it shouldn't have some features like undeserved million-dollar bags falling from the sky, where undeserved is another function of stuff. An important subtlety of preference is that it makes different features of perhaps mutually exclusive possible scenarios depend on each other, so the fact that one should care about what could be and how it's related to what could be otherwise and even to how it's chosen what to actually realize is about scope of what preference describes, not about specific instance of preference. That is, in a manner of speaking, it's saying that you need an Int32, not a Bool to hold this variable, but that Int32 seems big enough.

Furthermore, considering the kind of dependence you described in that post you linked seems fundamental from a certain logical standpoint, for any system (not even "AI"). If you build the ontology for FAI on its epistemology, that is you don't consider it as already knowing anything but only as having its program that could interact with anything, then the possible futures and its own decision-making are already there (and it's all there is, from its point of view). All it can do, on this conceptual level, is to craft proofs (plans, designs of actions) that have the property of having certain internal dependencies in them, with the AI itself being the "current snapshot" of what it's planning. That's enough to handle the "free to optimize" requirement, given the right program.

Hmm, I'm essentially arguing that universal-enough FAI is "computable", that there is a program that computes a FAI for any given "creature", within a certain class of "creatures". I guess this problem is void, since obviously on the too-big-class side, for a small enough class this problem is in principle solvable, and for a big enough class it'll hit problems, if not conceptual then practical.

So the real question is about the characteristics of such class of systems for which it's easier to build an abstract FAI, that is a tool that takes a specimen of this class as a parameter and becomes a custom-made FAI for that specimen. This class needs to at least include humanity, and given the size of humanity's values, it needs to also include a lot of other stuff, for itself to be small enough to program explicitly. I currently expect a class of parameters of a manageable abstract FAI implementation to include even rocks and trees, since I don't see how to rigorously define and use in FAI theory the difference between these systems and us.

This also takes care of human values/humanity's values divide: these are just different systems to parameterize the FAI with, so there is no need for a theory of "value overlaps" distinct from a theory of "systems values". Another question is that "humanity" will probably be a bit harder to specify as parameter than some specific human or group of people.