It is odd how you apparently seem to think you are using the conventional definition of "should" - when you have several people telling you that your use of "should" and "ought" is counter-intuitive.

Most people are familiar with the idea that there are different human cultures, with somewhat different notions of right and wrong - and that "should" is often used in the context of the local moral climate.

For example:

• If the owner of the restaurant serves you himself, you should still tip him;

• You should not put your elbows on the table while you are eating;

• Women should curtsey - "a little bob is quite sufficient".

The problem here is that "should" is already bound fairly tightly to certain concepts, no matter what sort of verbal definitions people think they're deploying, and if they expand the verbal label beyond that, it has consequences for e.g. how they think aliens and AIs will work, and consequences for how they emotionally experience their own moralities.

I see, and that is an excellent point. Daniel Dennett has taken a similar attitude towards qualia, if I interpret you correctly - he argues that the idea of qualia is so inextricably bound with its standard properties (his list goes ineffable, intrinsic, private, and directly or immediately apprehensible by the consciousness) that to describe a phenomenon lacking those properties by that term is as wrongheaded as using the term elan vital to refer to DNA.

I withdraw my implied criticism.

Re: If the FAI allows machine intelligence researchers to create an uFAI we will all die

Yes, that's probably just the kind of paranoid delusional thinking that a psychopathic superintelligence with no respect for the law would use to justify its murder of academic researchers.

Hopefully, we won't let it get that far. Constructing an autonomous tool that will kill people is conspiracy to murder - so hopefully the legal system will allow us to lock up researchers who lack respect for the law before they do some real damage. Assassinating your competitors is not an acceptable business practice.

Hopefully, the researchers will learn the error of their ways before then. The first big and successful machine intelligence project may well be a collaboration. Help build my tool, or be killed by it - is a rather aggressive proposition - and I expect most researchers will reject it, and expend their energies elsewhere - hopefully on more law-abiding projects.

It is a misconception to think of law as a set of rules. Even more so to understand them as a set of rules that apply to non-humans today. In addition, rules won't be very effective constraints on superintelligences.