Below is a sketch of a moral anti-realist position based on the map-territory distinction, Hume and studies of psychopaths. Hopefully it is productive.

The Map is Not the Territory Reviewed

Consider the founding metaphor of Less Wrong: the map-territory distinction. Beliefs are to reality as maps are to territory. As the wiki says:

Since our predictions don't always come true, we need different words to describe the thingy that generates our predictions and the thingy that generates our experimental results. The first thingy is called "belief", the second thingy "reality".

Of course the map is not the territory.

Here is Albert Einstein making much the same analogy:

Physical concepts are free creations of the human mind and are not, however it may seem, uniquely determined by the external world. In our endeavor to understand reality we are somewhat like a man trying to understand the mechanism of a closed watch. He sees the face and the moving hands, even hears its ticking, but he has no way of opening the case. If he is ingenious he may form some picture of a mechanism which could be responsible for all the things he observes, but he may never be quite sure his picture is the only one which could explain his observations. He will never be able to compare his picture with the real mechanism and cannot even imagine the possibility or the meaning of such a comparison. But he certainly believes that, as his knowledge increases, his picture of reality will become simpler and simpler and will explain a wider and wider range of his sensuous impressions. He may also believe in the existence of the ideal limit of knowledge and that it is approached by the human mind. He may call this ideal limit the objective truth.

The above notions about beliefs involve pictorial analogs, but we can also imagine other ways the same information could be contained. If the ideal map is turned into a series of sentences we can define a 'fact' as any sentence in the ideal map (IM). The moral realist position can then be stated as follows:

Moral Realism: ∃x(x ⊂ IM) & (x = M)

In English: there is some set of sentences x such that all the sentences are part of the ideal map and x provides a complete account of morality.

Moral anti-realism simply negates the above.  ¬(∃x(x ⊂ IM) & (x = M)).

Followup To:  Are You Anosognosic?, The Strangest Thing An AI Could Tell You

Over this past weekend I listened to an episode of This American Life titled Pro Se.  Although the episode is nominally about people defending themselves in court, the first act of the episode was about a man who pretended to act insane in order to get out of a prison sentence for an assault charge.  There doesn't appear to be a transcript, so I'll summarize here first.

A man, we'll call him John, was arrested in the late 1990s for assaulting a homeless man.  Given that there was plenty of evidence to prove him guilty, he was looking for a way to avoid the likely jail sentence of five to seven years.  The other prisoners he was being held with suggested that he plead insanity:  he'd be put up at a hospital for several months with hot food and TV and released once they considered him "rehabilitated".  So he took bits and pieces about how insane people are supposed to act from movies he had seen and used them to form a case for his own insanity.  The court believed him, but rather than sending him to a cushy hospital, they sent him to a maximum security asylum for the criminally insane.

Within a day of arriving, John realized the mistake he had made and sought to find a way out.  He tries a variety of techniques:  engaging in therapy, not engaging in therapy, dressing like a sane person, acting like a sane person, acting like an incurably insane person, but none of it works.  Over a decade later he is still being held.

As the story unravels, we learn that although John makes a convincing case that he faked his way in and is being held unjustly, the psychiatrists at the asylum know that he faked his way in and continue to hold him anyway, though John is not aware of this.  The reason:  through his long years of documented behavior John has made it clear to the psychiatrists that he is a psychopath/sociopath and is not safe to return to society without therapy.  John is aware that this is his diagnosis, but continues to believe himself sane.

Similar to trying to determine if you are anosognosic, how do you determine if you are insane?  Some kinds of insanity can be self diagnosed, but in John's case he has lots of evidence (he has access to read all of his own medical records) that he is insane, yet continues to believe himself not to be.  To me this seems a level trickier than anosognosis, since there's no physical tests you can make, but perhaps it's only a level of difference significant to people but not to an AI.

Edited to add a footnote:  By "sane" I simply mean normative human reasoning:  the way you expect, all else being equal, a human to think about things.  While the discussion in the comments about how to define sanity might be of some interest, it really gets away from the point of the post unless you want to argue that "sanity" is creating a question here that is best solved by dissolving the question (as at least one commenter does).