That's a good point, but consider this contrast:
It seems clear to me that customs and agreements involve "something shared" in the sense Wei intends. As a result, it sounds straightforwardly odd to say that "This is our custom, but no one does it that way" or "We agreed on it, though he didn't know it at the time." Sentences like these demand some explanation or qualification.
On the other hand, nothing seems obviously weird about saying "You should X, though you wouldn't accept any standard that would recommend that." It may be that there are problems with using the word 'should' without reference to a shared standard (as opposed to just a standard) but it's another thing to say that this is packed into the meaning of 'should'. It sounds awfully like a theory of normativity, rather than an analysis of the meaning of a word. In fact, Wei Dai calls it a theory of normativity, though it seems to me that it must either be such a theory, or a suggestion about the meaning of a word. It can't be both.
How about, "You should X, and you should accept a standard that would recommend it?" Thereby appealing to a third (shared) standard, possibly one having to do with rationality of moral beliefs. Applying an analogous moral version of Aumann's Agreeement Theorem could lead us to a theory which suggests that you can never say this quoted sentence unless you're willing to believe that you should accept the standard you recommend.
I do hope to avoid discussion about the common usage of "should" in favor of a theory that would allow us (if n...
I find Eliezer's explanation of what "should" means to be unsatisfactory, and here's an attempt to do better. Consider the following usages of the word:
All of these seem to be sensible sentences, depending on the speaker and intended audience. #1, for example, seems a reasonable translation of what a pebblesorter would say after discovering that X = Y*Z. Some might argue for "pebblesorter::should" instead of plain "should", but it's hard to deny that we need "should" in some form to fill the blank there for a translation, and I think few people besides Eliezer would object to plain "should".
Normativity, or the idea that there's something in common about how "should" and similar words are used in different contexts, is an active area in academic philosophy. I won't try to survey the current theories, but my current thinking is that "should" usually means "better according to some shared, motivating standard or procedure of evaluation", but occasionally it can also be used to instill such a standard or procedure of evaluation in someone (such as a child) who is open to being instilled by the speaker/writer.
It seems to me that different people (including different humans) can have different motivating standards and procedures of evaluation, and apparent disagreements about "should' sentences can arise from having different standards/procedures or from disagreement about whether something is better according to a shared standard/procedure. In most areas my personal procedure of evaluation is something that might be called "doing philosophy" but many people apparently do not share this. For example a religious extremist may have been taught by their parents, teachers, or peers to follow some rigid moral code given in their holy books, and not be open to any philosophical arguments that I can offer.
Of course this isn't a fully satisfactory theory of normativity since I don't know what "philosophy" really is (and I'm not even sure it really is a thing). But it does help explain how "should" in morality might relate to "should" in other areas such as decision theory, does not require assuming that all humans ultimately share the same morality, and avoids the need for linguistic contortions such as "pebblesorter::should".