I don't see anything here that is not a mixture of physical facts and logical facts (that is, truths about causal events and truths about premise-conclusion links).
Eliezer, a couple of comments ago I switched my focus from whether there is more than just physical and logical facts to whether "morality" refers to something independent of humanity, like (as I claimed) "rationality", "integer" and "P!=NP" do. Sorry if I didn't make that clear, and I hope I'm not being logically rude here, but the topic is confusing to me and I'm trying different lines of thought. (BTW, what kind of fact is it that there are only two kinds of facts?)
Quoting some background from Wikipedia:
When the Peano axioms were first proposed, Bertrand Russell and others agreed that these axioms implicitly defined what we mean by a "natural number". Henri Poincaré was more cautious, saying they only defined natural numbers if they were consistent; if there is a proof that starts from just these axioms and derives a contradiction such as 0 = 1, then the axioms are inconsistent, and don't define anything.
My question is, how can these questions even arise in our minds, unless we already had a notion of "natural number" that is independent of Peano axioms? There is something about integers that compels us to think about them, and the compelling force is not a set of axioms that is stored in our minds or spread virally from one mathematician to another.
Maybe the compelling force is that in the world that we live in, there are objects (like pebbles) whose behaviors can be approximated by the behavior of integers. I (in apparent disagreement with you) think this isn't the only compelling force (i.e., aliens who live in a world with no discrete objects would still invent integers), but it's enough to establish that when we talk about integers we're talking about something at least partly outside of ourselves.
To restate my position, I think it's unlikely that "morality" refers to anything outside of us, but many people do believe that, and I can't rule it out conclusively myself (especially given Toby Ord's recent comments).
On Wei_Dai's complexity of values post, Toby Ord writes:
The kind of moral realist positions that apply Occam's razor to moral beliefs are a lot more extreme than most philosophers in the cited survey would sign up to, methinks. One such position that I used to have some degree of belief in is:
Strong Moral Realism: All (or perhaps just almost all) beings, human, alien or AI, when given sufficient computing power and the ability to learn science and get an accurate map-territory morphism, will agree on what physical state the universe ought to be transformed into, and therefore they will assist you in transforming it into this state.
But most modern philosophers who call themselves "realists" don't mean anything nearly this strong. They mean that that there are moral "facts", for varying definitions of "fact" that typically fade away into meaninglessness on closer examination, and actually make the same empirical predictions as antirealism.
Suppose you take up Eliezer's "realist" position. Arrangements of spacetime, matter and energy can be "good" in the sense that Eliezer has a "long-list" style definition of goodness up his sleeve, one that decides even contested object-level moral questions like whether abortion should be allowed or not, and then tests any arrangement of spacetime, matter and energy and notes to what extent it fits the criteria in Eliezer's long list, and then decrees goodness or not (possibly with a scalar rather than binary value).
This kind of "moral realism" behaves, to all extents and purposes, like antirealism.
I might compare the situation to Eliezer's blegg post: it may be that moral philosophers have a mental category for "fact" that seems to be allowed to have a value even once all of the empirically grounded surrounding concepts have been fixed. These might be concepts such as "would aliens also think this thing?", "Can it be discovered by an independent agent who hasn't communicated with you?", "Do we apply Occam's razor?", etc.
Moral beliefs might work better when they have a Grand Badge Of Authority attached to them. Once all the empirically falsifiable candidates for the Grand Badge Of Authority have been falsified, the only one left is the ungrounded category marker itself, and some people like to stick this on their object level morals and call themselves "realists".
Personally, I prefer to call a spade a spade, but I don't want to get into an argument about the value of an ungrounded category marker. Suffice it to say that for any practical matter, the only parts of the map we should argue about are parts that map-onto a part of the territory.