That's a good point and needs expanding on.
In science, we want to choose theories that are (among other things) predictive. Certainly, the preference for predicting the future - as opposed to being surprised by the future, or any number of other possible preferences - is arbitrary, in the sense that there exists minds that don't endorse it. There is no universally compelling argument that will convince every possible mind to want to predict the future correctly. But given our desire to do so, our scientific theories necessarily follow.
Math is similar: there's no UCA to use the axioms we do and not some others. But we choose our axioms to create mathematical structures that correspond to reality in some useful way (or to our thoughts, which are part of reality); and given our axioms, the rest of our mathematical theories follow.
In both cases, we choose and build our science and math due to our preexisting goals and the properties of our thought. It's those goals that are really arbitrary in the sense of no UCA; but given those basic goals and properties, science and math can be derived.
Moral realism, on the other hand, claims (AFAICS) that there are objectively true morals out there, which one ought to follow. Whether they are compatible with one's preconceived notions of morality, or goals, desires, beliefs, or anything else that is a property of the person holding moral beliefs, is irrelevant: they are true in and of themselves.
That means they should not be compared to "computability theory". They should be compared to "the desire to correctly predict whether there can exist any physical machine that would solve this problem". We can judge the objective truth of a scientific theory by how well it predicts things; but we can't judge the objective truth of a purported moral-realistic statement, because the very definition of moral realism means its truth cannot be judged. It's a kind of dualism, postulating an inherently undetectable property of "objective truth" to moral statements.
Continuing my quest to untangle people's confusions about Eliezer's metaethics... I've started to wonder if maybe some people have the intuition that the orthogonality thesis is at odds with moral realism.
I personally have a very hard time seeing why anyone would think that, perhaps in part because of my experience in philosophy of religion. Theistic apologists would love to be able to say, "moral realism, therefore a sufficiently intelligent being would also be good." It would help patch some obvious holes in their arguments and help them respond to things like Stephen Law's Evil God Challenge. But they mostly don't even try to argue that, for whatever reason.
You did see philosophers claiming things like that back in the bad old days before Kant, which raises the question of what's changed. I suspect the reason is fairly mundane, though: before Kant (roughly), it was not only dangerous to be an atheist, it was dangerous to question that the existence of God could be proven through reason (because it would get you suspected of being an atheist). It was even dangerous to advocated philosophical views that might possibly undermine the standard arguments for the existence of God. That guaranteed that philosophers could used whatever half-baked premises they wanted in constructing arguments for the existence of God, and have little fear of being contradicted.
Besides, even if you think an all-knowing would also necessarily be perfectly good, it still seems perfectly possible to have an otherwise all-knowing being with a horrible blind spot regarding morality.
On the other hand, in the comments of a post on the orthogonality thesis, Stuart Armstrong mentions that:
This is not super-enlightening, partly because Stuart is talking about people whose views he admits he doesn't understand... but on the other hand, maybe Stuart agrees that there is some kind of conflict there, since he seems to imply that he himself rejects moral realism.
I realize I'm struggling a bit to guess what people could be thinking here, but I suspect some people are thinking it, so... anyone?