Continuing my quest to untangle people's confusions about Eliezer's metaethics... I've started to wonder if maybe some people have the intuition that the orthogonality thesis is at odds with moral realism.
I personally have a very hard time seeing why anyone would think that, perhaps in part because of my experience in philosophy of religion. Theistic apologists would love to be able to say, "moral realism, therefore a sufficiently intelligent being would also be good." It would help patch some obvious holes in their arguments and help them respond to things like Stephen Law's Evil God Challenge. But they mostly don't even try to argue that, for whatever reason.
You did see philosophers claiming things like that back in the bad old days before Kant, which raises the question of what's changed. I suspect the reason is fairly mundane, though: before Kant (roughly), it was not only dangerous to be an atheist, it was dangerous to question that the existence of God could be proven through reason (because it would get you suspected of being an atheist). It was even dangerous to advocated philosophical views that might possibly undermine the standard arguments for the existence of God. That guaranteed that philosophers could used whatever half-baked premises they wanted in constructing arguments for the existence of God, and have little fear of being contradicted.
Besides, even if you think an all-knowing would also necessarily be perfectly good, it still seems perfectly possible to have an otherwise all-knowing being with a horrible blind spot regarding morality.
On the other hand, in the comments of a post on the orthogonality thesis, Stuart Armstrong mentions that:
I've read the various papers [by people who reject the orthogonality thesis], and they all orbit around an implicit and often unstated moral realism. I've also debated philosophers on this, and the same issue rears its head - I can counter their arguments, but their opinions don't shift. There is an implicit moral realism that does not make any sense to me, and the more I analyse it, the less sense it makes, and the less convincing it becomes. Every time a philosopher has encouraged me to read a particular work, it's made me find their moral realism less likely, because the arguments are always weak.
This is not super-enlightening, partly because Stuart is talking about people whose views he admits he doesn't understand... but on the other hand, maybe Stuart agrees that there is some kind of conflict there, since he seems to imply that he himself rejects moral realism.
I realize I'm struggling a bit to guess what people could be thinking here, but I suspect some people are thinking it, so... anyone?
I would say that the orthogonality thesis does not necessarily imply moral non-realism... but some forms of moral non-realism do imply the orthogonality thesis, in which case rejecting the orthogonality thesis would require rejecting at least that particular kind of moral non-realism. This may cause moral non-realists of that variety to equate moral realism and a rejection of the OT.
For example, if you are a moral non-cognitivist, then according to the SEP, you believe that:
This would seem to imply the orthogonality thesis: different agents will have different desires and goals, and if goals have no inherent truth value and moral statements simply reflect our desires and goals, then there is no particular reason to expect agents with a higher intelligence to converge on the same goals/moral beliefs. They'll just keep their original desires/goals, since no amount of increased intelligence could reveal facts which would cause those desires/goals to change (with the possible exception of cases where increased intelligence reveals a goal to have been logically incoherent).
Non-cognitivism strictly speaking doesn't imply the orthogonality thesis. For instance, one could consistently hold that increased intelligence leads to a convergence of the relevant non-cognitive attitudes. Admittedly, such a position appears implausible, which might explain the fact (if it is a fact) that non-cognitivists are more prone to accept the orthogonality thesis.