I wanted to answer as OP, but I don't have a particularly informed view on this. And my view is biased by having spent a lot of time around moral philosophers who are hugely, hugely disproportionately likely to fall into the "moral reasoning is real" camp — and by having evolved from conservative Evangelical ethics to classical utilitarianism!
One fairly robust observation in psychology is that people do feel pressure to engage in moral consistency reasoning. One example of this comes from the literature on the Meat Eating Paradox. Let me reproduce an overview from pp. 573-574 here:
... (read 430 more words →)In a series of five studies, Brock Bastian and colleagues have demonstrated a link between seeing
Epistemic status: shower thought quickly sketched, but I do have a PhD in this.
As we approach AGI and need to figure out what goals to give it we will need to find tractable ways to resolve moral disagreement. One of the most intractable moral disagreements is between the moral realists and the moral antirealists.
There's an oversimplified view of this disagreement that goes:
Just wanted to say thanks so much to the fact checkers here! It's nice that I was able to update my half-baked thread with your more researched takes. Didn't expect my shower thought to blow up quite so much!
I feel sad about this too. But this is common in impure scientific disciplines, e.g. medical studies often refer to value-laden concepts like proper functioning. The ideal would be to gradually naturalize all of this so we can talk to each other about observables without making any assumptions about interpretation of open-textured terminology. What I want to show here is primarily an existence proof that we can fully naturalize this discussion, but I haven't yet managed to do this.
I think this is a very good question about arguments. And I do think we will have to make value judgments about what kinds of moral deliberation processes we think are "good" otherwise we... (read more)