If it's worth saying, but not worth its own post, then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "
It seems to me that there's no difference in kind between moral intuitions and religious beliefs, except that the former are more deeply held. (I guess that makes me a kind of error theorist.)
If that's true, that means FAI designers shouldn't work on approaches like "extrapolation" that can convert a religious person to an atheist, because the same procedure might convert you into a moral nihilist. The task of FAI designers is more subtle: devise an algorithm that, when applied to religious belief, would encode it "faithfully" as a utility function, despite the absence of God.
Does that sound right? I've never seen it spelled out as strongly, but logically it seems inevitable.
That just doesn't seem true to me. I agree that there's often difference between religious beliefs and ordinary factual beliefs, but I don't think that religious beliefs are the same sort of thing as moral intuitions. They just feel different to me.
For one thing religious beliefs are often a "belief in belief" whereas I don't think moral beliefs are like that.
Also moral beliefs seem more instinctual, whereas religious beliefs are taught.