I don't see how this is any different for folk morality than for folk physics, folk medicine, folk sociology, or any other aspect of human psychology. For my own part, I find that formalizing my intuitions (moral and otherwise) is a useful step towards identifying the biases that those intuitions introduce into my thinking.
Oh yeah. My point - if I have a point, which I may or may not do - is that you can't do it on the level of the morality itself and get good results, as that's all cached derived resuits; you have to go to metamorality, i.e. game theory (at least), not to risk going over the edge into silliness. It's possible this says nothing and adds up to normality, which is the "may not do" bit.
I'm currently reading back through abstruse game theory posts on LessWrong and particularly this truly marvellous book and realising just how damn useful this stuff is going to be in real life.
the dynamic whereby a structure that remains covert is thereby protected from attack and can operate without accountability. Moral intuitions are frequently used this way.
Oh!
(blink)
That's actually a very good point. I endorse having it, should you ever do.
The trolley problem
In 2009, a pair of computer scientists published a paper enabling computers to behave like humans on the trolley problem (PDF here). They developed a logic that a computer could use to justify not pushing one person onto the tracks in order to save five other people. They described this feat as showing "how moral decisions can be drawn computationally by using prospective logic programs."
I would describe it as devoting a lot of time and effort to cripple a reasoning system by encoding human irrationality into its logic.
Which view is correct?
Dust specks
Eliezer argued that we should prefer 1 person being tortured for 50 years over 3^^^3 people each once getting a barely-noticeable dust speck in their eyes. Most people choose the many dust specks over the torture. Some people argued that "human values" includes having a utility aggregation function that rounds tiny (absolute value) utilities to zero, thus giving the "dust specks" answer. No, Eliezer said; this was an error in human reasoning. Is it an error, or a value?
Sex vs. punishment
In Crime and punishment, I argued that people want to punish criminals, even if there is a painless, less-costly way to prevent crime. This means that people value punishing criminals. This value may have evolved to accomplish the social goal of reducing crime. Most readers agreed that, since we can deduce this underlying reason, and accomplish it more effectively through reasoning, preferring to punish criminals is an error in judgement.
Most people want to have sex. This value evolved to accomplish the goal of reproducing. Since we can deduce this underlying reason, and accomplish it more efficiently than by going out to bars every evening for ten years, is this desire for sex an error in judgement that we should erase?
The problem for Friendly AI
Until you come up with a procedure for determining, in general, when something is a value and when it is an error, there is no point in trying to design artificial intelligences that encode human "values".
(P.S. - I think that necessary, but not sufficient, preconditions for developing such a procedure, are to agree that only utilitarian ethics are valid, and to agree on an aggregation function.)