Isn't the real danger that you will formalize the goal function wrong, not that the deductions will be invalid?

Both are huge difficulties, but most of the work in FAI is probably in the AI part, not the F part.

This is all nifty and interesting, as mathematics, but I feel like you are probably barking up the wrong tree when it comes to applying this stuff to AI. I say this for a couple of reasons:

First, ZFC itself is already comically overpowered. Have you read about reverse mathematics? Stephen Simpson edited a good book on the topic. Anyway, my point is that there's a whole spectrum of systems a lot weaker than ZFC that are sufficient for a large fraction of theorems, and probably all the reasoning that you would ever need to do physics or make real word, actionable decisions. The idea that physics could depend on reasoning of a higher consistency strength than ZFC just feels really wrong to me. Like the idea that P could really equal NP. Of course my gut feeling isn't evidence, but I'm interested in the question of why we disagree. Why do you think these considerations are likely to be important?

Second, Isn't the the whole topic of formal reasoning a bike shed? Isn't the real danger that you will formalize the goal function wrong, not that the deductions will be invalid?

I don't think you've chosen your examples particularly well.

Abortion certainly can be a 'central' case of murder. Immagine aborting a fetus 10 minutes prior to when it would have been born. It can also be totally 'noncentral': the morning after pill. Abortions are a grey area of central-murder depending on the progress of neural devlopment of the fetus.

Affermative action really IS a central case of racism. It's bad for the same reason as segregation was bad, because it's not fair to judge people based on their race. The only difference is that it's not nearly AS bad. Segregation was brutal and oppressive, while affermative action doesn't really affect most peopel enough for them to notice.

"[N]ot to know things that we do in fact know," and "Confidently inform them when we know they're wrong." Except, as a rationalist, you can't say that you know there is no god. You may be able to say that you believe it to be unlikely that there is a god, or that you have seen no evidence that would make you believe that there is a god. The fact is that it is (near) impossible to prove a negative. Likewise, you cannot say that you know there is are no purple polar bear, fairies, unicorns or black swans. The burden of proof does always fall to the affirmative, but you can't rationally and conclusively prove the negative.

"Trust your intuitions, but don't waste too much time arguing for them"

This is an excellent point. Intuition plays an absolutely crucial point in human thought, but there's no point in debating an opinion that (by definition, even) you're incapable of verbalizing your reasons for. Let me suggest another maxim:

Intuitions tell you where to look, not what you'll find.

From reading histories of him. He took classes on how to yell at crowds; studied it in great detail.

What makes you think Hitler didn't deliberately think about how to yell at crowds?

You're confusing "reason" with inappropriate confidence in models and formalism.

I vote for the meta-thread convention, or for any other mechanism that keeps meta off the front page.

I think the main problem with mormon2's submission was not where it was posted, but that it was pointless and uninformed.