All of someperson's Comments + Replies

I'm uncomfortable with patching my moral intuition with math. Wouldn't that imply that we should be willing to use violence to shut down promising AGI labs who don't take friendliness concerns into consideration?

If your moral system leads you to do things that make your moral intuition queasy, you should question your moral system.

Biting bullets is an overly simple solution to moral dilemma. You find yourself making monsters without much effort.

1Mass_Driver
Mmm, depends whether you are using "question" as a euphemism for "reject." Certainly, you should re-examine your explicit reasoning about ethics if the conclusions you reach conflict with many of your moral intuitions. However, you should also re-examine your moral intuitions when they fail to agree with your explicit reasoning about ethics. Otherwise, there would be very little point in conducting ethical analysis -- if your analysis can't ever validly prompt you to discard or ignore a moral intuition, then you may as well stop searching your conscience and just do whatever 'feels right' at any given moment. Sometimes your intuitions give way, and sometimes your formal reasoning gives way -- that's how you reach reflective equilibrium. Ah, but is "don't make monsters" your most important moral objective? Suppose you had to become a monster in the eyes of your friends in order to save a village full of innocent children. Is it obvious that it would be wrong to become a monster in this sense?