Thanks for this! This page will keep me busy for a while. Ethics is my favorite branch of philosophy, which is my favorite hobby (having abandoned the idea of philosophizing for money); and until this page, pondering the use of ethics in the development of Friendly AI was not on my mental radar.
What is meant by that question about "should"? If it's a general inquiry, I have always considered it like so: If it is said you "should" do action Y, then action Y is thought to cause some outcome X, and this outcome X is thought to be desirable.
In common usage, there's weak 'should' and strong 'should'.
Weak should is simply a suggestion.
A: "What should I do?" (Please suggest a course of action.)
B: "You should do X." (Have you considered X?)
Strong should argues for a new course of action instead of the one the listener originally proposed. It comes with the subtext that the listener may initially disagree with the suggested course of action, but suggests that the listener re-evaluate their disagreement in light of the speaker's conviction.
A: "I want to steal the cookies...
I've been working on metaethics/CEV research for a couple months now (publishing mostly prerequisite material) and figured I'd share some of the sources I've been using.
CEV sources.
Motivation. CEV extrapolates human motivations/desires/values/volition. As such, it will help to understand how human motivation works.
Extrapolation. Is it plausible to think that some kind of extrapolation of human motivations will converge on a single motivational set? How would extrapolation work, exactly?
Metaethics. Should we use CEV, or something else? What does 'should' mean?
Building the utility function. How can a seed AI be built? How can it read what to value?
Preserving the utility function. How can the motivations we put into a superintelligence be preserved over time and self-modifcation?
Reflective decision theory. Current decision theories tell us little about software agents that make decisions to modify their own decision-making mechanisms.
Additional suggestions welcome. I'll try to keep this page up-to-date.