I think its important to remember what ethics is and why humanity created it in the first place. Ethics is essentially a code of behavior using Morality as a guide to structure one's behavior to be both beneficial to yourself while not harming the greater society.
As early humans we decided that if we were going to work in a group we had to do two things: 1) Not kill each other) and 2) Behave in a manner that is beneficial.
This is essentially the Golden Rule. We had to decide that in order to use numbers and perseverance and ultimately reason to get to the top of the food chain.
Where is the machine's motivation to continue in this path? If its only desire is to propagate itself lots of things could get in the way of that, especially humanity. I think, for machines to get progressively more intelligent and for that to be beneficial we have to consider these problems and encode, at a very deep level accepted social norms into machines.
There is a lot of mainstream interest in machine ethics now. Here are some links to some popular articles on this topic.
By Zeynep Tufecki, a professor at the I School at UNC, on Facebook's algorithmic newsfeed curation and why Twitter should not implement the same.
By danah boyd, claiming that 'tech folks' are designing systems that implement an idea of fairness that comes from neoliberal ideology.
danah boyd (who spells her name with no capitalization) runs the Data & Society, a "think/do tank" that aims to study this stuff. They've recently gotten MacArthur Foundation funding for studying the ethical and political impact of intelligent systems.
A few observations:
First, there is no mention of superintelligence or recursively self-modifying anything. These scholars are interested in how, in the near future, the already comparatively powerful machines have moral and political impact on the world.
Second, these groups are quite bad at thinking in a formal or mechanically implementable way about ethics. They mainly seem to recapitulate the same tired tropes that have been resonating through academia for literally decades. On the contrary, mathematical formulation of ethical positions appears to be ya'll's specialty.
Third, however much the one-true-morality may be indeterminate or presently unknowable, progress towards implementable descriptions of various plausible moral positions could at least be incremental steps forward towards an understanding of how to achieve something better. Considering a slow take-off possible future, iterative testing and design of ethical machines with high computational power seems like low-hanging fruit that could only better inform longer-term futurist thought.
Personally, I try to do work in this area and find the lack of serious formal work in this area deeply disappointing. This post is a combination heads up and request to step up your game. It's go time.
Sebastian Benthall
PhD Candidate
UC Berkeley School of Infromation