Looks interesting. Thanks for doing this. It would be useful to me to get links to some of the things you mention (like eric drexler's work, I am not familiar with).
I think there might be some categories missed here, it only presents the AI building problems. There are further classes of problem such as social problems. For example the problem of all agreeing not to develop AI until the other problems are solved.
There are also questions around whether we can make effective agents at all. We have existence proof on effective non-agent intelligences, but none around effective agent intelligences.
We invented the model of rational agents for humans, then we had to add lots and lots of exceptions (heuristics and biases). So much that you have disavowed it in this post. Yet we keep with the same model for AIs. Perhaps we need a kuhnian paradigm shift in our understanding of agents.
Crossposted at the Intelligent agent foundation.
There have been various attempts to classify the problems in AI safety research. Our old Oracle paper that classified then-theoretical methods of control, to more recent classifications that grow out of modern more concrete problems.
These all serve their purpose, but I think a more enlightening classification of the AI safety problems is to look at what the issues we are trying to solve or avoid. And most of these issues are problems about humans.
Specifically, I feel AI safety issues can be classified as three human problems and one central AI issue. The human problems are:
And the central AI issue is:
Obviously if humans were agents and knew their own values and could predict whether a given AI would follow those values or not, there would be not problem. Conversely, if AIs were weak, then the human failings wouldn't matter so much.
The points about human values is relatively straightforward, but what's the problem with humans not being agents? Essentially, humans can be threatened, tricked, seduced, exhausted, drugged, modified, and so on, in order to act seemingly against our interests and values.
If humans were clearly defined agents, then what counts as a trick or a modification would be easy to define and exclude. But since this is not the case, we're reduced to trying to figure out the extent to which something like a heroin injection is a valid way to influence human preferences. This makes both humans susceptible to manipulation, and human values hard to define.
Finally, the issue of humans having poor predictions of AI is more general than it seems. If you want to ensure that an AI has the same behaviour in the testing and training environment, then you're essentially trying to guarantee that you can predict that the testing environment behaviour will be the same as the (presumably safe) training environment behaviour.
How to classify methods and problems
That's well and good, but how to various traditional AI methods or problems fit into this framework? This should give us an idea as to whether the framework is useful.
It seems to me that:
Putting this all in a table:
Further refinements of the framework
It seems to me that the third category - poor predictions - is the most likely to be expandable. For the moment, it just incorporates all our lack of understanding about how AIs would behave, but this might more useful to subdivide.