The closest thing that we have in real life to the 'rational agent' concept in game theory and artificial intelligence are psychopaths. Psychopaths act entirely out of self-interest, without any regard for others in their utility function. Taking this idea further, it's easy to see why a rational superintelligence would become a UFAI - it is a psychopath. One thing that normal humans have that psychopaths lack is empathy for others. We have some degree of 'empathizing' in our utility functions - if we make someone feel bad, we feel bad as well. Our empathy does not have laser-guided precision, and as such is directed not just at human beings but at animals (and sometimes even inanimate objects).
Thus it seems that the best way to create FAI wouldn't be Coherent Extrapolated Volition, it would be Coherent Extrapolated Emotion. This is probably a stupid question, but why does the concept of 'artificial empathy' seem to get such little attention?
The closest thing that we have in real life to the 'rational agent' concept in game theory and artificial intelligence are psychopaths.
Maybe corporations, nation-states, and other institutional actors come even closer? It sure would be nice to be able to add some "artificial empathy" to Nestlé et al.
This is a thread where people can ask questions that they would ordinarily feel embarrassed for not knowing the answer to. The previous "stupid" questions thread is at almost 500 questions in about a month, so I think it's time for a new one.
Also, I have a new "stupid" question.