Most philosophers talk about ‘ideal preference theories’, but I prefer to call them ‘value extrapolation algorithms’. If we want to develop Friendly AI, we may not want to just scan human values from our brains and give those same values to an AI. I want to eat salty foods all day, but I kind of wish I didn’t want that, and I certainly don’t want an AI to feed me salty foods all day. Moreover, I would probably change my desires if I knew more and was more rational. I might learn things that would change what I want. And it’s unlikely that the human species has reached the end of moral development. So we don’t want to fix things in place by programming an AI with our current values. We want an AI to extrapolate our values so that it cares about what we would want if we knew more, were more rational, were more morally developed, and so on.
What exactly is human moral development?
And it’s unlikely that the human species has reached the end of moral development.
...
...IAnissimov: What is metaethics and how is it relevant to Friendly AI theory?
Metaethics goes one level deeper. What do terms like ‘good’ and ‘right’ even mean? Do moral facts exist, or is it all relative? Is there such a thing as moral progress? These questions are relevant to friendliness content because presumably, if moral facts exist, we would want an AI to respect them. Even if moral facts do not exist, our moral attitudes are part of what we value, and that is re
Yesterday I sat down with Lukeprog for a few hours and we produced this ten-page interview for the Singularity Institute blog. This interview contains information about the Singularity Institute's technical research program and recent staff changes that hasn't been announced anywhere else! We hope you find it informative.
http://singinst.org/blog/2011/09/15/interview-with-new-singularity-institute-research-fellow-luke-muehlhuaser-september-2011/