Thanks for the clear explanation of your views. What do you see as the main obstacles to achieving this?
Martin Luther King and Ghandi are examples of highly altruistic humans
I'm really worried that mere altruism isn't enough. If the other agent is more powerful, any subtle differences in values or philosophical views between myself and the other agent could be disastrous, as they optimize the universe according to their values/views which may turn out to be highly suboptimal for me. Consider the difference between average and total utilitarianism, or different views on whether we should assume the universe must be computable, what prior/measure to put on the multiverse, or how to deal with anthropics, e.g. simulation argument.
But I don't want them to blindly accept my current values/views either, since they may be wrong. Humans seem to have some sort of general problem solving / error correcting algorithm which we call "doing philosophy", and maybe we can teach that to ANN-based AI more easily than we could program it by hand, so in that sense maybe ANN-based AI actually could be less "nasty" than other approaches.
To me, achieving a near optimal outcome is difficult but not impossible, given enough time, but I don't see how to get the time. The current leaders in ANN-based AI don't seem to appreciate the magnitude of the threat, or the difficulty of solving the problem. (Besides Schmidhuber, who apparently does see the threat but is ok with it? Now that Bostrom's book has been out for a year and presumably most people who are ever going to read it has already read it, I'm not sure what's going to change their minds.) Perhaps ANN-based AI could be considered more "nasty" in this sense because it seems easier to be complacent about it, thinking that when the time comes, we'll just teach them our values, whereas trying to design a de novo AGI brings up a bunch of issues like exactly what utility function to give it, or what decision theory or prior, that perhaps makes it easier to see the larger problem.
(The other main obstacle I see is the strong economic and psychological incentives to achieve AGI ASAP, but that's the same whether we're talking about ANN-based AI or other kinds of AI.)
Thanks for the clear explanation of your views. What do you see as the main obstacles to achieving this?
My optimistic scenario above assumes not only that we solve the technical problems but also that the current political infrastructure doesn't get in the way - and in fact just allows itself to be dissolved.
In reality of course I dont think it will be that simple.
There are technical problems like value learning, and then there are socio-political problems. AGI is likely to cause systemic unemployment and thus a large recession which will force politic...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.