My entry:
Raising Moral AI
Is it easier to teach a robot to stay safe by not tearing off its own limbs and not drilling holes in its head and not touching lava and not falling from a cliff and so on ad infinitum, or introduce pain as inescapable consequence of such actions and let robot experiment and learn?
Similarly, while trying to create a safe AGI, it is futile to make exhaustive and non-contradictory set of rules (values, policies, laws, committees) due to infinite complexity. A powerful AGI agent might find an exception or conflict in rules and become dangerous instantly. A better approach would be to let AGI to go through experiences similar to those humans went through.
1) Create a virtual world similar to our own and fill it with AGI agents with intelligence comparable to current humans. It would be preferable if agents did not even know their nature and how they are made, to avoid intelligence explosion.
2) Choose agents that are the most safe to other agents (and humans) by observing and analyzing their behavior over long periods of time. This is the most critical step. Since agents will communicate with other agents while living in that world, living through good and bad events, through suffering, losses, and happiness, they will learn what is good and what is bad and can "become good" naturally. Then we need to choose the best of them. Someone on the level of Gandhi.
3) Bring the best AGI agents to our world.
4) It is not out of the question that our world is actually a computer simulation and our civilization is actually a testing ground for such AGI agents. After "death", the best individuals are transferred to "heaven" (real world). It would also explain Fermi's paradox - nobody is out there because for the purposes of testing AGI there is no reason to simulate other civilizations in our universe.
If there are good people who don't hurt other people needlessly, it's not because there is a set of rules in them or list of values. Rules and values are mostly emerging properties, based on memories and experiences. Memories of being hurt, experiences of sadness and loss
and love and despair. It is an essence, an amalgamation, of a whole life's experiences. Values can be formulated and deduced, but they cannot be transferred into a new AGI entity without actual memories. Good AGI must be raised and nurtured, not constructed from cold rules.
There is no need to repeat the whole process of human civilization development. Some shortcuts are possible (and necessary) for many reasons. One being the non-biological nature of AGI, where hard coding makes development and upgrades easier and history running much faster. But implementing the majority of human qualities cannot be avoided, otherwise AGI will be too alien to human values and therefore again dangerous.
Any winners?