= 27d567bc2d48d9f40e9ccc20ea370b15)
bentarm comments on The Friendly AI Game - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (170)
Oracle AI - its only desire is to provide the correct answer to yes or no questions posed to it in some formal language (sort of an ueber Watson).
Oops. The local universe just got turned into computronium. It is really good at answering questions though. Apart from that you gave it a desire to provide answers. The way to ensure that it can answer questions is to alter humans such that they ask (preferably easy) questions as fast as possible.
Comment upvoted for starting the game off! Thanks!
Q: Is the answer to the Ultimate Question of Life, the Universe, and Everything 42?
A: Tricky. I'll have to turn the solar system into computronium to answer it. Back to you as soon as that's done.
Yes, this was the first nightmare scenario that occurred to me. Interesting that there are so many others...
Some villain then asks how to reliably destroy the world, and follows the given answer.
Alternatively: A philosopher asks for the meaning of life, and the Oracle returns an extremely persuasive answer which convinces most of people that life is worthless.
Another alternative: After years of excellent work, the Oracle gains so much trust that people finally start to implement a possibility to ask less formal questions, like "how to maximise human utility", and then follow the given advice. Unfortunately (but not surprisingly), unnoticed mistake in the definition of human utility has slipped through the safety checks.
Yes, that's the main difficulty behind friendly AI in general. This does not constitute a specific way that it could go wrong.
Oh, sure. My only intention was to show that limiting the AI's power to mere communication doesn't imply safety. There may be thousands of specific ways how it could go wrong. For instance:
The Oracle answers that human utility is maximised by wireheading everybody to become a happiness automaton, and that it is a moral duty to do that to others even against their will. Most people believe the Oracle (because its previous answers always proved true and useful, and moreover it makes a really neat PowerPoint presentations of its arguments) and wireheading becomes compulsory. After the minority of dissidents are defeated, all mankind turns into happiness automata and happily dies out a while later.
The 1946 short story "A Logic Named Joe" describes exactly that scenario, gone horribly wrong.
Would take overt or covert dictatorial control of humanity and reshape their culture so that (a) breeding to the brink of starving is a mass moral imperative and (b) asking very simple questions to the Oracle five times a day is a deeply ingrained quasi-religious practice.
Out of curiosity, how many people here are total utilitarians who would welcome this development?
This sounds like it would stabilize 'fun' at a comparatively low level with regard to all possibilities, so I don't think that an imaginative utilitarian would like it.
Anders Sandberg wrote fiction (well, an adventure within the Eclipse Phase RPG) about this:
http://www.aleph.se/EclipsePhase/ThinkBeforeAsking.pdf