Dumb agent could also cause human extinction. "To kill all humans" is computationly simpler task than to create superintelligence. And it may be simplier by many orders of magnitude.

Okay, so we just have to determine human terminal values in detail, and plug them into a powerful maximizer.

No - not at all. Perhaps you have read too much MIRI material, and not enough of the neuroscience and machine learning I referenced. An infant is not born with human 'terminal values'. It is born with some minimal initial reward learning circuitry to bootstrap learning of complex values from adults.

Stop thinking of AGI as some wierd mathy program. Instead think of brain emulations - and then you have obvious answers to all of these questions.

Saying the phrase "safe sandbox sim" is much easier than making a virtual machine that can withstand a superhuman intelligence trying to get out of it.

You apparently didn't read my article or links to earlier discussion? We can easily limit the capability of minds by controlling knowledge. A million smart evil humans is dangerous - but only if they have modern knowledge. If they have only say medieval knowledge, they are hardly dangerous. Also - they don't realize they are in a sim. Also - the point of the sandbox sims is to test architectures, reward learning systems, and most importantly - altruism. Designs that work well in these safe sims are then copied into less safe sims and finally the real world.

Consider the orthogonality thesis - AI of any intelligence level can be combined with any values. Thus we can test values on young/limited AI before scaling up their power.

Sandbox sims can be arbitrarily safe. It is the only truly practical workable proposal to date. It is also the closest to what is already used in industry. Thus it is the solution by default.

Even if your software is perfect, it can still figure out that its world is artificial and figure out ways of blackmailing its captors

Ridiculous nonsense. Many humans today are aware of the sim argument. The gnostics were aware in some sense 2,000 years ago. Do you think any of them broke out? Are you trying to break out? How?

If it's maximizing its own utility, which is necessary if you want it to behave anything like a child, what's to stop it from learning human greed and cruelty, and becoming an eternal tyrant?

Again, stop thinking we create a single AI program and then we are done. It will be a largescale evolutionary process, with endless selection, testing, and refinement. We can select for super altruistic moral beings - like bhudda/gandhi/jesus level. We can take the human capability for altruism, refine it, and expand on it vastly.

For starters, you want to be able to prove formally that its goals will remain stable as it self-modifies,

Quixotic waste of time.

The point is that it's ridiculous to say that human beings are 'universal learning machines'

No - it is not. See the article for the in depth argument and citations backing up this statement.

you can just raise any learning algorithm as a human child and it'll turn out fine.

Well almost - A ULM also requires a utility function or reward circuitry with some initial complexity, but we can also use the same universal learning algorithms to learn that component. It is just another circuit, and we can learn any circuit that evolution learned.

And that's all it takes to make them consistently UnFriendly, regardless of how well they're raised.

Sure - which is why I discussed sim sandbox testing. Did you read about my sim sandbox idea? We test designs in a safe sandbox sim, and we don't copy sociopaths.

Obviously, AIs are going to be more different from us than that

No, this isn't obvious at all. AGI is going to be built from the same principles as the brain - because the brain is a universal learning machine. The AGI's mind structure will be learned from training and experiential data such that the AI learns how to think like humans and learns how to be human - just like humans do. Human minds are software constructs - without that software we would just be animals (feral humans). An artificial brain is just another computer that can run the human mind software.

That hard coding is not going to be present in an arbitrary AI, which means we have to go and duplicate it out of a human brain. Which is HARD.

Yes, but it's only a part of the brain and a fraction of the brain's complexity, so obviously it can't be harder than reverse engineering the whole brain.

Here's one from a friend of mine. It's not exactly an argument against AI risk, but it is an argument that the problem may be less urgent than it's traditionally presented.

1. There's plenty of reason to believe that Moore's Law will slow down in the near future

2. Progress on AI algorithms has historically been rather slow.

3. AI programming is an extremely high level cognitive task, and will likely be among the hardest things to get an AI to do.

4. These three things together suggest that there will be a 'grace period' between the development of general agents, and the creation of a FOOM-capable AI.

5. Our best guess for the duration of this grace period is on the order of multiple decades.

6. During this time, general-but-dumb agents will be widely used for economic purposes.

7. These agents will have exactly the same perverse instantiation problems as a FOOM-capable AI, but on a much smaller scale. When they start trying to turn people into paperclips, the fallout will be limited by their intelligence.

8. This will ensure that the problem is taken seriously, and these dumb agents will make it much easier to solve FAI-related problems, by giving us an actual test bed for our ideas where they can't go too badly wrong.

This is a plausible-but-not-guaranteed scenario for the future, which feels much less grim than the standard AI-risk narrative. You might be able to extend it into something more robust.

http://kruel.co/2012/07/17/ai-risk-critiques-index/

Kruel's critique sounded very convincing when I first read it.

Super obvious re-rebut: sociopaths exist, and yet civilization endures.

Also, we can rather obviously test in safe simulation sandboxes and avoid copying sociopaths. The argument that sociopaths are a fundemental showstopper must be based then on some magical view of the brain (because obviously evolution succeeds in producing non sociopaths, so we can copy its techniques if they are nonmagical).

Remember the argument is against existential threat level UFAI, not some fraction of evil AIs in a large population.

Here is a novel argument you may or may not have heard: We live in the best of all probable worlds due to simulation anthropics. Future FAI civs spend a significant amount of their resources to resimulate and resurrect past humanity - winning the sim race by a landslide (as UFAI is not strongly motivated to sim us in large numbers). As a result of this anthropic selection force, we find ourselves in a universe that is very lucky - it is far more likely to lead to FAI than you would otherwise think.

The best standard argument is this: the brain is a universal learning machine - the same general architecture that will necessarily form the basis for any practical AGI. In addition the brain is already near optimal in terms of what can be done for 10 watts with any irreversible learning machine (this is relatively easy to show from wiring energy analysis). Thus any practical AGI is going to be roughly brain like, similar to baby emulations. All of the techniques used to raise humans safely can thus be used to raise AGI safely. LW/MIRI historically reject this argument based - as far as I can tell - on a handwavey notion of 'anthropomorphic bias', which has no technical foundation.

I've presented the above argument about four years ago, but I never bothered to spend the time backing it up in excruciating formal detail. Until more recently. The last 5 years of progress in AI strongly supports this anthropomorphic AGI viewpoint.

Sure. Basically, there are two groups, each of which has made a major contribution:

1) Shawn Mikula and his group. They have made substantial progress (some would say, almost solved) of how to make the neuronal connections and other brain structures such as white matter tracts in a full mouse brain traceable using electron microscopy. Electron microscopy is the lowest level of imaging currently feasible, and can clearly resolve structures that are thought to be key to memory such as synapses.

2) The 21CM group, including Robert McIntyre. They have developed a totally new method of preserving a brain that should yield both highly practical and technical sound preservation. In a sense it combines the methods discussed by Gwern in his article Plastination vs Cryonics, because it first uses a method traditionally associated with "plastination" (glutaraldehyde perfusion), and then uses a method traditionally associated with cryonics, i.e. perfusion with a cryoprotective agent and then low temperature storage and, presumably, vitrification, which means that damage from ice crystal formation should be avoided and the brain should turn a glass state.

Apologies if this is still too technical and I'm happy to answer any follow-up questions. Many key steps remain but this is progress worthy of celebrating and, in my view, supporting.

Here are a few.

The press typically describes DeepMind as "highly secretive," but actually they publish a ton of their research — including this paper — in all the usual venues: NIPS, arxiv, etc.