Even though I come from a somewhat different viewpoint, I was fairly impressed with the case you are presenting. Nonetheless . ..

1. Creating a powerful AI system without understanding why it works is dangerous.

A large portion of the risk from machine superintelligence comes from the possibility of people building systems that they do not fully understand. Currently, this is commonplace in practice: many modern AI researchers are pushing the capabilities of deep neural networks in the absence of theoretical foundations that describe why they’re working so well or a solid idea of what goes on beneath the hood.

This is not quite fully accurate. Yes anybody can download a powerful optimizer and use it to train a network that they don't understand. But those are not the people you need to worry about, that is not where the danger lies.

The concern that ML has no solid theoretical foundations reflects the old computer science worldview, which is all based on finding bit exact solutions to problems within vague asymptotic resource constraints.

Old computer science gave us things like convex optimization, which is nothing interesting at all (it only works well for simple uninteresting problems). Modern AI/ML is much more like computer graphics or simulation, where everything is always an approximation and traditional computer science techniques are mostly useless. There is no 'right answer', there are just an endless sea of approximations that have varying utility/cost tradeoffs.

A good ML researcher absolutely needs a good idea of what is going on under the hood - at least at a sufficient level of abstraction. The optimization engine does most of the nitty gritty work - but it is equivalent to the researcher employing an army of engineers and dividing the responsibility up so that each engineer works on a tiny portion of the circuit. To manage the optimizer, the researcher needs a good high level understanding of the process, although not necessarily the details.

Also - we do have some theoretical foundations for DL - bayesian inference for one. Using gradient descent on the joint log PDF is a powerful approximate inference strategy.

When designing a superintelligent agent, by contrast, we will want an unusually high level of confidence in its safety before we begin online testing: trial and error alone won’t cut it, in that domain.

It appears you are making the problem unnecessarily difficult.

Why not test safety long before the system is superintelligent? - say when it is a population of 100 child like AGIs. As the population grows larger and more intelligent, the safest designs are propagated and made safer.

1. This is an approach researchers have used successfully in the past.

Our main open-problem generator — “what would we be unable to solve even if the problem were easier?” — is actually a fairly common one used across mathematics and computer science. It’s more easy to recognize if we rephrase it slightly: “can we reduce the problem of building a beneficial AI to some other, simpler problem?”

This again reflects the old 'hard' computer science worldview, and obsession with exact solutions.

If it seems really really really impossibly hard to solve a problem even with the 'simplification' of lots of computing power, perhaps the underlying assumptions are wrong. For example - perhaps using lots and lots of computing power makes the problem harder instead of easier.

How could that be? Because with lots and lots of compute power, you are naturally trying to extrapolate the world model far far into the future, where it branches enormously and grows in complexity exponentially. Then when you try to define a reasonable utility/value function over the future world model, it becomes almost impossible because the future world model has exploded exponentially in complexity.

So it may actually be easier to drop the traditional computer science approach completely. Start with a smaller more limited model that doesnt explode, and then approximately extrapolate both the world model and the utility/value function together.

This must be possible in principle, because human children learn that way. Realistically there isn't room in the DNA for a full adult utility/value function, and it wouldn't work in an infant brain anyway without the world model. But evolution solved this problem approximately, and we can learn from it and make do.