V_V comments on How to Study Unsafe AGI's safely (and why we might have no choice) - Less Wrong

10 Post author: Punoxysm 07 March 2014 07:24AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (47)

You are viewing a single comment's thread.

Comment author: V_V 07 March 2014 04:25:45PM *  4 points [-]

Very good post!

I agree that experimentation with near-human level AI (assuming that it is possible) is unlikely to have catastrophic consequences as long as standard safety engineering practices are applied.
And in fact, experimentation is likely the only way to make any real progress in understanding the critical issues of AI safety and solving them.

In engineering, "Provably safe", "provably secure" designs typically aren't, especially when dealing with novel technologies: once you build a physical system, there is always some aspect that wasn't properly addressed by the theoretical model but turns out to make your system fail in unanticipated ways.
Careful experimentation is needed to gain knowledge of the critical issues of a design in a controlled environment, and once the design has been perfected, you still can't blindly trust it, rather you need to apply extensive redundancy and impact mitigation measures.
That's how we have made productive use of potentially dangerous stuff such as fire, electricity, cars, trains, aeroplanes, nuclear power and microorganisms, without wiping ourselves out so far. I don't think that AI should or could be an exception.
Gambling our future on a brittle mathematical proof, now that would be foolish, in my humble opinion.

Much of the current discussion about AI safety suggests me an analogy with some hypothetical eighteen century people trying to discuss air traffic safety:
They could certainly imagine flying machines, they could understand that these machines would have to work according to Newtonian mechanics and known fluid dynamics, and they could probably foresee some of the inherent dangers of operating such machines. But obviously, their discussions wouldn't produce any significant result, because they would lack knowledge key facts about the architecture of actually workable designs. They wouldn't know anything about internal combustion engines, aluminium, radio communication, radars, and so on.
Present-day self-proclaimed "AI risk experts" look much like those hypothetical eighteen century "aviation risk experts": they have little or no idea of an actual AI design is going to look like, and yet they attempt to argue from first principles (moral philosophy, economic theories and mathematical logic) about its safety.
It goes without saying that I don't have much confidence in their approach.

Comment author: Squark 08 March 2014 07:31:57PM 1 point [-]

The difference between AI safety and e.g. car safety is that humanity can survive a single car crash. Provably safe AI is needed because it's the only way to get it right, not because it's the easiest way.