but deep learning is really a new thing under the sun.

On the contrary, the idea of making deeper nets is nearly as old as ordinary 2-layer neural nets, successful implementations dates back to the late 90's in the form of convolutional neural nets, and they had another burst of popularity in 2006.

Advances in hardware, data availability, heuristics about architecture and training, and large-scale corporate attention have allowed the current burst of rapid progress.

This is both heartening, because the foundations of its success are deep, and tempering, because the limitations that have held it back before could resurface to some degree.