The interesting thing is that they have essentially implemented a (sort of) Turing machine with differentiable functions, which enables training using gradient-based techniques. In principle you could always train resource-bounded Turing machines using combinatorial optimization techniques such as classical planners, SAT solvers or ILP solvers, theoretical complexity will be PSPACE-complete or NP-complete depending on the constraints that you impose. In practice, this is very much computationally expensive and doesn't scale past small examples. If this differentiable approach generalizes past toy examples, it will allow efficient training. In addition to the obvious practical applications (*) I think it may also have a deeper philosophical value: Consider word embeddings and the Vector space model: words are appear like discrete objects at syntactic level, while they represent concepts which intuitively feel like they lie in some sort of differentiable manifold: "David Cameron", "Angela Merkel" and "Jürgen Schmidhuber" are all quite similar to each other on the "species" dimension, Cameron and Merkel are more similar to each other than to Schmidhuber on the "job" dimension, Cameron and Schmidhuber are more similar on the "gender" dimension, Merkel and Schmidhuber are more similar on the "nationality" dimension. The concepts of "species", "job", "gender" and "nationality" also have their own topology, and so on. It is plausible that this ability to perceive an essentially continuous topology over discrete things is a fundamental property of how our mind works. Modern techniques in natural language processing and information retrieval are able to learn these kind of representations from data, but these are generally representations of individual words or short phrases. It is hypothesized that you can represent the meaning of arbitrarily complex sentences, and thus ideas, by combining the representations of their constituents, but nobody currently has a clue on how

If you want to put it that way, nothing conceptually interesting is going on in any neural network paper - we already know they're universal. My problem is the details: I visualize neural networks as like pachinko machines or Galton's quincunx - you drop a bunch of bits (many balls) into the top (bottom) layer of neurons (pins) and they cascade down to the bottom based on the activation functions (spacing of pins & how many balls hit a pin simultaneously), and at the bottom (top) is emitted a final smaller output (1 ball somewhere). I don't get the details of what it means to add a memory to this many-to-one function.