= 27d567bc2d48d9f40e9ccc20ea370b15)
gwern comments on [Link]"Neural Turing Machines" - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (21)
A neural network is just a function of some sort (inputs to outputs). So is a finite state machine (which is what a regular Turing machine uses). Put that way, nothing conceptually interesting is going on. As usual, the devil is in the details here.
If you want to put it that way, nothing conceptually interesting is going on in any neural network paper - we already know they're universal.
My problem is the details: I visualize neural networks as like pachinko machines or Galton's quincunx - you drop a bunch of bits (many balls) into the top (bottom) layer of neurons (pins) and they cascade down to the bottom based on the activation functions (spacing of pins & how many balls hit a pin simultaneously), and at the bottom (top) is emitted a final smaller output (1 ball somewhere). I don't get the details of what it means to add a memory to this many-to-one function.
Recurrent Neural Networks feed their outputs back into their inputs, and run continuously. Instead of visualizing a river flowing directly downstream, visualize a river with a little cove off to one side, where vortices swirl around.
The same way you add memory to many-to-one boolean functions: feedback loops.
I think another way to look at neural networks is they are nested non-linear regression models.
I am probably in the minority here, but I don't think the stuff in the OP is that interesting.