In Disguised Queries, I talked about a classification task of "bleggs" and "rubes". The typical blegg is blue, egg-shaped, furred, flexible, opaque, glows in the dark, and contains vanadium. The typical rube is red, cube-shaped, smooth, hard, translucent, unglowing, and contains palladium. For the sake of simplicity, let us forget the characteristics of flexibility/hardness and opaqueness/translucency. This leaves five dimensions in thingspace: Color, shape, texture, luminance, and interior.

Suppose I want to create an Artificial Neural Network (ANN) to predict unobserved blegg characteristics from observed blegg characteristics. And suppose I'm fairly naive about ANNs: I've read excited popular science books about how neural networks are distributed, emergent, and parallel just like the human brain!! but I can't derive the differential equations for gradient descent in a non-recurrent multilayer network with sigmoid units (which is actually a lot easier than it sounds).

Then I might design a neural network that looks something like this:

Network 1 is for classifying bleggs and rubes. But since "blegg" is an unfamiliar and synthetic concept, I've also included a similar Network 1b for distinguishing humans from Space Monsters, with input from Aristotle ("All men are mortal") and Plato's Academy ("A featherless biped with broad nails").

A neural network needs a learning rule. The obvious idea is that when two nodes are often active at the same time, we should strengthen the connection between them—this is one of the first rules ever proposed for training a neural network, known as Hebb's Rule.

Thus, if you often saw things that were both blue and furred—thus simultaneously activating the "color" node in the + state and the "texture" node in the + state—the connection would strengthen between color and texture, so that + colors activated + textures, and vice versa. If you saw things that were blue and egg-shaped and vanadium-containing, that would strengthen positive mutual connections between color and shape and interior.

Let's say you've already seen plenty of bleggs and rubes come off the conveyor belt. But now you see something that's furred, egg-shaped, and—gasp!—reddish purple (which we'll model as a "color" activation level of -2/3). You haven't yet tested th...

bacon

bacon

bacon

bacon

bacon

bacon