I don't know, that comment really seemed to suggest Bayesian networks. I guess you could allow for a distribution of possible activation functions, but that doesn't really fit what he said about learning the "exact" nonlinear function for every possible function. That fits more with bayes nets, which use a lookup table for every node.

Your example sounds like a bayesian net. But it doesn't really fit his description of learning optimal nonlinearities for functions.