Grand pronouncements with an ass-covering move look silly :-)

One obvious problem is that you are assuming stability. Consider modeling something that changes (in complex ways) with time -- like the economy of the United States. Is "training data" from the 1950s relevant to the currrent situation?

Generally speaking, the speed at which your "training data" gets stale puts an upper limit on the relevant data that you can possibly have and that, in turn, puts an upper limit on the complexity of the model (NNs included) that you can build on its basis.