There's been some work on getting machines to try to make reasonable conjectures and definitions (see for example the work of Simon Colton) but I'm not aware of any work of actually trying to teach machines. I suspect this would be very difficult since most machine learning systems work best when the problems are in some vague sense fuzzy rather than formal.
The problem of deciding which definitions are interesting is far from being formal.
Does anyone know of work that attempts to build a theorem prover by learning-from-examples? I'm imagining extracting a large corpus of theorems from back issues of mathematical journals, then applying unsupervised structure discovery techniques from machine learning to discover recurring patterns.
Perhaps a model of the "set of theorems that humans tend to produce" would be helpful in proving new theorems.
The unsupervised-structure-discovery bit does seem within the realm of current machine learning.
Any references to related work?