Most uses of the word "insight" mean something similar to "seeing into the nature of things," but it's not clear that the particular use you have here meshes well with at least one other common use of the word. Eliezer captured it well:
an "insight" is a chunk of knowledge which, if you possess it, decreases the cost of solving a whole range of governed problems.
As a simple example, let's say you were trying to prove the statement "there are infinitely many primes." To progress on this problem at all, you'll probably need to realize:
Insight 1 - The statement "there are infinitely many primes" can be re-expressed as "it is not the case that there are finitely many primes."
Insight 2 - A statement of the form "not P" can sometimes be proven by assuming "P" and showing that this assumption leads to contradiction.
After assuming there are finitely many primes (i.e. there exists an n such that P = {p1, p2, ..., pn} is the set of all primes), insight again comes into play when one realizes:
In this latter case, the insight consisted in using the fundamental theorem of arithmetic to transform the previous goal of "deriving a contradiction" to a more specific goal of "finding an integer that is not divisible by any prime in P."
I realize that the context of problem solving is somewhat removed from the context of assessing the probability of hypotheses, but perhaps we should clarify what particular usage of the word "insight" is meant if we're going to be analyzing it in detail.
This is a response to Eliezer Yudkowsky's The Logical Fallacy of Generalization from Fictional Evidence and Alex Flint's When does an insight count as evidence? as well as komponisto's recent request for science fiction recommendations.
My thesis is that insight forms a category that is distinct from evidence, and that fiction can provide insight, even if it can't provide much evidence. To give some idea of what I mean, I'll list the insights I gained from one particular piece of fiction (published in 1992), which have influenced my life to a large degree:
So what is insight, as opposed to evidence? First of all, notice that logically omniscient Bayesians have no use for insight. They would have known all of the above without having observed anything (assuming they had a reasonable prior). So insight must be related to logical uncertainty, and a feature only of minds that are computationally constrained. I suspect that we won't fully understand the nature of insight until the problem of logical uncertainty is solved, but here are some of my thoughts about it in the mean time:
So a challenge for us is to distinguish true insights from unhelpful distractions in fiction. Eliezer mentioned people who let the Matrix and Terminator dominate their thoughts about the future, and I agree that we have to be careful not to let our minds consider fiction as evidence. But is there also some skill that can be learned, to pick out the insights, and not just to ignore the distractions?
P.S., what insights have you gained from fiction?
P.P.S., I guess I should mention the name of the book for the search engines: A Fire Upon the Deep by Vernor Vinge.