First I ask you, "give me a probability distribution on the outcomes of a future event". Then you observe some relevant data. Then I ask you again for a probability distribution on outcomes.
If I can compare your prior probabilities with your posterior probabilities, I can infer what likelihood ratios you assigned to the evidence, i.e. P(E|H1) : P(E|H2) : P(E|H3).
If I trusted your rationality, I'd take my prior and do a Bayesian update using your implied likelihood ratios. But I scoff at your implied likelihood ratios, because I know the likelihood values are determined by the operation of some intuitive algorithm that is unequipped for the domain. So instead of using your implied likelihood ratios wholesale, I need some other way of analyzing how your conclusions should affect my conclusions.
Insight, almost by definition, gives you a better mental algorithm for assigning posterior probabilities to hypotheses and making predictions -- i.e. an algorithm with a higher expected Bayes-score (defined in Eliezer's Technical Explanation).
Your algorithm provides "increased evidence" to me, an outside observer, because now I will do something closer to trusting your implied likelihood ratios, and I will rationally allow your analysis of the evidence to have more sway over my own.
The "outside observer" is actually you as well. You're the one who knows to listen to your analysis more if it's an insightful one.
I originally wanted to answer the question, "When does an insight count as evidence?" So now I have given a precise description of the relationship between insight and evidence.
This is a response to Eliezer Yudkowsky's The Logical Fallacy of Generalization from Fictional Evidence and Alex Flint's When does an insight count as evidence? as well as komponisto's recent request for science fiction recommendations.
My thesis is that insight forms a category that is distinct from evidence, and that fiction can provide insight, even if it can't provide much evidence. To give some idea of what I mean, I'll list the insights I gained from one particular piece of fiction (published in 1992), which have influenced my life to a large degree:
So what is insight, as opposed to evidence? First of all, notice that logically omniscient Bayesians have no use for insight. They would have known all of the above without having observed anything (assuming they had a reasonable prior). So insight must be related to logical uncertainty, and a feature only of minds that are computationally constrained. I suspect that we won't fully understand the nature of insight until the problem of logical uncertainty is solved, but here are some of my thoughts about it in the mean time:
So a challenge for us is to distinguish true insights from unhelpful distractions in fiction. Eliezer mentioned people who let the Matrix and Terminator dominate their thoughts about the future, and I agree that we have to be careful not to let our minds consider fiction as evidence. But is there also some skill that can be learned, to pick out the insights, and not just to ignore the distractions?
P.S., what insights have you gained from fiction?
P.P.S., I guess I should mention the name of the book for the search engines: A Fire Upon the Deep by Vernor Vinge.