Sufficiently Advanced Sanity

Eliezer Yudkowsky

From an unpublished story confronting Vinge's Law, written in 2004, as abstracted a bit:

"If you met someone who was substantially saner than yourself, how would you know?"

"The obvious mistake that sounds like deep wisdom is claiming that sanity looks like insanity to the insane. I would expect to discover sanity that struck me as wonderfully and surprisingly sane, sanity that shocked me but that I could verify on deeper examination, sanity that sounded wrong but that I could not actually prove to be wrong, and sanity that seemed completely bizarre."

"Like a history of 20th-century science, presented to a scientist from 1900. Much of the future history would sound insane, and easy to argue against. It would take a careful mind to realize none of it was more inconsistent with present knowledge than the scientific history of the 19th century with the knowledge of 1800. Someone who wished to dismiss the whole affair as crackpot would find a thousand excuses ready to hand, plenty of statements that sounded obviously wrong. Yet no crackpot could possibly fake the parts that were obviously right. That is what it is like to meet someone saner. They are not infallible, are not future histories of anything. But no one could counterfeit the wonderfully and surprisingly sane parts; they would need to be that sane themselves."

Spot the Bayesian problem, anyone? It's obvious to me today, but not to the me of 2004. Eliezer₂₀₀₄ would have seen the structure of the Bayesian problem the moment I pointed it out to him, but he might not have assigned it the same importance I would without a lot of other background.

Reply to: Shalmanese's Third Law

From an unpublished story confronting Vinge's Law, written in 2004, as abstracted a bit:

"If you met someone who was substantially saner than yourself, how would you know?"

"The obvious mistake that sounds like deep wisdom is claiming that sanity looks like insanity to the insane. I would expect to discover sanity that struck me as wonderfully and surprisingly sane, sanity that shocked me but that I could verify on deeper examination, sanity that sounded wrong but that I could not actually prove to be wrong, and sanity that seemed completely bizarre."

"Like a history of 20th-century science, presented to a scientist from 1900. Much of the future history would sound insane, and easy to argue against. It would take a careful mind to realize none of it was more inconsistent with present knowledge than the scientific history of the 19th century with the knowledge of 1800. Someone who wished to dismiss the whole affair as crackpot would find a thousand excuses ready to hand, plenty of statements that sounded obviously wrong. Yet no crackpot could possibly fake the parts that were obviously right. That is what it is like to meet someone saner. They are not infallible, are not future histories of anything. But no one could counterfeit the wonderfully and surprisingly sane parts; they would need to be that sane themselves."

You can require priors in order to make a computation. Your requirement doesn't cause the priors to magically appear; but you still require them.

I think Benquo means that you need enough observations to have reliable priors.

That's right -- the problem isn't that you need priors and don't have them, but that you need those two particular priors, and getting them probably involves enough work to raise your own sanity level quite a bit.