Sufficiently Advanced Sanity

Eliezer Yudkowsky

From an unpublished story confronting Vinge's Law, written in 2004, as abstracted a bit:

"If you met someone who was substantially saner than yourself, how would you know?"

"The obvious mistake that sounds like deep wisdom is claiming that sanity looks like insanity to the insane. I would expect to discover sanity that struck me as wonderfully and surprisingly sane, sanity that shocked me but that I could verify on deeper examination, sanity that sounded wrong but that I could not actually prove to be wrong, and sanity that seemed completely bizarre."

"Like a history of 20th-century science, presented to a scientist from 1900. Much of the future history would sound insane, and easy to argue against. It would take a careful mind to realize none of it was more inconsistent with present knowledge than the scientific history of the 19th century with the knowledge of 1800. Someone who wished to dismiss the whole affair as crackpot would find a thousand excuses ready to hand, plenty of statements that sounded obviously wrong. Yet no crackpot could possibly fake the parts that were obviously right. That is what it is like to meet someone saner. They are not infallible, are not future histories of anything. But no one could counterfeit the wonderfully and surprisingly sane parts; they would need to be that sane themselves."

Spot the Bayesian problem, anyone? It's obvious to me today, but not to the me of 2004. Eliezer₂₀₀₄ would have seen the structure of the Bayesian problem the moment I pointed it out to him, but he might not have assigned it the same importance I would without a lot of other background.

Reply to: Shalmanese's Third Law

From an unpublished story confronting Vinge's Law, written in 2004, as abstracted a bit:

"If you met someone who was substantially saner than yourself, how would you know?"

"The obvious mistake that sounds like deep wisdom is claiming that sanity looks like insanity to the insane. I would expect to discover sanity that struck me as wonderfully and surprisingly sane, sanity that shocked me but that I could verify on deeper examination, sanity that sounded wrong but that I could not actually prove to be wrong, and sanity that seemed completely bizarre."

"Like a history of 20th-century science, presented to a scientist from 1900. Much of the future history would sound insane, and easy to argue against. It would take a careful mind to realize none of it was more inconsistent with present knowledge than the scientific history of the 19th century with the knowledge of 1800. Someone who wished to dismiss the whole affair as crackpot would find a thousand excuses ready to hand, plenty of statements that sounded obviously wrong. Yet no crackpot could possibly fake the parts that were obviously right. That is what it is like to meet someone saner. They are not infallible, are not future histories of anything. But no one could counterfeit the wonderfully and surprisingly sane parts; they would need to be that sane themselves."

One problem is the assumption that being right and novel on some things implies being consistently right/sane. An important feature that separates "insanity" and stupidity is that "insanity" doesn't preclude domain-specific brilliance. Certainly a person being unusually right on some things is evidence for them being consistently right on others, but not overwhelmingly strong evidence.

Good point. Many advances were made by people of dubious sanity.