Crankery in old age is really sad. (And, weirdly enough, the topic of the latest SMBC strip.) I've wondered sometimes about why this happens; it seems to correlate with age but not young or middle age, so I discard the default hypothesis that a false idea finally got lucky and slipped through one's memetic defenses (since that feels like more of a Poisson distribution).
As you age, your general intelligence takes a real beating but your personality traits like Openness don't change so much (and great scientists will tend to be very Open). Openness and low IQ is also correlated with being New Agey crap and that sort of thing (IIRC, citations in Miller's Spent), and these bizarre false theories do seem New Agey in some respects ('it's all, like, circles of energy man!').
Could this be the problem? When you're young and sharp, you can keenly examine new theories and ideas, although you have the handicap of being ignorant and not having spent much time on matters; so your productivity rises in your middle age to your early 40s; but by that point, your raw intelligence has become blunt and dull, and your knowledge may be increasingly out of date, while your interest in new ideas remains the same. So you continue to seek out or look at new ideas, while you are no longer able to evaluate them.
So to test this we'd want to test the following:
Check the New Age <-> high Openness/low IQ correlation
(If it doesn't hold true in unscientifically capable or trained populations, why would it hold true for old eminent scientists?)
Test that 'weird and false ideas' do disproportionately pop up in old age
(as opposed to being adopted as impressionable grad students but only worked on & espoused in the leisure of old age/tenure/retirement, or following some 'lightning strike' model, due to say the simultaneous occurrence of emotional trauma and a meme impression)
That their rate does correlate with being Open
(If there is no such correlation, it may be that the entire effect would be IQ-related.)
That their rate does inversely correlate with IQ at that time, ideally, at the instant they accepted the false idea
(If it turned out the acceptors of false ideas had most of their IQ intact, and the more conventional ones were stupid, this would be rather odd.)
Come to think of it, compartmentalization effects (like with religion) might allow one to be brilliant for a while until the disjoint beliefs finally interact. Were the physicists who became cranks eventually ever particularly good at dealing with the flaws in reasoning of crankery while they were still respectable?
Similarly, it's possible that as they became more prominent, they needed less and less to justify any given statement to the people with which they interacted, having gained more and more authority, and thus just lost socially enforced habits of thought...
Actually, the simplest explanation I can think of is that it takes the span of time from youth to middle age just to build up the reputation necessary for your eventual fall into crankery to be considered newsworthy... Are physicists really more likely to develop cranky beliefs over a set period of time as the general population?
Similarly, it's possible that as they became more prominent, they needed less and less to justify any given statement to the people with which they interacted, having gained more and more authority, and thus just lost socially enforced habits of thought...
Then why don't they retract when they run into criticism, as they inevitably do?
Are physicists really more likely to develop cranky beliefs over a set period of time as the general population?
Gosh, I hope not. The general population is pretty pathetic. I'm not interested so much in whether old eminent scientists or old eminent physicists in particular are worse than the general public, but the reasons why any eminent old physicists who adopts a crank view has fallen prey to that crank view. What, exactly, went wrong in their heads that had previously always gone right?
Then why don't they retract when they run into criticism, as they inevitably do?
Well, why would they? After all, the last time they said that revolutionary theory X was correct and everybody thought they were wrong, it turned out they were right! That's why they were prominent!
Strictly speaking, incredible levels of sanity are not necessary to find the truth, they only make it easier to do so. There are sufficiently large populations of physicists who aren't ludicrously awesomely good that some of them are bound to turn out to be right every so often, even if they got to the right reasons for the wrong reasons. Maybe.
What, exactly, went wrong in their heads that had previously always gone right?
Well, why would it be any different for them than for the rest of us? The reason great scientists are great, I assume, has to do with the greatness of their discoveries. But why would only incredibly sane people make great discoveries? Can not just a little cleverness and a sufficiently large amount of luck (a sufficiently large pool of peers) do the same thing?
That's why they were prominent!
But they also had to correctly pick out the one which was the revolution - every such scientist faces tons of ideas and hypotheses to consider. Is your hypothesis here a kind of regression to the mean: all scientists are equally vulnerable to holding crankery?
Essentially, yes. They just happened to have had a string of sixes when they threw the dice, culminating in prominence. If you suppose that the crank-susceptible scientists significantly outnumber the crank-immune, you get predictions which resemble our observations that many prominent scientists are susceptible to crank.
Where by crank-susceptible I mean, approximately, susceptible to infection by crank...
I hypothesise a mechanism in the brain that works as a black-box evaluator of idea quality - you feed it ideas and it just gives you an oracular answer of how good the idea is. c.f. the widely-experienced phenomenon where you wake up in the middle of the night with a brilliant idea, write it down and then in the morning realise it's rubbish. This idea evaluator going awry might lead to many an accepted crackpot offer.
Your hypothesis seems to be not the idea evaluator going awry, but the ability to judge its outputs going awry.
This matches up nicely with the process-1/process-2 set of theories, where IQ is just a sort of 'algorithmic' or simulation thinking which is either invoked or not invoked based on one's rational or reflective tendencies. Stanovich extends this to include 'mindware', cached bits of logical or statistical reasoning which the IQ can apply to problems.
The problem with that, and why I didn't mention it, is that we currently have few good ways to measure rational/reflective tendencies, so we would have a hard time finding the possible inverse correlation.
Even if we did, we might not expect to find anything: the point of reflective thinking is to know when to switch over into expensive IQ and critical thinking, but how would this not happen at some point as these eminent scientists write up their ideas and argue with other people? If it was all reflectiveness, the first time they tipped over into non-heuristic thinking, they'd realize how stupid they were being. So do they manage to never do it? Or do they uncritically accept their ideas and then by the time anything might cause critical thinking, they've already hardened around it with confirmation bias and stubbornness and whatnot?
Doesn't mean too much; Dyson started off with a lot of raw ability, obviously, and if you look at the graphs, things like vocabulary or facts are the categories that decline very little or actually increase with age. I don't believe Dyson is inventing any new ideas in his writing, and the ones I hear about like geoengineering tend to be controversial and possible things that he would one day regret espousing.
I dunno, I'm 26 and I've been getting more and more scientifically conservative as of the last 10 years. Excluding the topics of intense interest, of course, but in vast majority of those I tend to end up even more conservative as I learn why the scientists in the field have priors they have, and so i understand why there is apparently strong 'prejudice' among scientists, aka, priors. Yes, true, this 'prejudice' ends up wrong now and then, but the public has a cherry picked selection of when it ended wrong, without thousands times the cases where the 'prejudice' was right, and it is a very correct prior overall. Note: this absolutely doesn't apply to the consensus among non-scientists, which includes proto-sciences, and engineer 'consensus'.
Well, that goes more for the unsettled, like popular debates, upcoming new theories, and such. I always loved settled science i guess, for it being actually something to learn reliably from. I just had more 'neutral' attitude towards much of unsettled. I am glad about publication of neutrinos going at FTL though; that was cool thing they published it, rather than just hid it under the rug until they find the wiring problem; that creates bias otherwise.
Funnily enough, this article from The Atlantic accuses Dyson of similar: http://www.theatlantic.com/magazine/print/2010/12/the-danger-of-cosmic-genius/8306/
The comparison between Carter and Kelvin/Tait seems inaccurate. Tait and Kelvin had an incorrect hypothesis but both did a lot of good work. Carter has a hypothesis which has been discounted for a long time and nothing else. Moreover, when Tait was doing his work, the hypothesis seemed plausible given what was known at the time. It took years to understand atoms enough to know that their hypothesis didn't work. The situation is not similar to Carter's.
There is good reason to pay more attention to scientific experts than to amateurs, so long as science is based on experiments. Only trained experts can do experiments with the care and precision that experiments demand.
This appears to me to be, at best, a non sequitur. The problem with cranks is not that they are careless in design or execution of experiments. Carter's problem is that he doesn't have a theory.
Does anyone have any idea what Dyson meant?
Presumably the stuff scientists learn by apprenticeship, per Teaching The Unteachable (EY noting that 155 out of 503 Nobel laureates were the students of Nobel laureates).
Freeman Dyson writes in the New York Review of Books about people who took up the crackpot offer. Not just complete cranks, but eminent scientists such as Eddington who got into crankery in their later years.
New thing I learnt: Dyson was not only a good friend of Immanuel Velikovsky, but considers him a greatly underappreciated poet.