The rest of the time Harry thinks up a clever explanation, and once the explanation is clever enough to solve all the odd constraints placed on it, (1) he stops looking for other explanations, and (2) he doesn't check to see if he is actually right.
Examples:
Hariezer decides in this chapter that comed-tea MUST work by causing you to drink it right before something spit-take worthy happens. The tea predicts the humor, and then magics you into drinking it. Of course, he does no experiments to test this hypothesis at all (ironic that just a few chapters ago he lecture Hermione about only doing 1 experiment to test her idea).
Wizards losing their power in chap. 22
Here is the thing about science, step 0 needs to be make sure you’re trying to explain a real phenomena. Hariezer knows this, he tells the story of N-rays earlier in the chapter, but completely fails to understand the point.
Hariezer and Draco have decided, based on one anecdote (the founders of Hogwarts were the best wizards ever, supposedly) that wizards are weaker today than in the past. The first thing they should do is find out if wizards are actually getting weaker. After all, the two most dangerous dark wizards ever were both recent, Grindelwald and Voldemort. Dumbledore is no slouch. Even four students were able to make the marauders map just one generation before Harry. (Incidentally, this is exactly where neoreactionaries often go wrong- they assume things are getting worse without actually checking, and then create elaborate explanations for non-existent facts).
Anyway, for the purposes of the story, I’m sure it’ll turn out that wizards are getting weaker, because Yudkoswky wrote it. But this would have been a great chance to teach an actually useful lesson, and it would make the N-ray story told earlier a useful example, and not a random factoid.
Using literally the exact same logic that Intelligent Design proponents use (and doing exactly 0 experiments), Hariezer decides while thinking over breakfast:
Some intelligent engineer, then, had created the Source of Magic, and told it to pay attention to a particular DNA marker.
The obvious next thought was that this had something to do with “Atlantis”.
Gateway to the after life in chap. 39:
Here is Hariezer’s response to the gateway to the afterlife:
That doesn’t even sound like an interesting fraud,“ Harry said, his voice calmer now that there was nothing there to make him hope, or make him angry for having hopes dashed. "Someone built a stone archway, made a little black rippling surface between it that Vanished anything it touched, and enchanted it to whisper to people and hypnotize them.”
Do you see how incurious Hariezer is? If someone told me there was a LITERAL GATEWAY TO THE AFTERLIFE I’d want to see it. I’d want to test it, see it. Can we try to record and amplify the whispers? Are things being said?
No surprise, then, that the wizarding world lived in a conceptual universe bounded - not by fundamental laws of magic that nobody even knew - but just by the surface rules of known Charms and enchantments…Even if Harry’s first guess had been mistaken, one way or another it was still inconceivable that the fundamental laws of the universe contained a special case for human lips shaping the phrase ‘Wingardium Leviosa’. …What were theultimate possibilities of invention, if the underlying laws of the universe permitted an eleven-year-old with a stick to violate almost every constraint in the Muggle version of physics?
You know what would be awesome? IF YOU GOT AROUND TO DOING SOME EXPERIMENTS AND EXPLORING THIS IDEA. The absolute essence of science is NOT asking these questions, it’s deciding to try to find out the fucking answers! You can’t be content to just wonder about things, you have to put the work in! Hariezer’s wonderment never gets past the stoned-college-kid wondering aloud and into ACTUAL exploration, and its getting really frustrating.
Vision and the invisibility cloak in chap. 95:
arry had set the alarm upon his mechanical watch to tell him when it was lunchtime, since he couldn’t actually look at his wrist, being invisible and all that. It raised the question of how his eyeglasses worked while he was wearing the Cloak. For that matter the Law of the Excluded Middle seemed to imply that either the rhodopsin complexes in his retina were absorbing photons and transducing them to neural spikes, or alternatively, those photons were going straight through his body and out the other side, but not both. It really did seem increasingly likely that invisibility cloaks let you see outward while being invisible yourself because, on some fundamental level, that was how the caster had - not wanted - but implicitly believed - that invisibility should work.
This would be an excellent fucking question to explore, maybe via some experiments. But no. I’ve totally given up on this story exploring the magic world in any detail at all. Anyway, Hariezer skips straight from “I wonder how this works” to “it must work this way, how could we exploit it"
Centaurs and astrology in chap. 101
Still in the woods, Hariezer encounters a centaur who tries to kill him, because he divines that Hariezer is going to make all the stars die.
There are some standard anti-astrology arguments, which again seems to be fighting the actual situation because the centaurs successfully use astrology to divine things.
We get this:
“Cometary orbits are also set thousands of years in advance so they shouldn’t correlate much to current events. And the light of the stars takes years to travel from the stars to Earth, and the stars don’t move much at all, not visibly. So the obvious hypothesis is that centaurs have a native magical talent for Divination which you just, well, project onto the night sky.”
There are so, so many other hypothesis Hariezer. Maybe starlight has a magical component that waxes and wanes as stars align into different magical symbols or some such. The HPMOR scientific method:
observation -> generate 1 hypothesis -> assume you are right -> it turns out that you are right.
Using literally the exact same logic that Intelligent Design proponents use (and doing exactly 0 experiments), Hariezer decides while thinking over breakfast:
Some intelligent engineer, then, had created the Source of Magic, and told it to pay attention to a particular DNA marker.
The obvious next thought was that this had something to do with “Atlantis”.
How is this 'literally the exact same logic that ID proponents use?' Creationists fallacize away the concept of natural selection, but I don't see how Harry is being unreasonable, given what he knows about the universe.
You are unlikely to see me posting here again, after today. There is a saying here that politics is the mind-killer. My heretical realization lately is that philosophy, as generally practiced, can also be mind-killing.
As many of you know I am, or was running a twice-monthly Rationality: AI to Zombies reading group. One of the bits I desired to include in each reading group post was a collection of contrasting views. To research such views I've found myself listening during my commute to talks given by other thinkers in the field, e.g. Nick Bostrom, Anders Sandberg, and Ray Kurzweil, and people I feel are doing “ideologically aligned” work, like Aubrey de Grey, Christine Peterson, and Robert Freitas. Some of these were talks I had seen before, or generally views I had been exposed to in the past. But looking through the lens of learning and applying rationality, I came to a surprising (to me) conclusion: it was philosophical thinkers that demonstrated the largest and most costly mistakes. On the other hand, de Grey and others who are primarily working on the scientific and/or engineering challenges of singularity and transhumanist technologies were far less likely to subject themselves to epistematic mistakes of significant consequences.
Philosophy as the anti-science...
What sort of mistakes? Most often reasoning by analogy. To cite a specific example, one of the core underlying assumption of singularity interpretation of super-intelligence is that just as a chimpanzee would be unable to predict what a human intelligence would do or how we would make decisions (aside: how would we know? Were any chimps consulted?), we would be equally inept in the face of a super-intelligence. This argument is, however, nonsense. The human capacity for abstract reasoning over mathematical models is in principle a fully general intelligent behaviour, as the scientific revolution has shown: there is no aspect of the natural world which has remained beyond the reach of human understanding, once a sufficient amount of evidence is available. The wave-particle duality of quantum physics, or the 11-dimensional space of string theory may defy human intuition, i.e. our built-in intelligence. But we have proven ourselves perfectly capable of understanding the logical implications of models which employ them. We may not be able to build intuition for how a super-intelligence thinks. Maybe—that's not proven either. But even if that is so, we will be able to reason about its intelligent behaviour in advance, just like string theorists are able to reason about 11-dimensional space-time without using their evolutionarily derived intuitions at all.
This post is not about the singularity nature of super-intelligence—that was merely my choice of an illustrative example of a category of mistakes that are too often made by those with a philosophical background rather than the empirical sciences: the reasoning by analogy instead of the building and analyzing of predictive models. The fundamental mistake here is that reasoning by analogy is not in itself a sufficient explanation for a natural phenomenon, because it says nothing about the context sensitivity or insensitivity of the original example and under what conditions it may or may not hold true in a different situation.
A successful physicist or biologist or computer engineer would have approached the problem differently. A core part of being successful in these areas is knowing when it is that you have insufficient information to draw conclusions. If you don't know what you don't know, then you can't know when you might be wrong. To be an effective rationalist, it is often not important to answer “what is the calculated probability of that outcome?” The better first question is “what is the uncertainty in my calculated probability of that outcome?” If the uncertainty is too high, then the data supports no conclusions. And the way you reduce uncertainty is that you build models for the domain in question and empirically test them.
The lens that sees its own flaws...
Coming back to LessWrong and the sequences. In the preface to Rationality, Eliezer Yudkowsky says his biggest regret is that he did not make the material in the sequences more practical. The problem is in fact deeper than that. The art of rationality is the art of truth seeking, and empiricism is part and parcel essential to truth seeking. There's lip service done to empiricism throughout, but in all the “applied” sequences relating to quantum physics and artificial intelligence it appears to be forgotten. We get instead definitive conclusions drawn from thought experiments only. It is perhaps not surprising that these sequences seem the most controversial.
I have for a long time been concerned that those sequences in particular promote some ungrounded conclusions. I had thought that while annoying this was perhaps a one-off mistake that was fixable. Recently I have realized that the underlying cause runs much deeper: what is taught by the sequences is a form of flawed truth-seeking (thought experiments favored over real world experiments) which inevitably results in errors, and the errors I take issue with in the sequences are merely examples of this phenomenon.
And these errors have consequences. Every single day, 100,000 people die of preventable causes, and every day we continue to risk extinction of the human race at unacceptably high odds. There is work that could be done now to alleviate both of these issues. But within the LessWrong community there is actually outright hostility to work that has a reasonable chance of alleviating suffering (e.g. artificial general intelligence applied to molecular manufacturing and life-science research) due to concerns arrived at by flawed reasoning.
I now regard the sequences as a memetic hazard, one which may at the end of the day be doing more harm than good. One should work to develop one's own rationality, but I now fear that the approach taken by the LessWrong community as a continuation of the sequences may result in more harm than good. The anti-humanitarian behaviors I observe in this community are not the result of initial conditions but the process itself.
What next?
How do we fix this? I don't know. On a personal level, I am no longer sure engagement with such a community is a net benefit. I expect this to be my last post to LessWrong. It may happen that I check back in from time to time, but for the most part I intend to try not to. I wish you all the best.
A note about effective altruism…
One shining light of goodness in this community is the focus on effective altruism—doing the most good to the most people as measured by some objective means. This is a noble goal, and the correct goal for a rationalist who wants to contribute to charity. Unfortunately it too has been poisoned by incorrect modes of thought.
Existential risk reduction, the argument goes, trumps all forms of charitable work because reducing the chance of extinction by even a small amount has far more expected utility than would accomplishing all other charitable works combined. The problem lies in the likelihood of extinction, and the actions selected in reducing existential risk. There is so much uncertainty regarding what we know, and so much uncertainty regarding what we don't know that it is impossible to determine with any accuracy the expected risk of, say, unfriendly artificial intelligence creating perpetual suboptimal outcomes, or what effect charitable work in the area (e.g. MIRI) is have to reduce that risk, if any.
This is best explored by an example of existential risk done right. Asteroid and cometary impacts is perhaps the category of external (not-human-caused) existential risk which we know the most about, and have done the most to mitigate. When it was recognized that impactors were a risk to be taken seriously, we recognized what we did not know about the phenomenon: what were the orbits and masses of Earth-crossing asteroids? We built telescopes to find out. What is the material composition of these objects? We built space probes and collected meteorite samples to find out. How damaging an impact would there be for various material properties, speeds, and incidence angles? We built high-speed projectile test ranges to find out. What could be done to change the course of an asteroid found to be on collision course? We have executed at least one impact probe and will monitor the effect that had on the comet's orbit, and have on the drawing board probes that will use gravitational mechanisms to move their target. In short, we identified what it is that we don't know and sought to resolve those uncertainties.
How then might one approach an existential risk like unfriendly artificial intelligence? By identifying what it is we don't know about the phenomenon, and seeking to experimentally resolve that uncertainty. What relevant facts do we not know about (unfriendly) artificial intelligence? Well, much of our uncertainty about the actions of an unfriendly AI could be resolved if we were to know more about how such agents construct their thought models, and relatedly what language were used to construct their goal systems. We could also stand to benefit from knowing more practical information (experimental data) about in what ways AI boxing works and in what ways it does not, and how much that is dependent on the structure of the AI itself. Thankfully there is an institution that is doing that kind of work: the Future of Life institute (not MIRI).
Where should I send my charitable donations?
Aubrey de Grey's SENS Research Foundation.
100% of my charitable donations are going to SENS. Why they do not get more play in the effective altruism community is beyond me.
If you feel you want to spread your money around, here are some non-profits which have I have vetted for doing reliable, evidence-based work on singularity technologies and existential risk:
I wish I could recommend a skepticism, empiricism, and rationality promoting institute. Unfortunately I am not aware of an organization which does not suffer from the flaws I identified above.
Addendum regarding unfinished business
I will no longer be running the Rationality: From AI to Zombies reading group as I am no longer in good conscience able or willing to host it, or participate in this site, even from my typically contrarian point of view. Nevertheless, I am enough of a libertarian that I feel it is not my role to put up roadblocks to others who wish to delve into the material as it is presented. So if someone wants to take over the role of organizing these reading groups, I would be happy to hand over the reigns to that person. If you think that person should be you, please leave a reply in another thread, not here.
EDIT: Obviously I'll stick around long enough to answer questions below :)