But, when phlogiston was actually proposed, it and the idea that fire took something from the air really did have close to the same amount of evidence.
How do you figure?
The point is that the general region of evidence that causes one to locate a hypothesis and not have a hideously large amount of evidence for it is a pretty small region.
The point is that the general region of evidence that causes one to locate a hypothesis and yet not have a hideously large amount of evidence that someone moderately smarter than you could use to decide decisively whether or not the hypothesis is true is a pretty small region.
So is the upshot? Sometimes things go the way Eliezer depicts, and for much of the history of physics and astronomy this was a really good approximation. But it doesn't always go that way and it especially doesn't go that way if one is looking at fields other than physics.
It's not a post about how things usually go. It's a post about the minimum requirements to know something with near certainty for an intelligent agent.
Now, how likely is it that Einstein would have exactly enough observational evidence to raise General Relativity to the level of his attention, but only justify assigning it a 55% probability? Suppose General Relativity is a 29.3-bit hypothesis. How likely is it that Einstein would stumble across exactly 29.5 bits of evidence in the course of his physics reading?
Not likely! If Einstein had enough observational evidence to single out the correct equations of General Relativity in the first place, then he probably had enough evidence to be damn sure that General Relativity was true.
Einstein already had plenty of observational evidence, so he used it properly. You say: "Will someone come up with a sudden idea that makes everything work nicely and explains dark matter and lots of other stuff besides just as SR explained multiple apparently unconnected problems at once? Maybe, but I doubt it." In physics, if they've truly narrowed it down like that, the conclusion is that they ought not need more evidence, not that the social forces of science will deterministically overturn every confusion dividing a professional field with a single "Eureka!", even those resulting from ignorance, politics, or stupidity.
If physicists have inefficiently parsed the data and gotten it down to the last three or so bits, then improving their thought would get them down to one theory as surely as providing enough mountains of data and analysis would. We pretty much know that as humans they are inefficient thinkers and have such room to improve, and even if they are at their ceiling, other possible intelligent systems could vastly improve upon individual or collective human thought even with less than all of our resources.
How do you figure?
They both fit the same basic evidence: burning a candle or similar object in a small enclosure made it go out. Similar remarks applied to small living animals and combinations of candles and animals. Moreover, many forms of combustion visibly gave off something into the air. Indeed the theory "combustion occurs when something from the substance goes into the air" is simpler than "combustion occurs when something from the air combines with the substance and sometimes but not always something else is added back into the ai...
Today's post, Einstein's Arrogance was originally published on 25 September 2007. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was How Much Evidence Does It Take?, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.