Well thank the benevolence of the Friendly AI that this intelligence didn't see a helium balloon first. Just imagine the kinds of theories it might produce then!
If you see one object falling in a particular way, you might infer that all objects fall that way - but it's an extremely weak inference, as the strength of a single observation is spread over the entirety of "all things". We were so confident in Newton's formulation for such a long time because we had a vast store of observations, and were aware of confounding influences that masked the underlying pattern: things like air resistance and buoyancy. The understanding that all things fall at a given rate was a strong and reliable inference because we observed it to hold across many, many things. Once we knew that, we could show that such behavior was consistent with Newton's hypothesized force. More importantly, we had already determined through observation that the objects in the Solar system moved in elliptical orbits, but we didn't know why. We were able to show that Newton's hypothesized forces would result in objects moving in such a way, and so concluded that his description was correct.
Eliezer is almost certainly wrong about what a hyper-rational AI could determine from a limited set of observations. It would probably notice the implications of Maxwell's laws that require Relativity to fully explain - something real physicists missed for a generation - because the implications follow directly from the mathematics. Actually producing the laws in the first place requires a lot of data regarding electricity and magnetism.
His projected super-intelligence would very quickly outleap its data and rush to all sorts of unsupportable inferences. If it confused those inferences with conclusions, it would fall into error faster than we could possibly correct it, and if it lacked the long, slow, tedious process of checking and re-checking data that science uses, it would be unlikely to ever correct those errors.
New Scientist on changing the definition of science, ungated here:
I'm a good deal less of a lonely iconoclast than I seem. Maybe it's just the way I talk.
The points of departure between myself and mainstream let's-reformulate-Science-as-Bayesianism is that:
(1) I'm not in academia and can censor myself a lot less when it comes to saying "extreme" things that others might well already be thinking.
(2) I think that just teaching probability theory won't be nearly enough. We'll have to synthesize lessons from multiple sciences like cognitive biases and social psychology, forming a new coherent Art of Bayescraft, before we are actually going to do any better in the real world than modern science. Science tolerates errors, Bayescraft does not. Nobel laureate Robert Aumann, who first proved that Bayesians with the same priors cannot agree to disagree, is a believing Orthodox Jew. Probability theory alone won't do the trick, when it comes to really teaching scientists. This is my primary point of departure, and it is not something I've seen suggested elsewhere.
(3) I think it is possible to do better in the real world. In the extreme case, a Bayesian superintelligence could use enormously less sensory information than a human scientist to come to correct conclusions. First time you ever see an apple fall down, you observe the position goes as the square of time, invent calculus, generalize Newton's Laws... and see that Newton's Laws involve action at a distance, look for alternative explanations with increased locality, invent relativistic covariance around a hypothetical speed limit, and consider that General Relativity might be worth testing. Humans do not process evidence efficiently—our minds are so noisy that it requires orders of magnitude more extra evidence to set us back on track after we derail. Our collective, academia, is even slower.