The world is indeed often surprising but frequently surprising in a way that scientific experiments open new questions that weren't in the mind of the scientists beforehand.
When looking at something like Reiki, Bill Nye and friends argue don't consider it an open question whether or not Reiki works just because we don't have well controlled studies investigating it. They consider it not to work because they don't believe that there's ki.
You might argue that they are wrong to do so, but that's still how they operate.
I've seen people like Bill Nye repeat this, and seen plenty of science-themed reminders that test results are often surprising
The phrase science-themed sounds to me more like it refers to science mythology than serious history of science. To the extend that you want to sensible talk about what scientists do, you have to listen to people who study what scientists do and that's not the speciality of a rocket scientist like Wernher Von Braun.
Ah, that's the definition about which we were talking past each other. I certainly wouldn’t say that "Reiki might work, and until we test it we just don't know!" Perhaps it "works" somewhat through the placebo effect, but even in the unlikely event of a study showing some random placebo controlled health benefit, it would still be astronomically unlikely that ki was the mechanism. (That's not to say that no one will look at the real mechanism after the fact, and try to pick out some superficial similarity to the idea of "ki".)...
This is the fourth of Tim Urban's series on Elon Musk, and this time it's about some reasoning processes that are made explicit, which LW readers should find very familiar. It's a potentially useful explicit model of how to make decisions for yourself.
http://waitbutwhy.com/2015/11/the-cook-and-the-chef-musks-secret-sauce.html