"Now, just to be clear," Harry said, "if the professor does levitate you, Dad, when you know you haven't been attached to any wires, that's going to be sufficient evidence. You're not going to turn around and say that it's a magician's trick. That wouldn't be fair play. If you feel that way, you should say so now, and we can figure out a different experiment instead."
Harry's father, Professor Michael Verres-Evans, rolled his eyes. "Yes, Harry."
"And you, Mum, your theory says that the professor should be able to do this, and if that doesn't happen, you'll admit you're mistaken. Nothing about how magic doesn't work when people are sceptical of it, or anything like that."
An important aspect of a scientific experiment is that you figure out the design of your experiment and how you are going to interpret the resulting data before you execute it. When you are using an experiment to resolve a disagreement, everyone involved should agree to this procedure in advance (assuming trust in intellectual honesty, which I think holds here).
Polls and surveys have self selection issues. It is good to take some step to counteract the tendency of blog readers to like reading stuff on blogs, but is it sufficient? Should Luke be convinced the bias has been remedied?
Luke was arguing in part that academic papers published in journals helps to reach a small but highly valued class of people. Should a poll that doesn't track his value of reaching the participants influence his policy decision?
I think it would be a better process, if Tom first presented the design of the poll, and allow some time for the community to critique the design. Only after modifications have been made to address criticisms, and Luke and Tom agree that it is a valid test of the question they are interested in, should the poll itself (if that is still the form of the experiment) be conducted.
I would agree if I were going to spend a lot of hours on this, but I unfortunately don't have that kind of time.
In our discussion of academic papers, Lukeprog argued that lots of smart people preferred to read ideas in academic paper format. Based on my observations, I mostly disagree. But that's just anecdotal evidence. Let's use Science!
Suppose someone at the Singularity Institute thought up a cool new idea: it could be about rationality, Friendly AI, decision theory, making money, or any of the other topics we discuss here on LW. Explaining it takes about ten pages, and it's nontechnical enough that it can be explained to a general audience of non-mathematicians. Which of the following explanations would you be most likely to actually sit down and read through?
EDIT: To state the obvious, this poll will be biased in favor of blog postings, since it's on a blog. However, I still think it'll provide data that's much better than anecdotal guessing. I've emailed a few rationalist mailing lists to try and counteract this effect.