The opinion of folks at Singularity Institute is VERY relevant when discussing the methodology of studies done at Singularity Institute. They laughed at the thought of going through the motions of collecting the extra evidence and doing the extra rituals with statistics to impress journal editors. They did the Bayesian version instead.

If you want an enormous, controlled, statistically-significant study that's been published in a high-quality, peer reviewed journal, check out this study of brain training software in Nature.

I actually just found this study in the past hour. Their conclusion: Brain Training software doesn't work. But you get better at the games!

They cover Lumosity too which is funny because I was just looking into them myself. I was a bit concerned when I tried to look up the evidence for how "scientifically proven" Lumosity was (since they claim it ALL OVER THEIR SITE) and I later realized that the extent of their published findings were 2 conference posters, that aren't available anywhere online, and that were accepted to conferences I've never even heard of.

I think I'm gonna go with the researchers from Nature and Singularity Institute on this one.

the right response is to try to do better

Agreed. Better is better.

and nail down our results with more precision, not less

I disagree. For example, you can rule out Dual N-Back as a possible Intelligence Amplification intervention with less precision than Jaeggi used to repeatedly mis-prove it as one. Depends on what you mean by precision I suppose. If you mean more time, effort, people, and statistical significance then precision is not needed. If by precision, you just mean being more right... well, I agree, we should be more right.

Most bogus science is very precise: That's why it looks stronger than it is. Poor methodology and experimental design will still allow someone to prove any correlation with p < 0.05 significance. If I want to disprove someone who published an incorrect result, should I have to expend more time, people, and resources than they used just to over-prove the counter-claim with "more precision" -- even though their claim was never wrong due to "lack of precision" in the first place?

Calling for "more precision" is like calling for "more preparation". It has 100% applause appeal and costs nothing for people to call for. But it costs people actually doing research a lot of time. When you advocate for smarter people to use "more precision", you're also advocating for smarter people to do "less research"... the extra precision comes from somewhere.

Are you actually in favor of smarter people doing less research than they currently do?