Meta-analysis on cognitive effects of modafinil (my bachelor thesis)

Well, meta-analyses certainly are an area of interest to me, and I was disappointed in 2012 by "Cognition Enhancement by Modafinil: A Meta-Analysis" (Kelley et al 20120) which used only 3 studies, and so was not very informative. A new meta-analysis would be great. But... I read quickly through it, and I saw no meta-analysis. Just a literature review. What's with the post title?

Modafinil significantly improved performance in 26 out of 102 cognitive tests, but significantly decreased performance in 3 cognitive tests.

Nitpick: I really hate this use of 'significantly' and I ban it from my own writing. Is this referring to effect sizes or p-values?

Notably, modafinil appears to have detrimental effects on mental flexibility. Although 4 studies employed the Intra/Extradimensional Set Shift task (ID/ED), no performance improvements could be detected. Performance was even reduced in a study by Randall et al. (2004). Furthermore, Müller et al. (2012) found that subjects on modafinil had lower flexibility scores in the Abbreviated Torrance task for adults.

Eh. Absence of improvement != damage. Randal 2004 didn't find a statistically-significant decrease (and it's not clear whether it should, given that it reports 25 datasets for 3 groups, so hunting for decreases incurs worries about multiplicity). And I have to point out, as far as Müller et al 2012 goes, the decrease didn't reach p<0.05 (just 0.053), and if you're willing to accept just trending, then you should also be accepting the increase in the GEFT/Group Embedded Figures Task (p=0.08).

How important are these observations...? Well, as you found out, it can be hard to compare or meta-analyze psychology studies since studies may cover the same topic but use different sets of tests, frustrating the most obvious approach 'just univariate meta-analyze everything!'

Reprinted from Baranski et al. (2004) without permission.

Hah.