where when you're not on caffeine you're below baseline and caffeine just brings you back to normal

This is a hypothesized explanation for the acute performance-enhancing effects of caffeine that fits well with the Algernon argument, but it is not a conclusive result of the literature. For instance, the following recent review disputes that.

Einöther SJL, Giesbrecht T (2013). Caffeine as an attention enhancer: reviewing existing assumptions. Psychopharmacology, 225:251–74.

Abstract (emphasis mine):

Rationale: Despite the large number of studies on the behavioural effects of caffeine, an unequivocal conclusion had not been reached. In this review, we seek to disentangle a number of questions.

Objective: Whereas there is a general consensus that caffeine can improve performance on simple tasks, it is not clear whether complex tasks are also affected, or if caffeine affects performance of the three attention networks (alerting, orienting and executive control). Other questions being raised in this review are whether effects are more pronounced for higher levels of caffeine, are influenced by habitual caffeine use and whether there [sic] effects are due to withdrawal reversal.

Method: Literature review of double-blind placebo controlled studies that assessed acute effects of caffeine on attention tasks in healthy adult volunteers.

Results: Caffeine improves performance on simple and complex attention tasks, and affects the alerting, and executive control networks. Furthermore, there is inconclusive evidence on dose-related performance effects of caffeine, or the influence of habitual caffeine consumption on the performance effects of caffeine. Finally, caffeine’s effects cannot be attributed to withdrawal reversal.

Conclusions: Evidence shows that caffeine has clear beneficial effects on attention, and that the effects are even more widespread than previously assumed.

The authors' conclusions:

• Caffeine improves performance on both simple and complex attention tasks.
• Caffeine improves alerting, executive control and potentially also orienting.
• There is inconclusive evidence on dose-related performance effects of caffeine.
• There is inconclusive evidence on the influence of habitual caffeine consumption on the performance effects of caffeine.
• Caffeine’s effects cannot be attributed to withdrawal reversal.

Note the following conflict of interest:

The authors are employees of Unilever, which markets tea and tea-based beverages.