Doesn't circuit design (and therefore computer processor design) require fairly large computational resources (for mathematical modelling)? Thus faster hardware now can be used to create even faster hardware.. faster.
There is some positive feedback in circuit design (although sublinear, I think), but hardware serial speed is essentially limited by the size of the surface features on the IC, which is in turn limited by the manufacturing process and ultimately by the physical limits of CMOS technology.
Master version on gwern.net