To put it another way, there is no average drift towards larger brains, but individual lineages do have trajectories up or down. Encephalization is one adaptation suite out of many possible ones, and depends on context.
And even if there were no unusual evolutionary pressures on individual lineages toward greater intelligence - then it would be a random walk, starting from zero. The expected distance from the origin still grows over time.
"The Planet-of-the-Apes Hypothesis" Revisited --Will Intelligence be a Constant in the Universe?