Increased encephalization has evolutionary costs - high energy expenditure and (in humans) difficult birth of underdeveloped infants due to head size. Natural selection will not allow encephalization beyond a certain size unless it has offsetting benefits, which depend on the species and its environment. So it's not a random walk (=neutral genetic drift) once you get far enough from the average, like we have.