because conduction distances grow too, and therefore you need a higher conduction speed, and this can only be achieved by having a larger neuron

That only suggests that peripheral neurons should grow with body size, not neurons in the brain. I suppose that it's simpler to let central and peripheral neurons be the same size, but that doesn't seem to me like a good reason.