Good point about the higher pressure to conserve the nutrients. Essentially, it's a matter of relative selection for whatever aspects of intelligence rely on brain size, versus the selection to conserve nutrients.

With regards to being pushed by nutrition... The population can be expected to quickly rise to equilibrium level where the nutrition limits the population. At equilibrium level, the hunters would be bigger and stronger (and perhaps brainier) than farmers, because the farmer that starved this much will survive and the hunter won't. All while both societies have their population capped by available nutrients.