Hmm. If it's been going on for 20,000 years, then we have groups of humans living in something like ancestral conditions who've been isolated that long. The Sentinelese come to mind (~30,000 years of isolation from the rest of humanity). I can't offhand think of any ethical way to check this, though. Some isolated peoples in New Guinea might also qualify, but a surprising number of them are farmers and have been for a long time. Regardless, if you checked the average brain volume of people from this group (who haven't been effected by agriculture, and may not have the technology to make fire, although it's also conceivable that such a thing could be lost and then regained later) you'd possibly have some idea of whether or not lifestyle changes in the last 10,000 years were a primary factor.