The "tool use" theory of the origin of intelligence is widely discredited.
Neanderthal man was bigger than us, and had bigger brains. Extinction is too common to mean very much. Most species trend larger - until they get reset by meteorite strikes. I'm not aware of any partiularly noteworthy growth of primates. Our ancestors have mostly grown, if that's what you mean.
Neanderthal cranial capacity is often thought to have been as large or larger than modern humans, indicating that their brain size may have been the same or greater; however, a 1993 analysis of 118 hominid crania concluded that the cranial capacity of H.s. neandertal averaged 1,412 cc (86 cu in) while that of fossil modern H.s. sapiens averaged 1,487 cc (91 cu in).[6] On average, the height of Neanderthals was comparable to contemporaneous Homo sapiens.
Neanderthal brains weren't bigger than modern human. Wikipedia - Neanderthal
We have a sample of one modern human civilization, but there are some hints on how likely it was to happen.
Major types of hints are:
Data for:
Data against:
To me it looks like life, animals with nervous systems, Upper Paleolithic-style Homo, language, and behavioral modernity were all extremely unlikely events (notice how far ago they are - vaguely ~3.5bln, ~600mln, ~3mln, ~200k or ~600k, ~50k years ago) - except perhaps language and behavioral modernity might have been linked with each other, if language was relatively late (Homo sapiens only) and behavioral modernity more gradual (and its apparent suddenness is an artifact). Once we have behavioral modernity, modern civilization seems almost inevitable. Your interpretation might vary of course, but at least now you have a lot of data to argue for your position, in convenient format.