The genetic bottleneck around the time of the eruption was not as "near" as all that - in part since there were Neanderthals around at that time as an additional backup mechanism, complementing the surviving humans. Plus, of course, Homo floresiensis! ;-)

http://en.wikipedia.org/wiki/Toba_catastrophe_theory estimates we got down to the last 5,000-10,000 backup copies of the human genome.

Figures from before the eruption appear to have not been dramatically higher:

Scientists from the University of Utah in Salt Lake City in the U.S. have calculated that 1.2 million years ago, at a time when our ancestors were spreading through Africa, Europe and Asia, there were probably only around 18,500 individuals capable of breeding (and no more than 26,000).

• http://www.physorg.com/news183278038.html

There just weren't that many homos around at the time.