And for that matter, driving mammoths to extinction is no mean feat either (though it was probably not just pressure from human hunting - apparently we got help from climate changes). Herbivores have been optimized to limit their risk from predation, and for the heavier ones (which can't hope to outrun a predator) those adaptions usually involve fighting back. Them not being naturally aggressive doesn't mean it's trivial to bring one down.