Being known as "the kid who walks expressionlessly up to other kids and punches them in the testicles without warning" may be a significant penalty too.

I was more referring to that time period when the bully was working himself with bluster right in the non-victim's face. :)