Try not to Kobayashi Maru this question, at least not yet. I know you can criticize the scenario and find it unrealistic. For instance, you may say you won't push because the man might fight back, and you'd both fall but not till after the trolley had passed so everyone dies. So imagine the fat man in a wheelchair, so he can be lightly rolled off the bridge. And if you're too socially constrained to consider hurting a handicapped person, maybe the five people tied to the tracks are also in wheelchairs. If you think that being pushed off a bridge is more terrifying than being hit by a train, suppose the fat man is thoroughly anesthetized. Yes, this is an unrealistic thought experiment; but please play along for now.

Just so you know, the common term for this around here is don't fight the hypothetical.