But then this can't be the sole explanation, because people are most likely to act when they're alone and there's nobody else around to know about your duty.

Actually I think concerns about plausible deniability explain that just fine. Someone who observes trouble while alone is in a fairly safe position, since they can withdraw at any point during investigation or intervention without any witnesses to accuse them of cowardice. That initial sense of safety motivates a greater degree of risk-taking, in the form of willingness to render assistance.