You don't need to add them ALL up at the same time, just notice that as you get further and further from the middle, each part begins canceling with nearer and nearer neighbors. To be more concrete: at some point, you start sending your pulse. The shortest path/specular reflection gets the signal there first; other paths begin contributing later. After a short time, the time offset to get to the destination is large enough that the beginning of the pulse from one angle is cancelling with the middle of the pulse from a neighboring angle. Beyond that point, unless the packet had some special structure, there's not much in the way of reflection.

To be perfectly frank, the mirror isn't necessary for this problem to work - all it really needs to do is justify Huygens' principle.

This also goes a way towards addressing DonGeddis's question - pretend the mirror isn't there, and reflect the upward rays down. The mirror no longer exists, and this now becomes the question of why light doesn't spontaneously turn angles for no reason at all. Is that better?