Nitpick:

people are less annoyed at the person who leaves early than the person who leaves late,

Do you mean comes late?

Aside from that, Viliam Bur has a point, and also people who leave early generally have unavoidable conflicts and are seen as busy instead of lazy/irresponsible. Also, people who come late keep the others waiting, wondering if they should start without them or give them another minute.