On the other hand, if I take "he's only three minutes late" as evidence that he did it, and "he's more than three minutes late" as evidence that he did it, I'm violating a principle of rationality.

If you take "he's just a few minutes late" as strong evidence that he did it, "he's quite a while late" as weak evidence that he did it and "he's early" as very strong evidence that he did not do it, this violation disappears.