If it appears to be happening "for no reason", most people will infer a much more plausible causal explanation than time-traveling punishment — for instance, that this type of hardship contributes to people becoming criminals.
Wasn't there some Twilight Zone episode about this, where a Jewish time traveller used a mind-control device to torment Hitler, which caused his anti-semitism?
Here's an edited version of a puzzle from the book "Chuck Klosterman four" by Chuck Klosterman.
When should you punish someone for a crime they will commit in the future? Discuss.