Of course it wouldn't be time travel. People who were especially good at predicting other people's life paths would just do so and punish accordingly, or something.

Edit: I accept your point that future consequences don't suffice to justify time-travelling punishment.