You're right, I misremembered. Still, I think there is a lot to suggest that interacting directly with one's time-clones (as opposed to waiting with one's eyes closed while a clone drops off a message, for example), particularly in battle, is a bad idea. You would never observe a future-you doing something ineffective in combat---failing to dodge or block, or casting an ineffective spell, for example---since, after observing that mistake, you would be prepared for it in the future. So the only consistent possibilities involve losing or running away before you can go back in time, or winning right away.

But, since you know these are the only consistent possibilities, if you showed up to a battle intending to use a Time Turner and didn't see your time clones appear at the very beginning, you would deduce that you would not win, and therefore (if you could) you would run away. So, I expect that from Time's perspective, the most stable solution for people who intend to use Time Turners in battle is for them to not do battle at all.