Jumping to the conclusion that its fixated on him and won't just attack some other child is really arbitrary.

Maybe this is just my narrative epistemic advantage talking, but: the centaur mentioned stuff relevant to the prophecy, knew specifically who Harry was, and probably was trying to kill him for a very specific reason.

Even if Harry doesn't realize what the centaur is talking about, it seems to me that of the people who try to attack and kill Harry Potter, probably most of them are actually trying to kill him specifically and are not just random psychopaths.