We have seen him to be afraid of this one. We've seen him express intent to stop that destiny at every point of intervention. Killing Harry is a really obvious and seemingly certain point of intervention.

"All thiss, all I have done, iss to ssmassh that desstiny at every point of intervention. If ssome fate makess me fail in what comess next, idiot-child of foretold desstruction, then you musst kill yoursself to ssave girl-child. Elsse all you claim to value diess by your own hand."

He certainly seems to think that Harry's death would solve the problem, and he's willing enough to have Harry kill himself, but not willing to kill Harry. Why?