I think that the issue was the number of planets. If you had just one planet orbiting the sun, that orbit would be a nice stable one. But if you have multiple bodies orbiting the sun, their paths will interfere chaotically. I think that Newton expected that, in general, you would get wildly erratic orbits, with some planets being thrown clear of the system altogether. As I understand it, he expected such catastrophes to be inevitable, unless you started with a very carefully-selected initial state. God was then necessary to explain how the solar system started out in such an improbable state. But in fact Newton just lacked the mathematical sophistication to see that, according to his own theory, typical initial arrangements could result in systems that are stable for billions of years.