If I were Quirrell, and I wanted Hermione out of Hogwarts, and Dumbledore has warded her against magic, and I failed to convince her to leave, what would I try next?

I would identify those people who have the most influence over her, and attempt to convince them to convince her to leave. Who have we seen to have influence over her? By "influence", I mean that she respects them or might for some reason listen to them. Harry, Dumbledore, McGonagall, Flitwick, Mandy, her parents.

Quirrell likely won't be able to (or won't attempt to) talk Dumbledore, McGonagall, or Flitwick into persuading Hermione to leave. He can put pressure on Harry. Putting pressure on Mandy (either with mind magic or just psychology) might also be effective. Some interrogation techniques involve prolonged deprivation followed by small kindnesses. If everyone hates Hermione, a single friendly face could persuade her to do what she otherwise might not.

He could arrange for Hermione's parents to learn of the events. As McGonagall points out in ch 84, and as Hermione later thinks to herself during her chat with Quirrell, "Mum would want her to RUN AWAY and her father would have a heart attack if he even knew she was being faced with the question."

What other avenues does Quirrell have, besides persuasion? "Hostile magic" and a "spirit [touching]" her would be detected. Can he slip her a potion? Attack her physically? Use non-hostile magic, whatever that might be? Convince her to hex herself? Use hostile magic on someone else force them to attack her physically?