I've seen a few variations on "why does Lucius prefer vengeance to maintaining/expanding his political power?" First of all, for someone like Lucius, what's the point of power if you can't get stuff you want, like vengeance for your family members? Like how it's pointless to save if you're never going to spend.

And second, something that's on my mind because I just finished The Origins of Virtue by Matt Ridley. Emotions like vengeance evolved for a reason. Harry sees it as a weakness of Lucius's that he would give up anything politically to avenge his son. And it's true, that can be manipulated by people who want to see him lose his political power. However, it is also a powerful deterrent to his enemies. If Lucius is publically known as someone who would stop at nothing to avenge his family, then those that would provoke him will be few and far between, and he will probably not have to make the sacrifices that his vengefulness commits him to.

When he accused Dumbledore in public of the murder of his wife, even at political cost, I'm sure everyone got the message: "if I hurt Malfoy's family, even if I make sure it would be very politically costly for him to pursue me for it, he will pursue me for it anyway." And this was a protection to his family.

Malfoy never got his vengeance for Narcissa, but Dumbledore is the most powerful wizard alive; everyone knows that they could not get away with what Dumbledore did (I'm describing this from Lucius's perspective; I'm not committing to a judgment on Dumbledore's guilt). But if he let Hermione Granger get away with an attack on his family, that's a different thing entirely. If Hermione Granger, a child and, by birth, a second-class citizen in wizard society, can get away with an attack on Malfoy's family because of all the benefits you pointed out to taking Harry's deal, then maybe I can get away with an attack on Malfoy's family because of something like that. Lucius is making sure that the class of people who can get away with attacking his family is very, very small; whatever it costs him is an investment in a reputation as someone not to be messed with.

I don't get how a cunning Slytherin like him can discard such an advantage for pure vengeance over an "attempt" which didn't do any real harm.

On the contrary:

And Draco was getting angry again. "Dumbledore killed Mother, it's not enough to just say it's sad! I don't understand what you think you have to do, but the Malfoys have to take revenge!" Not avenging the deaths of family went beyond weakness, beyond dishonor, you might as well not exist.

Lucius never meant for Harry to accept the monetary bargain. This is clear from his reaction. He wanted his revenge, and Harry was interfering.

If Harry is Voldemort in hiding, and Harrymort needs Hermione for something, Lucius should absolutely get out of the deal. Lucius is not a supporter of Voldemort per se, he allied himself with Voldemort because he thought he could get an advantage out of it. Possibly he regretted it deeply when it cost him his wife. And this new Voldemort seems likely to cost him his son? Lucius should do what he can to destroy him while he's weak and ensure he never becomes strong.

I'm wondering why Lucius tried to get out of the deal.

I see it as a status game. Previously Lucius had suggested that Dumbledore take Hermione's place in Azkaban, not because Lucius thought there was much of a chance that Dumbledore accept this offer, but rather to strike at the idea of Dumbledore's supposed heroic altruism.

Now when Lucius named 100,000 galleons as the price to erase the debt, he was trying a similar tactic -- Lucius didn't expect that Harry accept this, Lucius wanted to strike at the idea of his enemies being heroic and altruistic.

But Harry accepts - which in terms of impressiveness is a blow in Harry's favor and against Lucius, despite the fact that Harry's now endebted to Lucius, and because the sum named is so outrageously large.

But then Lucius thought he saw a opportunity to strike back at Dumbledore and Harry, because he thought they were pretending a play to raise Harry's status even further (not just heroically altruistic, but super-heroically tough, able to destroy Azkaban, all by himself).

So he attempted to call Harry and Dumbledore's bluff. And it failed again.

Someone tried to murder his son, and all he got for it was a bit of cash. As if he gives a damn about cash. I can't imagine anyone, rich or poor, who'd say "A fat stack of bills and clearing an old favour, in exchange for attempting to murder my child? Sure, sounds fair!". I'd be throwing around terms like "sociopath" if anyone was actually okay with that(whether or not it's the proper psychological term).