Assuming the attack takes a nontrivial amount of time (more than an hour or so) the spend will already have a few confirmations before the attacker can write their double spend, so they'll be uselessly behind.
Even if the attacker takes on average a day to crack a key that doesn't mean that he doesn't have 0.01% to crack a key in a minute.
If there a good reason to assume that the attack needs to last a minimum of an hour to produce a key?
Yes, you are talking about a search in a 2^80 keyspace, even with quantum speedups. It'll be a long, long time before anyone has the capability to o 2^80 large quantum computations in the span of 10 minutes, much longer until the cost of doing that computation will be less than the value of the coins.
So long as you are not stupid enough to reuse keys, it's a non-issue.