Hrm? Suppose you're trying to accomplish some problem X. There are range of algorithms and heuristics available to you. You try a few of them. At some point -- usually a very quick point -- one of them is good enough for your practical purpose, and you stop.

We don't typically go too far in formalizing our purposes, generally. But I don't see what the deep point is that you're driving at. For practical purposes, algorithms are chosen by people in order to solve practical problems. Usually there are a few layers of formalized intermediaries -- compilers and libraries and suchlike. But not very far down the regress, there's a human. And humans settle for good enough. And they don't have a formal model of how they do so.

There isn't an infinite algorithmic regress. The particular process humans use to choose algorithms is unquestionably not a clean formal algorithm. Nobody ever said it was. The regress stops when you come to a human, who was never designed and isn't an algorithm-choosing algorithm. But that doesn't shed any light on whether a formal algorithm exists that could act similarly to a human, or whether there is an algorithm-choice procedure that's as good or better than a human.

Better algorithms maybe are known to exist, but these itself can't ultimately be selected by an algorithm, as you would have an infinite regress of picking algorithms then.

This is fallacious.

Correct conclusion: you would then have a 1-step regress of picking algorithms.

Watch that slippery slope.