For my own part, the idea that we might build tools better at algorithm-development than our own brains are doesn't seem counterintuitive at all...

So you believe that many mathematical problems are too hard for humans to solve but that humans can solve all of mathematics?

I already asked Timothy Gowers a similar question and I really don't understand how people can believe this.

In order to create an artificial mathematician it is first necssary to discover, prove and encode the mathematics of discovering and proving non-arbitrary mathematics (i.e. to encode a formalization of the natural language goal “be as good as humans at mathematics”). This seems much more difficult than solving any single problem. And that's just mathematics...

Neither does it seem implausible that there exist problems that are solvable by algorithm-development, but whose solution requires algorithms that our brains aren't good enough algorithm-developers to develop algorithms to solve.

I do not disagree with this in theory. After all, evolution is an example of this. But it was not computationally simple for evolution to do so and it did do so by a bottom-up approach, piece by piece.

So it seems reasonable enough that there are problems which we'll solve faster by developing algorithm-developers to solve them for us, than by trying to solve the problem itself.

To paraphrase your sentence: It seems reasonable that we can design an algorithm that can design algorithms that we are unable to design.

This can only be true in the sense that this algorithm-design-algorithm would run faster on other computational substrates than human brains. I agree that this is possible. But are relevant algorithms in a class for which a speed advantage would be substantial?

Again, in theory, all of this is fine. But how do you know that general algorithm design can be captured by an algorithm that a.) is simpler than most specific algorithms b.) whose execution is faster than that of evolution c.) which can locate useful algorithms within the infinite space of programs and d.) that humans will discover this algorithm?

Some people here seem to be highly confident about this. How?

ETA: Maybe this post better highlights the problems I see.