Our brains are mysterious to us not simply because they're our brains and no one can fully understand themselves, but because our brains are the result of millions of years of evolutionary kludges and because they're made out of hard-to-probe meat. We are baffled by chimpanzee brains or even rabbit brains in many of the same ways as we're baffled by human brains.

Imagine an intelligent agent whose thinking machinery is designed differently from ours. It's cleanly and explicitly divided into modules. It comes with source code and comments and documentation and even, in some cases, correctness proofs. Maybe there are some mysterious black boxes; they come with labels saying "Mysterious Black Box #115. Neural network trained to do X. Empirically appears to do X reliably. Other components assume only that it does X within such-and-such parameters.". Its hardware is made out of (notionally) discrete components with precise specifications, and comes with some analysis to show that if the low-level components meet the spec then the overall function of the hardware should be as documented.

Suppose that's your brain. You might, I guess, be reluctant to experiment on it in any way in place, but you might feel quite comfortable changing EXPLICITFACTSTORAGE_SIZE from 4GB to 8GB, or reimplementing the hardware on a new semiconductor substrate you've designed that lets every component run at twice the speed while remaining within the appropriately-scaled specifications, and making a new instance. If it causes disaster, you can probably tell; if not, you've got a New Smarter You up and running.

Of course, maybe you couldn't tell if some such change caused disasters of a sufficiently subtle kind. That's a reasonable concern. But this isn't an ice-pick-through-the-eye-socket sort of concern, and it isn't the sort of concern that makes it obvious that "recursive self-improvement is not possible".