So, I agree with this statement, but it still floors me when I think about it.

I sometimes suspect that the phrase "recursively self-improving intelligence" is self-defeating here, in terms of communicating with such people, as it raises all kinds of distracting and ultimately irrelevant issues of self-reference. The core issue has nothing to do with self-improvement or with recursion or even with intelligence (interpreted broadly), it has to do with what it means to be a sufficiently capable optimizing agent. (Yes, I do understand that optimizing agent is roughly what we mean by "intelligence" here. I suspect that this is a large inferential step for many, though.)

I mean, surely they would agree that a sufficiently capable optimizing agent is capable of writing and executing a program much like itself but without the failsafe.

Of course, you can have a failsafe against writing such a program... but a superior optimizing agent can instead (for example) assemble a distributed network of processor nodes that happens to interact in such a way as to emulate such a program, to the same effect.

And you can have a failsafe against that, too, but now you're in a Red Queen's Race. And if what you want to build is an optimizing agent that's better at solving problems than you are, then either you will fail, or you will build an agent that can bypass your failsafes. Pick one.

This just isn't that complicated. Capable problem-solving systems solve problems, even ones you would rather they didn't. Anyone who has ever trained a smart dog, raised a child, or tried to keep raccoons out of their trash realizes this pretty quickly.