Yes. I think the skeptics don't take self-improving AI very seriously. Self-modifying programs in general are too hard to engineer, except in very narrow specialized way. A self-modifying program that rapidly achieves across-the-board superhuman ability seems like a fairy tale, not a serious engineering concern.

If there were an example of a program that self-improves in any nontrivial way at all, people might take this concern more seriously.