Sure­ly there can be such su­per-in­tel­li­gences: Imag­ine a (per­haps autis­tic) IQ-200 guy who just wants to stay in his room and play with his pa­per­clips. He doesn’t re­al­ly care about the rest of the world, he doesn’t care about ex­tend­ing his in­tel­li­gence fur­ther, and the rest of the world doesn’t quite care about his pa­per­clips. Now re­place the guy with an AI with the same val­ues: it’s quite su­per-in­tel­li­gent al­ready, but it’s still safe (in the sense that ob­jec­tive­ly it poses no threat, other than the fact that the re­sources it uses play­ing with its pa­per­clips could be used for some­thing else); I have no prob­lem scal­ing its in­tel­li­gence much fur­ther and leav­ing it just as be­nign.

Of course, once it’s su­per-in­tel­li­gent (quite a bit ear­li­er, in fact), it may be very hard or im­pos­si­ble for us to de­ter­mine that it’s safe — but then again, the same is true for hu­mans, and quite a few of the bil­lions of ex­ist­ing and past hu­mans are or have been very dan­ger­ous.

The dif­fer­ence be­tween “X can’t be safe” and “X can’t be de­ter­mined to be safe” is im­por­tant; the first means “prob­a­bil­i­ty we live, given X, is zero”, and the other means “prob­a­bil­i­ty we live, given X, is strictly less than one”.