that actually requires degrees of self-reference that put you into the territory of paradox theorems.
Oh, OK, the Löbstacle thing. You're right, that's a matter of program verification and as such more in the territory of Problem One than of Problem Two.
At some point soon, I'm going to attempt to steelman the position of those who reject the AI risk thesis, to see if it can be made solid. Here, I'm just asking if people can link to the most convincing arguments they've found against AI risk.
EDIT: Thanks for all the contribution! Keep them coming...