OK, I think I understand you now. Thanks.
To answer your question as I understand it: it does not make sense for me to be confident that the result of some unspecified operation performed on some unknown object will be 12.
It does make sense to be confident that it won't be 12. (I might, of course, be confident and wrong. It's just unlikely.)
I consider the latter a more apposite analogy for the argument you challenge here. Being confident that an unspecified process (e.g., AGI) won't value paperclips makes more sense than being confident that it will, in the same way that being confident that it won't return "12" makes more sense than being confident than it will.
Perhaps we are using a different notion of 'confidence'. The uses of that term that I am familiar with have a separate specific meaning apart from probability. Confidence to me implies meta-level knowledge about the potential error in one's probability estimate, intervals, or something of that nature.
So in your analogy, I can't be confident about any properties of an unspecified process. I can of course assign a low probability estimate to the proposition that this unspecified process won't value paperclips purely from priors, but that will not be a hig...
It's just occurred to me that, giving all the cheerful risk stuff I work with, one of the most optimistic things people could say to me would be:
"You've wasted your life. Nothing of what you've done is relevant or useful."
That would make me very happy. Of course, that only works if it's credible.