EDIT: I meant to say "Yes more or less, but the closer mapping of that analogy is . .."
No, but the closer mapping of that analogy is: does it make sense to be confident that the "roll" of an unknown object will be 12 when you don't even know that it's a die?
OK, I think I understand you now. Thanks.
To answer your question as I understand it: it does not make sense for me to be confident that the result of some unspecified operation performed on some unknown object will be 12.
It does make sense to be confident that it won't be 12. (I might, of course, be confident and wrong. It's just unlikely.)
I consider the latter a more apposite analogy for the argument you challenge here. Being confident that an unspecified process (e.g., AGI) won't value paperclips makes more sense than being confident that it will, in...
It's just occurred to me that, giving all the cheerful risk stuff I work with, one of the most optimistic things people could say to me would be:
"You've wasted your life. Nothing of what you've done is relevant or useful."
That would make me very happy. Of course, that only works if it's credible.