I don't think it's useful to talk about whether we can have confidence in statements about the outcome of an AGI process while we still disagree about whether we can have confidence in statements about the outcome of rolling a hundred-sided die.
So, OK.
Given two statements, P1 ("my next roll of this hundred-sided die will not be 12") and P2 ("my next roll of this hundred-sided die will be 12"), I consider it sensible to be confident of P1 but not P2, you don't consider it sensible to be confident of either statement. This may be because of different uses of the term "confident", or it might be something more substantive.
Would you agree that there's a 99% chance of P1 being true, and a 99% chance of P2 being false, given a fair die toss?
If so, can you say more about the class of statements like P1, where I estimate a 99% chance of it being true but it's inappropriate for me to be confident in it?
while we still disagree about whether we can have confidence in statements about the outcome of rolling a hundred-sided die.
Ok. I'll attempt to illustrate confidence vs probability as I understand it.
Lets start with your example. Starting with the certain knowledge that there is an object which is a 100-sided die, you are correct to infer that P(roll(D) != 12 | D=100) = 99/100.
Further, you are correct (in this example) to have complete confidence in that estimate.
We can think of confidence as how closely one's probability estimate approaches the true...
It's just occurred to me that, giving all the cheerful risk stuff I work with, one of the most optimistic things people could say to me would be:
"You've wasted your life. Nothing of what you've done is relevant or useful."
That would make me very happy. Of course, that only works if it's credible.