What does it mean to have uncertainty reduction taking place outside of the frame of reference of the person being asked for a decision?

You're assuming humans are rational(as in the AI definition of a rational agent). We're not. So this knowledge that other person knows something for sure, that we don't know about, colours/biases one's judgement.

I am not saying one should update their beliefs based on another person knowing or not knowing, but that we do anyway, as part of perception. I would argue, that we should be learning to notice the confusion between the rational side of us vs the perceptive side which notes (the other agent's) confidence/lack there of. I know it is a hand-wavy explanation, but my point stands nevertheless. I agree with the OP that one shouldn't update their beliefs on the basis of Naomi/camera having no certainty about the outcome(of coin toss). Simply say that if it is Naomi, there could be cases where it is rational to update, though hard to actually observe/be-aware of these updations and therefore, safer to not update.