To convince me otherwise, explain how an AI that assigns probabilities to arbitrary labels about which it knows nothing will function in a superior fashion to an AI that only assigns probabilities to things for which it has nonzero notion of its truth condition.

If you're thinking truly reductionistically about programming an AI, you'll realize that "probability" is nothing more than a numerical measure of the amount of information the AI has. And when the AI counts the number of bits of information it has, it has to start at some number, and that number is zero.

The point is about the internal computations of the AI, not the output on the screen. The output on the screen may very well be "ERROR: SYNTAX" rather than "50%" for large classes of human inputs. The human inputs are not what I'm talking about when I refer to unspecified hypotheses like A,B, and C. I'm talking about when, deep within its inner workings, the AI is computing a certain number associated with a string of binary digits. And if the string is empty, the associated number is 0.

The translation of

-- "What is P(A), for totally unspecified hypothesis A?"

-- "50%."

into AI-internal-speak is

-- "Okay, I'm about to feed you a binary string. What digits have I fed you so far?"

-- "Nothing yet."

"If you know nothing, 50% prior probability" still strikes me as just plain wrong.

That's because in almost all practical human uses, "know nothing" doesn't actually mean "zero information content".