Thanks for the link. It sounds like Yudkowsky is arguing something quite close to Cromwell's Rule, with a slight technical difference. From the Wikipedia article:

...the use of prior probabilities of 0 or 1 should be avoided, except when applied to statements that are logically true or false.

Yudkowsky would argue that formal logic is not part of the territory, but rather part of our map (perhaps surveying equipment would be a good analogy, since the compass analogy is already taken by "moral compass"). As such, not even formal mathematical logic should be presumed to have 100% certainty.

Of course, this raises the problem of constantly having to include the term p(math is fundamentally flawed) everywhere. instead of just writing p(heads) when calculating the odds of a coin flip or flips, now we'd have to use p(heads | ~math is fundamentally flawed). As a matter of sheer convenience, it would be easier to just add it to the list of axioms supporting the fundamental theorems that the rest of mathematics is built on.

But that’s just semantics, I suppose. Wikipedia has a couple more interesting tidbits, that I’ve fished out for future readers:

The reference is to Oliver Cromwell. Cromwell wrote to the synod of the Church of Scotland on August 5 1650, including a phrase that has become well known and frequently quoted:

“I beseech you, in the bowels of Christ, think it possible that you may be mistaken.”

As Lindley puts it, assigning a probability should "leave a little probability for the moon being made of green cheese; it can be as small as 1 in a million, but have it there since otherwise an army of astronauts returning with samples of the said cheese will leave you unmoved." Similarly, in assessing the likelihood that tossing a coin will result in either a head or a tail facing upwards, there is a possibility, albeit remote, that the coin will land on its edge and remain in that position.