Is the prior that low or something?

Yes. Can you name a single fanfic that has ever won either prize for best novel? I can't. Keeping in mind that fanfics long predate the Internet, that many noted authors got started as fans or were published in fanzines (Charles Stross being the most recent example I can think of), etc. So Laplace's law gives us a starting point of single percents, and it only gets worse if we take into account nominations as well...

What should the world look like if HPMOR does have a chance of winning Best Novel?

Well, see above. Acceptable substitutes would include reviews and praise by the sort of people whose reviews & praise predict other nominees or winners - Brin is a good start, but we're talking novels that are discussed or reviewed in all the major SF organs and sell into the hundreds of thousands or millions, so the n should be into the dozens for starters.