Is the prior that low or something?

Yes. Can you name a single fanfic that has ever won either prize for best novel? I can't. Keeping in mind that fanfics long predate the Internet, that many noted authors got started as fans or were published in fanzines (Charles Stross being the most recent example I can think of), etc. So Laplace's law gives us a starting point of single percents, and it only gets worse if we take into account nominations as well...

What should the world look like if HPMOR does have a chance of winning Best Novel?

Well, see above. Acceptable substitutes would include reviews and praise by the sort of people whose reviews & praise predict other nominees or winners - Brin is a good start, but we're talking novels that are discussed or reviewed in all the major SF organs and sell into the hundreds of thousands or millions, so the n should be into the dozens for starters.

My guess is there'd need to be more respect for fanfic among the older sf fans, but I could be wrong about that. I'm not the only person for whom HPMOR is the only fanfic I read. (I've read some other good fanfic, but I don't get around to fanfic generally speaking.)

I think a sufficiently high proportion of likely voters are on line, but this is based on a feeling of plausibility rather than actual knowledge.

When you talk about which award you're going for, is this just a matter of which award you encourage people to nominate it for?

In terms of prestige, I think you'd be much better off being nominated for Best Novel and losing than winning the Best Fan Writer award.

I find it hard to imagine you winning, but I also find it hard to imagine specific reasons that would make it extremely unlikely. It's quite possible that I'm just engaging in availability bias (all the other winners have been conventionally published) rather than anything more solid.