I understand that what you're really complaining about is that some people are overconfident in their speculations (which is a fine and good thing to complain about) but the way you've phrased that objection here is a general counterargument against pretty much any statement that doesn't fall within mathematics, including all heuristics, priors, educated guesses, and parsimony intself.
(And the literal meaning of "I know nothing about this but here's my pontification" is very similar to "I have no evidence, but here is my prior assumption". You're just rewording it so it's a low status thing.)
Might be helpful to narrow down the objection a little, to explain where precisely you feel people are commonly overreaching?
A prior is a statement of one's knowledge (or to say exactly the same thing with an antonym, a statement of one's ignorance), as expressed before performing an experiment or observation. It stands in contrast to one's posterior, the state of belief after having updated on the evidence obtained. Outside of that context, one's beliefs are not prior to anything, and talking about one's priors is just, well, rewording it so it sounds like a high status thing.
But on reconsideration, I think I'm being unfair in making that response to your post. In the flying ex...
One fallacy that I see frequently in works of popular science -- and also here on LessWrong -- is the belief that we have strong evidence of the way things were in pre-history, particularly when one is giving evidence that we can explain various aspects of our culture, psychology, or personal experience because we evolved in a certain way. Moreover, it is held implicit that because we have this 'strong evidence', it must be relevant to the topic at hand. While it is true that the environment did effect our evolution and thus the way we are today, evolution and anthropology of pre-historic societies is emphasized to a much greater extent than rational thought would indicate is appropriate.
As a matter of course, you should remember these points whenever you hear a claim about prehistory:
Results that you should have reasonable levels of confidence in should be framed in generalities, not absolutes. E.g., "The great majority of human cultures that we have observed have distinct and strong religious traditions", and not "humans evolved to have religion". It may be true that we have areas in our brain that evolved not only 'consistent with holding religion', but actually evolved 'specifically for the purpose of experiencing religion'... but it would be very hard to prove this second statement, and anyone who makes it should be highly suspect.
Perhaps more importantly, these statements are almost always a red herring. It may make you feel better that humans evolved to be violent, to fit in with the tribe, to eat meat, to be spiritual, to die at the age of thirty.... But rarely do we see these claims in a context where the stated purpose is to make you feel better. Instead they are couched in language indicating that they are making a normative statement -- that this is the way things in some way should be. (This is specifically the argumentum ad antiquitatem or appeal to tradition, and should not be confused with the historical fallacy, but it is certainly a fallacy).
It is fine to identify, for example, that your fear of flying has a evolutionary basis. However, it is foolish to therefore refuse to fly because it is unnatural, or to undertake gene therapy to correct the fear. Whether or not the explanation is valid, it is not meaningful.
Obviously, this doesn't mean that we shouldn't study evolution or the effects evolution has on behavior. However, any time you hear someone refer to this information in order to support any argument outside the fields of biology or anthropology, you should look carefully at why they are taking the time to distract you from the practical implications of the matter under discussion.