Yeah, it's a fuzzy measure of magnitude. I was trying to quantify why a stereotype can be wrong (in addition to just bothering some people) and I think that what makes stereotypes actually incorrect is the human tendency to approximate "most" by "all."
In Kahneman and Tversky's prospect theory, there is evidence that people do not react to the differences between small probabilities. It's conceivable that sometimes people treat a small minority as though it were a tiny minority, virtually non-existent. (On the other hand, there's some evidence that people overestimate small probabilities, which makes this argument weaker. So I take it back.)
The other way stereotyping can be a mistake has to do with the conjunction fallacy. Perhaps most X's are A, most X's are B, and most X's are C. It does not follow that most X's are A and B and C. Something that is A and B and C is a "most representative" element, but most X's are not "representative."
This is the old platitude that "there is no typical student at our school." It would be truer to say that the most typical students are usually rare. But people will assume that a student from that school is like that rare "typical student." This is a form of stereotyping which is actually inaccurate. (As opposed to "offensive but accurate," which is what many people claim stereotypes to be.)
A third type of inaccurate stereotyping is mistaking P(B|A) for P(A|B). Most criminals are men, but most men are not criminals.
These are all good points. Given that 20-30% of scientists are women, it's misleading to say "scientists are normally men" without quantifying "normally." And though most users of this website are americans, and men, and hetero, and college-educated, possibly it is not normal for them to be all at once (I could have picked better examples). But I don't like the idea of people scoring less-parochial-than-thou points off of each other through trivial mistakes along these lines. Maybe that doesn't happen.
...A third type of inaccurate st
During discussion in my previous post, when we touched the subject of human statistical majorities, I had a side-thought. If taking the Less Wrong audience as an example, the statistics say that any given participant is strongly likely to be white, male, atheist, and well, just going by general human statistics, probably heterosexual.
But in my actual interaction, I've taken as a rule not to make any assumptions about the other person. Does it mean, I thought, that I reset my prior probabilities, and consciously choose to discard information? Not relying on implicit assumptions seems the socially right thing to do, I thought; but is it rational?
When I discussed it on IRC, this quote by sh struck me as insightful:
I came up with the following payoff matrix:
In this case, the second option is strictly preferable. In other words, I don't discard the information, but the repercussions to our social interaction in case of an incorrect guess outweigh the benefit from guessing correctly. And it also matters whether either Alice or Bob is an Asker or a Guesser.
One consequence I can think of is that with a sufficiently low p, or if Bob wouldn't be particularly offended by Alice's incorrect guess, taking the guess would be preferable. Now I wonder if we do that a lot in daily life with issues we don't consider controversial ("hmm, are you from my country/state too?"), and if all the "you're overreacting/too sensitive" complaints come from Alice incorrectly assessing a too low-by-absolute-value negative payoff in (0, 1).