Edit: Actually, on thinking about it, I realize I'm being a doofus. You almost undoubtedly meant, not inferring A from G when A is more common in G than in the general population, but inferring A from G when A is more common than -A in G, which is a far more unreasonable thing to be upset about. My apologies.
It's very interesting that you made this mistake (and I didn't notice it until you pointed it out, and would maybe have made the same).
It seems that the human mind doesn't make a sufficiently good distinction between the two, between "blacks are more likely than non-blacks to have a criminal record" and "blacks are more likely than not to have a criminal record". Maybe by default the non-verbal part of the brain stores the simpler version (the second one), and uses that part to constrain expectations and behavior.
I don't think it's a question of what gets stored so much as what gets activated.
That is, if I have three nodes that "represent" inferring A from G when A is more common in G than in the general population (N1), inferring A from G when A is more common than -A in G (N2), and the word "stereotyping" (N3), and my N1->N3 and N2->N3 links are stronger than N1 and N2's links to any other word, and the N3->N1 link is much stronger than the N3->N2 link, then lexical operations are going to make this sort of mistake... I might sta...
During discussion in my previous post, when we touched the subject of human statistical majorities, I had a side-thought. If taking the Less Wrong audience as an example, the statistics say that any given participant is strongly likely to be white, male, atheist, and well, just going by general human statistics, probably heterosexual.
But in my actual interaction, I've taken as a rule not to make any assumptions about the other person. Does it mean, I thought, that I reset my prior probabilities, and consciously choose to discard information? Not relying on implicit assumptions seems the socially right thing to do, I thought; but is it rational?
When I discussed it on IRC, this quote by sh struck me as insightful:
I came up with the following payoff matrix:
In this case, the second option is strictly preferable. In other words, I don't discard the information, but the repercussions to our social interaction in case of an incorrect guess outweigh the benefit from guessing correctly. And it also matters whether either Alice or Bob is an Asker or a Guesser.
One consequence I can think of is that with a sufficiently low p, or if Bob wouldn't be particularly offended by Alice's incorrect guess, taking the guess would be preferable. Now I wonder if we do that a lot in daily life with issues we don't consider controversial ("hmm, are you from my country/state too?"), and if all the "you're overreacting/too sensitive" complaints come from Alice incorrectly assessing a too low-by-absolute-value negative payoff in (0, 1).