I don't think it's a question of what gets stored so much as what gets activated.
That is, if I have three nodes that "represent" inferring A from G when A is more common in G than in the general population (N1), inferring A from G when A is more common than -A in G (N2), and the word "stereotyping" (N3), and my N1->N3 and N2->N3 links are stronger than N1 and N2's links to any other word, and the N3->N1 link is much stronger than the N3->N2 link, then lexical operations are going to make this sort of mistake... I might start out thinking about N2, decide to talk about it, therefore use the word "stereotyping," which in turn strongly activates N1, which displaces N2.
This is why having distinct words for minor variations in meaning can be awfully useful, sometimes. I'm willing to bet that if we agreed to use different words for N1 and N2, and we had enough conversations about stereotyping to reinforce that agreement, we'd find this error far less tempting, easier to notice, and easier to correct.
During discussion in my previous post, when we touched the subject of human statistical majorities, I had a side-thought. If taking the Less Wrong audience as an example, the statistics say that any given participant is strongly likely to be white, male, atheist, and well, just going by general human statistics, probably heterosexual.
But in my actual interaction, I've taken as a rule not to make any assumptions about the other person. Does it mean, I thought, that I reset my prior probabilities, and consciously choose to discard information? Not relying on implicit assumptions seems the socially right thing to do, I thought; but is it rational?
When I discussed it on IRC, this quote by sh struck me as insightful:
I came up with the following payoff matrix:
In this case, the second option is strictly preferable. In other words, I don't discard the information, but the repercussions to our social interaction in case of an incorrect guess outweigh the benefit from guessing correctly. And it also matters whether either Alice or Bob is an Asker or a Guesser.
One consequence I can think of is that with a sufficiently low p, or if Bob wouldn't be particularly offended by Alice's incorrect guess, taking the guess would be preferable. Now I wonder if we do that a lot in daily life with issues we don't consider controversial ("hmm, are you from my country/state too?"), and if all the "you're overreacting/too sensitive" complaints come from Alice incorrectly assessing a too low-by-absolute-value negative payoff in (0, 1).