I’ve also observed that people who come to believe that there are significant differences between the sexes/races/whatevers on average begin to discriminate against all individuals of the disadvantaged sex/race/whatever, even when they were only persuaded by scientific results they believed to be accurate and were reluctant to accept that conclusion. I have watched this happen to smart people more than once. Furthermore, I have never met (or read the writings of) any person who believed in fundamental differences between the whatevers and who was not also to some degree a bigot.

This is something I haven't observed, but it's seemed plausible to me anyway. Have there been any studies (even small, lightweight studies with hypothetical trait differences) showing that sort of overshoot? If there are, why don't they get the sort of publicity that studies which show differences get?

Speaking of AIs getting out of the box, it's conceivable to me that an AI could talk its way out. It's a lot less plausible that an AI could get it right the first time.

And here's a thought which may or may not be dangerous, but which spooked the hell out of me when I first realized it.

Different groups have different emotional tones, and these kept pretty stable with social pressure. Part of the social pressure is usually claims that the particular tone is superior to the alternatives (nicer, more honest, more fun, more dignified, etc.). The shocker was when I realized that the emotional tone is almost certainly the result of what a few high-status members of a group prefer or preferred, but the emotional tone is generally defended as though it's morally superior. This is true even in troll groups, who claim that emotional toughness is more valuable than anything which can be gained by not being insulting.