I was thinking about turning this chunk of vomited IM text I wrote for a non-LW cogsci-student friend into a post (with lots of links and references and stuff). Thoughts? I'll probably repost this for the new open thread so that more people see it and can give me feedback.

People make this identity for themselves... no, it's not that they make it. To some extent it's chosen, but a lot of it is sorta random stuff from the environment. Ya know, man is infinitely malleable, whatever. Like, if you happen to give money to a bum on the street, you'll start thinking of yourself as the type of person who gives money to poor people, constructing this internal narrative about who you are. Except a lot of the time it's not even conscious, you just do things and think they're representative of the type of person you are. And if you're that type of person, you do more of that type of a thing. So there's a chance for cascades, especially if there's in-group/out-group bias. 'Cuz if you start leaning towards Green over Blue, then you have more positive affect around Green, which polarizes you, leading to more positive affect, et cetera. So there are all these attractors in identity space that we unkowingly get sucked into, and they can cause bias or good habits or bad habits.

But the thing is, rationality is precise, and if you're being sucked into all these attractors unknowingly then the chance that you're being sucked into the attractors and identity that is best for finding truth or doing the best thing according to your preferences is pretty slim. Signaling comes into play via something I noticed a few months ago: you think a lot about the things you wish to signal, and you signal things that pertain to your identity. So someone like me has the identity of 'rationalist' and 'smart person' and 'person who wants to be good at everything' and 'person with strong sense of aesthetics', and so I end up wanting to signal those things. I'll try to be rational, because of consistency effects. (Cialdini has an excellent book reviewing lots of psychology literature about consistency effects and how, if you act a certain way, you'll try to act that way in the future to be consistent.) I'll try to get good at programming, because part of my identity has me wanting to be really good at everything, and programming is this big thing it'd be cool to be good at. And because I want to signal these things so that I and others can see them, I think about these things too, a lot of the time. My internal monologue is primed by thoughts about becoming more awesome or being rational.

To some extent that's good, but if I'm not careful about my identity, it could be bad. I could be biased by my identity in various ways. For instance, an easy way for this to happen is to identify with a political party or a religion or philosophical position. You support arguments on 'your side' and try to shoot down arguments on the 'opposing side', even when there's no reason other than identity to want facts to have been a certain way or another. From a Bayesian perspective, it doesn't even make any sense; it's an impossible error of probability theory that only humans could do, because we're crazy. So we have to be very careful and very cognizant of our identity, and thus what we wish to signal, and thus our thoughts, and because the brain is so leaky, we have to watch for potential for causation and feedback loops due to correlation in those three facets of thoughtspace, actionspace, and personalityspace.

Edit: Also, goal distortion.