The less you know less about someone's personality, the more you should infer about their personality from their behavior. So it is reasonable to infer more from behavior about others than yourself. The problem instead seems to be overconfidence - we infer far more than is reasonable given only a small behavior sample.

You know, I agree that anyone who gives a different answer from "press the green button" is overcomplicating the question. Our chief point of disagreement for years has been that it seems to me that your real life answer has long been "press the green button if I can do so without 'being a jerk' e.g. 'stealing the future'". That, it seems to me, is clearly the wrong answer.

There may be very good time tested rules telling you not to steal things, and even better though less tested rules telling you not to steal the One Ring, but maybe Gollum has it and he just plain isn't very likely to be convinced to take it to Mt. Doom on his own, even if he knows more, thinks faster, and is more the person he wants to be.

Everyone:

Since I spend all day thinking about my job, a lot of my best analogies, metaphors, and examples tend to involve Singularity/transhumanism. But the actual topic of this blog is cognitive bias and rationality. If you want to talk about transhumanism, take it to a transhumanist blog or mailing list.

This is exactly what I mean, there are strong cognitive biases underlying the singularitarian ideas. . .

I'm not sure what he means much of the time, but Kevembuangga hits this particular ball out of the park. Perhaps someone will write up a disagreement case study about the "Singularity" and post it here. That would be quite the treat. I'm already working on a different disagreement case study that will be posted to my own blog in the relatively near future. Cool concept, these disagreement case studies. . .

Matthew, I agree. The flip side of Hansen's recent post on freethinkers, is that we as inhabitants of a system with undiscriminating free thinkers in it would be rational not to reject their innovative good ideas simply because they're paired with a bunch of aesthetically off-putting contrarian ideas. I'm positing Kevembuangga to be such a free thinker in relation to many overcomingbias contributors.

Nick, I don't think we should all intrinsically care about saving the world. I think you, me, and whoever would socially contract with us and could add value should care about saving ourselves. Since we can't currently survive without the world (the Earth, Sun, and moon in their current general states) we need to conserve it to the degree that we need it to survive. Going beyond that in my opinion is bias, arbitrary aesthetics, irrational, or some combination of the three, and could problematically interfere with our mutual persistence.

Selfishness is at best no more rationally justifiable than altruism. (Why do so few rationalists see this?) My world-centered goals are at worst no more arbitrary than your self-centered ones. In fact, altruism may even be more reasonable, on grounds of symmetry and the fact that 'the self' is an illusion.

Nick, this is great, we have an interesting agreement. :) We may want to discuss this by email so we don't take over the thread, although I think it would be great if overcomingbias incorporate regular open threads and a sister message board. I don't care whether or not selfishness is more rationally justifiable than altruism or not. In fact, I'm not even sure what that means because the first principles behind that statement don't seem clear to me. Unless your point is that all first principles are arbitrary. I look at it from the perspective that I enjoy (apparently) existing as a subjective conscious entity, and I want to persist existing as a subjective conscious entity -forever, and in a real time sort of way. I think that defines me as an egoist (a classic egoist sentence in itself?). As a consequentialist, altruists only bother me to the extent that they may adversely impact my odds of persistence by engaging in their altruistic behavior, more rationally justifiable or not. To the extent that they positively impact -or even better, optimize- my odds of persistence, they're a phenomenon that I want to encourage. You live in a universe with me in it, Nick. And you seem to me to be a bright person. So, given that you seem to want us both to do what's most rationally justifiable, and I want us to do what will maximize my personal odds of persistence, I'm hoping there's some common ground we can meet, that will in the process MMPOOP (maximize my personal odds of persistence) -please pardon the unsavory acronym.

In fact, altruism may even be more reasonable, on grounds of symmetry and the fact that 'the self' is an illusion.

I think Richard Dawkins is on the right track with his idea of "memes". If the Buddha were alive today, I suspect he would call the self, and self-centered thinking a particularly prevalent and virulent meme infesting our cognitive facilities. And amazing but true, it is quite possible to visualize the operation of the "self" in its meme-hood and cease to identify with it, as even materialistic atheists like Susan Blackmore and Sam Harris can attest.

I look at it from the perspective that I enjoy (apparently) existing as a subjective conscious entity, and I want to persist existing as a subjective conscious entity -forever, and in a real time sort of way.

A persistent inquiry into the nature of the "I" apparently making those statements will start the Ourobouros eating its own tail and lead to the end of the "optical delusion of consciousness", as Einstein put it. In the end, reality trumps illusion. . .

Matthew, I'm not sure I completely understand your last statement, but it hasn't altered my my belief "that I enjoy (apparently) existing as a subjective conscious entity, and I want to persist existing as a subjective conscious entity -forever, and in a real time sort of way." I won't object if you decide to end your life and donate your current possessions and wealth to the charitable organization of your choice (UNICEF, Gates Foundation, Soros Foundation, or something else). But if you decide to persist as an interactive personality in the world with me, it's going to seem to me like you're an egoist yourself, and that you're just not being as transparent about it as I am (although admittedly I would only be this transparent about it anonymously, because of the -irrational in my opinion- social costs that many people seem to want to burden transparent egoists with.

I'll check out your link but a more detailed explanation from you of that last sentence would probably be welcome, too.

ps. I think there is some irony in naming people as being notable for having ceased to identified with self.

Matthew, Well, I checked out the link on Ourobouros and it didn't spark any great epiphany or change my mind about wanting to MMPOOP first and foremost. That doesn't make me opposed to other people being altruistic, but I do think that goal should be subordinated to MMPOOP. However, I'm willing to compromise on policy -if that's what's necessary to ... MMPOOP.

TGGP, Sam Harris doesn't believe in God, and I think that's the definition of an atheist. One need not shun all experiences associated with religion to qualify.

HA, Matthew and I are referring to the fact pointed out by, for instance, David Hume or Buddhism that what appears to be a unitary, unchanging essence-of-Nick-Tarleton-ness (or whoever-ness) is an illusion; all that really exists is a collection of perceptions and memories loosely bound together in the same brain; other people differ from me only in having different experiences and embodiments, not in having some distinct essence; Nick-Tarleton-fifty-years-in-the-future may have collected so many different experiences as to be as far from Nick-Tarleton-now as Hopefully-Anonymous-now is; and consequently, it seems more reasonable to serve sentient-beings-as-a-whole than this illusion of an essential self. Someone else can probably explain it better than me.

Actually hopefully, I don't think that one can be quite so transparent as you are about egoism and remain anonymous just by using a pseudonym, at least to those who live in NYC. How many people talk about MMPOOP?

Do you not understand what Matthew C and Nick T are saying, or do you just disagree.

HA,

The Ouroboros is simply a symbol.

The symbol represents the self consuming itself, which is a good description of the process that happens once "you" start investigating the nature of "you" seriously. That's what Nick and I are referring to, although I suspect Nick conceptually reduces it all to brain states, while I see brain states and personal egos as phenomena playing out within the fundamental unity of Awareness.

Nick did a very nice job explaining why seeing the reality of the "self" explodes egotism.

Michael, I think I understand what Nick and Matthew are saying, but if I don't I hope they or you jump in with a barrier-aesthetic/hide-the-ball denuded explanation. I think they're claiming something like onesself is always changing, or that it's arbitrarily defined where one's self ends and other phenomena in apparently reality begins, or that any concept of self becomes absurdly messy under sustained scrutiny. That's all fine and dandy as far as analysis and descriptions go, but I'm a bit skeptical that they're right, since as best I can tell the analysis has been done by a couple of people with 3 pound primate brains in a rather enormous and complex apparent reality. If they want to end their lives tonight and bequest all their personal wealth to me (I'll come out of anonymity for that), I'll accept that as their decision, and give it a good college try to have their "selves" live on through a "shared awareness" that exists between my ears. But as for me, I'll still be trying to MMPOOP, rather conservatively, in something closer to its present form of organization. I understand my odds of success may be vanishingly low, but I'm happy to collaborate with similarly inclined folks on this blog or elsewhere.

To understand why people act the way they do, we must first realize that everyone sees themselves as behaving normally. Don't ask what strange, mutant disposition they were born with, which directly corresponds to their surface behavior. Rather, ask what situations people see themselves as being in. Yes, people do have dispositions - but there are not enough heritable quirks of disposition to directly account for all the surface behaviors you see.

Most people see themselves as perfectly normal, from the inside. Even people you hate, people who do terrible things, are not exceptional mutants. No mutations are required, alas. When you understand this, you are ready to stop being surprised by human events.

This topic is something I have noticed is easy to explain to people. They understand it; they nod their head; then they return to being surprised by human events. For some reason it never makes into their predictors.

I remember my moment of epiphany when this topic clicked into place and suddenly people were predictable. The kicking of vending machines returned expectations of, "Wow, they must be having a rough day," instead of, "Wow, they have anger issues."

The next step in this process is learning that someone's output is different than yours. Not everyone kicks vending machines on bad days. Not everyone flashes their lights in road rage. People who are surprised by these events may have grasped the truth in what you have said above can never imagine themselves in a situation where they would kick a vending machine. When they see someone else kicking a vending machine, their internal self-predictor will never return "bad day" as a reason. The best they can come up with is, "angry person."

The point in me saying this is that acknowledging situational causes only helps when you understand the situational effects that result. Otherwise, you still get the wrong associative cause.

Another area that these predictors break down is in cultural differences. A strange example I can think of is man I knew from Africa. (I think Tanzania.) We were playing basketball and one of the American kids was constantly spitting on the ground. This utterly repulsed the African and he said, "Only pregnant women spit." This comes as a complete WTF moment for the rest of us and there was no way to compute his disgust without learning about his culture.

So, even if you begin to infer personality from a situational response, there is a small chance that whoever is kicking the vending machine is doing so because wherever he grew up people kick vending machines for good luck. It might be stupid, but so is kicking it because it ate your money.

Anecdote exemplifying the point.

My father used to appear in plays at university.

His mother attended a performance in which he played Lucky in http://en.wikipedia.org/wiki/Waiting_for_Godot (please, do tell me how to do this properly here!) . Lucky is Pozzo's slave, and is badly treated. Afterwards, she commented that the actor playing Pozzo seemed a deeply unpleasant character, and insisted she could tell, even when my father protested that he was only playing his part, and was in fact a nice chap.

6 months later, she attended another performance, in which the same actor played a very sympathetic character. After the performance, she commented on what a lovely fellow he was. On being confronted with her earlier assessment, she was deeply confused.

This reminds me in a way of that old saying, "We judge other people by their actions but ourselves by our intentions."

There is a further, independent bias in stranger-modelling [original research]. Not only do people assume persistent traits, they tend to assume the worst possible traits. For example, upon seeing a child reading a textbook at the bus stop, people will immediately assume they have a test this morning and are cramming, which stems from a natural inclination to procrastinate.

This can't be explained by correspondence bias alone. You could also assume that they're studious, and are studying for a test in two weeks; that they're anxious, and and are reassuring themselves they've studied enough for the test; that they're bookworms, and reading their textbook for fun. All of these are persistent character traits that could explain the behavior, but the least charitable explanation leaps to mind. Seriously, did you just pass negative judgement on someone you've seen for two seconds as you drove by? What a horrible person you must be!

To me, this is one of the most fundamental posts on LessWrong that has provided the greatest change to my thought processes. The vending-machine example is clear and comes to mind often.

THIS has given me a lot to think about.

A beautiful example from Clay Shirky's 2010 Cognitive Surplus, which he even identifies as fundamental attribution bias:

Napster acquired tens of millions of users in less than two years, making it the fastest-growing piece of software of its day. Its astounding success surely said something about the culture, and two conflicting interpretations were advanced in the early 2000s. The first was that young people had all become morally corrupt, willing to flout the sacred conventions of intellectual property. The second was that young people were so imbued with the spirit of sharing that they were happy to engage in the communal opportunity that Napster offered. The first explanation purported to explain why young people were so willing to take, the other why they were so willing to give. Both explanations couldn’t possibly be correct. In fact, neither of them was correct.

One of the weakest notions in the entire pop culture canon is that of innate generational difference, the idea that today’s thirtysomethings are members of a class of people called Generation X while twentysomethings are part of Generation Y, and that both differ innately from each other and from the baby boomers. The conceptual appeal of these labels is enormous, but the idea’s explanatory value is almost worthless, a kind of astrology for decades instead of months.

Generations do differ, but less because people differ than because opportunities do. Human nature changes slowly but includes an incredible range of mechanisms for adapting to our surroundings. Young people born in the decades after the baby boom ended were labeled Generation X, and they began entering the work-force in real numbers in the late 1980s. Gen Xers were said to be lazy—“slackers” in the parlance of the time—who didn’t exhibit the straightforward work ethic of their predecessors. (As someone born at the tail end of the baby boom, I loved this reasoning.) Commentators wrung their hands about the slackers in our midst, further evidence that society was going to hell in a handbasket. (Remember the Gin Laws?)

Then in the early 1990s a funny thing happened: Gen Xers started founding companies, joining start-ups, and working around the clock in pursuit of new opportunities. Gen Xers weren’t slackers at all—they were entrepreneurial! How could we have gotten it so wrong?

It’s simple: we didn’t factor in the environment in which the then-twentysomethings were living. The market crash of 1987 was followed by a fitful performance in the U.S. economy that, by the early 1990s, had tipped into a full-blown recession. In a recession, taking a dead-end job and conserving costs by hanging out with friends and drinking cheap beer are perfectly sensible responses. Maybe this generation wanted to be go-getters even in the depths of the recession, but people don’t behave in ways they don’t have the opportunity to behave in. Once the recession was over, the landscape of opportunity changed dramatically: it became easier to find a well-paying job, to start a company, or to join a start-up, all activities that the former slackers dove into with gusto.

At the moment of their earliest adulthood, Gen Xers were entering an economy that was inimical to ambition, and they behaved accordingly. Then, fairly suddenly, the economy started rewarding ambition, and the supposedly core psychological attributes of those young people simply vanished, to be replaced by an almost opposite set of attributes. You’d think this transformation would have broken people of their faith in such generalizations, but the desire to attribute people’s behavior to innate character rather than to local context runs deep. It runs so deep, in fact, that psychologists have a name for it: the fundamental attribution error. The fundamental attribution error is at work when we explain our own behavior in terms of the constraints on us (“I didn’t stop to help the stranded driver because I was late for work”) but attribute the same behavior in others to their character (“He didn’t stop to help the stranded driver because he’s selfish”). Similarly, we fell into the fundamental attribution error when we thought Gen Xers weren’t working hard because they were lazy.

Theories of generational difference make sense if they are expressed as theories of environmental difference rather than of psychological difference. People, especially young people, will respond to incentives because they have much to gain and little to lose from experimentation. To understand why people are spending so much time and energy exploring new forms of connection, you have to overcome the fundamental attribution error and extend to other people the set of explanations that you use to describe your own behavior: you respond to new opportunities, and so does everybody else, and these changes feed on one another, amplifying some kinds of behavior and damping others.

The Intelligence website links no longer function.

Neat.

The "overcomplicating the question" link is broken and I can't find the article on that site anymore. But this looks like the same one: http://www.yudkowsky.net/singularity/simplified/

And the next link is here, I think: http://www.yudkowsky.net/singularity/ai-risk/

I have noticed that people are often quite sloppy about what questions they ask in addition to how they think about the answers.

I suspect when most people ask you why you want to save the world, what they really mean is, "Why do you devote so much effort to trying to save the world when your odds of success are so abysmally low that it may as well be considered impossible?  Don't you have more practical things to do with your time?"