In make an extraordinary effort (and in other posts), Eliezer talks about how uncommon goals require uncommon means. He also notes that people seem incredibly unwilling to consider these unusual routes:

Even so, I think that we could do with more appreciation of the virtue "make an extraordinary effort".  I've lost count of how many people have said to me something like:  "It's futile to work on Friendly AI, because the first AIs will be built by powerful corporations and they will only care about maximizing profits."  "It's futile to work on Friendly AI, the first AIs will be built by the military as weapons."  And I'm standing there thinking:  Does it even occur to them that this might be a time to try for something other than the default outcome?  They and I have different basic assumptions about how this whole AI thing works, to be sure; but if I believed what they believed, I wouldn't be shrugging and going on my way.

I once talked to someone from China, who said that starting an organization like MIRI would be inconceivable there, because of the immense pressure to achieve traditional success. It's possible (this person continued) to start a charity, or to become a researcher, or to do other things which on the surface appear to be going after something other than money/status/power. But in some very solid way, these are just different ways to go about getting money/status/power (in a way that's not true in the West). If everyone around you is so hooked into the system, it's hard to really have thoughts that go in any other direction. Your peers and your parents will be on your case to justify your career moves.

I note that starting an organization like MIRI seems pretty inconceivable in America, too, and really what it takes is for a committed group of people to make an effort. But I think the point stands -- there's probably a degree of difference between America and China in this respect. For one thing, America has a start-up culture which is not present in a lot of other places. And from what I've heard, Chinese parents exert a lot more pressure on their children's life choices (career, spouse, etc) than American parents, which would have the claimed effect.

This reminds me of a quote from Zvi's summary of Moral Mazes:

[T]his is the result of a vicious cycle arising from competitive pressures among those competing for their own organizational advancement. Over time, those who focus more on and more value such competitions win them, gain power and further spread their values, unless they are actively and continuously opposed.

Once things get bad in an organization they tend to only get worse, but things in general get better because such organizations then decay and are replaced by new ones. Unfortunately, our society now slows or prevents that process, with these same organizations and their values increasingly running the show. 

Investment and flexibility become impossible. Even appearing to care about anything except the competition itself costs you your allies. Thus things inevitably decay and then collapse, flexibility returns, cycle repeats.

Involvement with such patterns is far more destructive to humans than is commonly known.

In Is Success the Enemy of Freedom?, Alkjash talks about cases where success makes it more difficult (psychologically or really) to do what you really want to do. In Success Buys Freedom, Lsusr makes a contrary argument that success creates more freedom in practice. Reading these two (apparently contradictory) accounts, I came to the conclusion that the two authors were talking about different types of success:

• Lsusr is primarily talking about success "outside of the usual system", which generally frees someone up even more from the usual system. Start-ups are the primary example of this.
• Alkjash is primarily talking about success within the existing system. The stereotypical successful career is an example of this.

I'm told that there's an Objectivist idea along these lines. (I have not read very much Objectivism directly, so I only have second-hand interpretations.) Objectivism holds that society is controlled by collectivists, called the Looters, who will guilt you into contributing to their version of the common good. Objectivism holds that you cannot change the system from the inside, because success within the Looter system only makes you increasingly beholden to the Looters (and requires you to make compromises which worsen your character by putting their ideological hooks into your brain, etc).

I don't buy into the whole Looter ontology, but something does seem true about this. Consider:

• Success in the start-up world tends to give you big lumps of money, which increase your freedom, allowing you to do whatever it is you want to do.
• Success in existing systems tends to give you a big salary, which can be cut off if you step out of line, making you dependent.

Success within existing systems is a variance-reducing strategy -- you're signing on to a large pre-existing organization which can fairly reliably give you money (and status, power, etc). These existing systems will tend to have norms (implicit and explicit) which maintain the status quo. Putting up an extraordinary effort will tend to disrupt these norms. Furthermore, it won't usually be rewarded proportionately to the risks; you basically get your salary one way or the other. (This is of course not 100% true, but you get the point.)

And, again, when you're surrounded by people with similar incentives, doing anything different starts to become unthinkable.

This also reminds me of the incredibly common equivocation between "weird" and "bad"; "that's not normal" / "that's really weird" are typically understood as objections in themselves.

And, as emphasized in More Dakka, there's an ultra-common failure mode where we optimize for the symbolic representation rather than the thing we really want, as if all we really want out of our goals is for a neutral observer to be able to say that we tried. It's easy to see why this mindset would be common in mazes.