Why Software Projects are terrible and how not to fix them (by Drew Crawford):

Unless you are having a meeting with the one person who is going to use the software that you’re writing, you’re not meeting with the real customer.  You’re meeting with a person who has to explain to someone who can explain to someone who can explain what you’re saying to the real customer.  It’s not enough to convince the person you’re sitting in the room with that Agile is a good idea.  He has to convince his boss.  That person has to convince his boss.  That person has to convince the sales team.  The sales team has to convince the customer.  If the customer is b2b, your contact at the customer organization has to convince his boss.  Who convinces his boss.  Who convinces the real customer.  Maybe.  Unless that sale is also b2b.  This is a very long game of telephone.  If the guy you’re talking too is thinking “This sounds like a really good idea but I’m concerned I can’t sell this upstairs,” you are dead in the water.  At any point in the chain, if somebody thinks that, you are dead in the water.  You can’t just say “It’s objectively better,” you have to show how he can turn around and sell the idea to someone else.

Put yourself in the middle manager’s shoes.  If the project goes bad, he has to “look busy”.  He has to put more developers on the project, call a meeting and yell at people, and other arbitrary bad ideas.  Not because he thinks those will solve the problem.  In fact, managers often do this in spite of the fact that they know it’s bad.   Because that’s what will convince upper management that they’re doing their best.

In other words, it's all about signaling, isn't it? Managers will take actions that actively harm the continued progress of the project if that action makes them look "decisive" and "in charge".  I've seen this on many projects I've been on, and it took me a while to realize that my managers weren't stupid or ignorant. It's just that the organization I was working in put a higher priority on process than on results. My managers, therefore quite rationally did things that maximized their apparent value in the eyes of their bosses, even if it meant that the project (and, as a result) the entire organization was hurt.

Crawford then goes on to detail why organizations with such maladaptive practices survive:

Yes, businesses are under pressure to gravitate toward bad engineering practices, but shouldn’t they be under equal market pressure to compete against companies that are using actually good software engineering practices?  Shouldn’t, at some point, bad companies simply implode under their own weight? Why sure, in the long run.  But as Keynes succinctly put it, “In the long run, we’ll all be dead.”  Eventually is a long time.  It’s months, years, or decades.  A project can be failing a long time before management is clued in.  And even longer before management’s management is clued in.  And it can be ages before it hits the user.

I think this is something that we as rationalists sometimes forget about. Irrationality has momentum. Humans have been thinking intuitively for thousands (hundreds of thousands, even) of years before we figured out how to think with rigorous rationality. Even if rationality had a massive advantage of intuitive thinking in everyday situations (it doesn't, as far as I can tell) it would take a very long time for rational thought to propagate through society.

So the next time you get frustrated at some bit of wanton irrationality, remind yourself, "Momentum," before you get frustrated.

EDIT: Fixed spelling as per RolfAndreassen's post.