Software development is hard. This has produced failure, which has led to mistrust. Companies have tried to manage this with command-and-control structures. Command-and-control contributes to an unhealthy work environment (i.e. very toxic work relationships). The cycle completes and we end up back at software development being hard...harder now, due to the unhealthy work environment. And so, the system feedback propagates throughout the cycle making things worse as time progresses. I like to think Kent Beck, when he came up with XP, was seeking a way to escape this cycle...and he did. He created XP, which has allowed us to effectively deliver software.
Now, at this point, is where the story of XP gets interesting, because it is happening right now. We are seeing "Agile" being practiced by two very different mindsets.
The leading practitioners of XP have cultivated trust in their work environment. This has led to companies becoming humane. When I refer to "humane" here, I do not mean that employees get to have fun, and that we should recognize and favor practices that have the side-effect of making employees happier. A humane organization is one where the entire point of the organization is to focus on creating work environments in which the people that inhabit them thrive. If you doubt that XP has moved in this direction, then I recommend viewing a recent presentation by Kent Beck on Ease at Work, and Brian Marick's recent keynote at XP Day Toronto, in which he declares Ease and Joy as values that are missing from the Agile Manifesto. The XP gurus are no longer focused on software being hard. They realize there's something bigger at stake.
It's not only XP that has evolved into a values-driven culture. The Rails community has defined its values. (That's actually a statement of the values espoused by 37signals, but still representative of the Rails community.) Toyota's Lean Production process uses the value of eliminating waste to align all employees in a company to enable process innovation. Fifth Discipline is applying systems thinking to solve business problems. All these approaches clearly articulate people in an organization are just that--people. Not resources. Not capital.
Contrast this with the command-and-control culture. This is an old (older than XP anyway) culture that resides in big organizations. They see the success of XP, and want a piece of it. I call this group the Immediate Results Camp. For this group, "joy" equals money.
At my presentation, Michael Buckley questioned whether success actually produces trust. He stated that software developers have been doing a poor job for a long time. Organizations have gotten used to developers being the company bottleneck, which allows other parts of the organization to be really ineffective...as ineffective as they want, since they're not the bottleneck. Now that developers can actually produce software, other parts of the organization feel threatened, because they become bottlenecks, acquire attention, and acquire pressure to become more effective.
This is related to a point Brian Marick made: In large organizations, XP may not gain traction, despite XP's clear ability to deliver business value, since upper management is not strongly motivated to deliver business value. One way to be successful is to deliver business value, but another (and easier, the larger the organization is) way is to use "deft political maneuvering". Come up with great ideas and hand them down. If execution succeeds, it's because of upper management. If execution fails, it's because of implementors. Agile practices address this by minimizing the distance between the user and the developer. The greater the distance between these two individuals is the less that business value will motivate software production. (One powerful tool for minimizing this distance is frequent delivery.)
First-time XP-ers would tell Marick, "This is my favorite project." He is hearing that less and less. I already mentioned how Marick stated that Joy and Ease were values missing from the Agile Manifesto. The other two values he mentioned were Discipline and Skills. I think that it's significant that he's mentioning these now, and they were not part of the original manifesto. The people who were leading XP and the Agile movement took these for granted. Everyone doing XP and Agile were very capable, self-motivated, and self-disciplined. This is no longer the case as large corporations attempt to push "Agile" onto their employees. Scrum is much more popular than XP with large organizations, because it has removed the technical practices and values, and, unlike XP, which works with self-regulating teams, Scrum provides clear titles and roles.