The (Not-so) Agile Waterfall

For those who have read my blog in the past, you know that I am a strong believer in the Agile method of software development, and that I also believe that there is not a single "agile method" that will work for each organization. Instead, I believe that there are a core set of fundamental "agile" concepts that need to be implemented in the way that best meets the needs of each organization, and the entire organization needs to be very disciplined in adhering to those concepts. To define a single implementation of agile for all organizations is an oxymoron.

I have recently been made aware of yet another organization that says they are "doing agile" when in fact they are not. They are in fact following almost a text-book example of the waterfall method, but using agile terms such as "iteration" and calling it agile. They define every iteration up front, and lock themselves into dates when each iteration will be completed, even for projects that will last six months or longer. The development team is strictly held to meeting these dates, and product is not delivered until the end of the project.

This led me to wonder, what is the very least that can be done to consider yourself an "agile" development organization? And, what causes organizations to try to be agile, yet fall back to the old, failed waterfall methodology? Here are my answers to those two questions - I would love to hear your comments on this subject as well.

Regarding the question of what are the minimal Agile concepts that must be present: I believe it comes down to three:

1. Test-First: In order to truly have a customer-focused, agile process, you must have a test-first mentality, including prototypes that the customer can interact with. The customer must drive the functionality, and the acceptance criteria needs to be well-known before any development starts.

2. Commitment to change: The waterfall methodology locks you into features and dates well into the future, and thus creates a significant problem for keeping up with the fast-changing world of software development. Any implementation of an agile process must, by definition, not only allow for but embrace change throughout the project. This means that you cannot lock yourself into features or strict deadlines at the beginning of a long project - you can only lock down these things within an iteration or maybe two. After that. expect things to change, and build it into your process.

3. Working apps at the end of each iteration: In order to meet the ever-changing desires of your customers, you need to have a process that allows you to release a product with very short notice. The customer needs to be able to say "I decided that I don't want these last few features, I would rather have the product as it is today". An agile process needs to allow for this, and to provide working code to the customer, that could be released, at the end of each iteration. This also implies that you would deliver product every few months at the most, since most customers want their new features immediately.

As to the question of why organizations fall back to the old, comfortable waterfall, I believe it comes down to a very simple answer: metrics. Upper management likes to have nice metrics to track the progress of their organization, and the easiest metric to track is on-time delivery. If you can tie the delivery of something to a date, it is very easy to track that over time. So, executive management says "developers can use their agile methods, as long as they commit to features and dates at the beginning of the project, and meet those commitments". So, you end up with a purely waterfall method, where you lock in the features and dates up-front, but break things down into "iterations" so developers will think they are "doing agile". In the end, you wind up with nice metrics for upper management to track, but applications that don't meet customer needs because they can't keep up with the changing environment. Your customers become frustrated because you aren't delivering product frequently enough to meet their needs, and when you finally do deliver it isn't what they need anymore. Developers become frustrated because they ask why Agile isn't working to solve these problems.

So, it is the on-going fight between upper management and the front line. Upper management has come from a long career using waterfall methods, and view agile processes as the "cowboy" mentality. When we can provide metrics for upper management that show customer satisfaction with our frequently-released products that come from short iterations based on customer input, we will finally be able to break the cycle of trying agile but falling back into the waterfall.

Just my 2 cents...