Waterfall versus Agile Development

A recent question from a client: "I attended your Agile for Teams class about two years ago for my company. What are the criteria for determining if a project should be run in a waterfall or Scrum methodology? I am continuing to get beat up by C-level executives stating that if we just run the project in mini waterfalls we can be agile and responsive to stakeholders as well." What would you say?

Here is my response to him:

Waterfall assumes you have comprehensive knowledge ahead-of-time of what your software product needs to be, before you actually write the software, which means before you validate it, and certainly before your customers start using it. Everything else, such as phase gate approval, serial handovers, siloed specialties, heavy documentation and rigorous change control procedures flow from this central assumption. After all, if you know this up-front, then you want to avoid anyone making changes further downstream. In waterfall the emphasis is on following the project plan and avoiding change as much as possible.

Agile assumes that not only do you not have this kind of knowledge up-front, but you should organize your process around being responsive to change at every stage as much as possible, so that you can incorporate significant new knowledge whenever it arises, for the advantage of your customers. So you maximize feedback loops, focus on growing teams of committed and skilled individuals, and repeatedly seek to get better at delivering valuable, quality working software as fast and frequently as possible.

So, my question is: Which assumption is true in your case in your business?

I’m curious what is meant by "mini-waterfalls." I’m assuming you mean cramming all the typical waterfall phases into a series of several-week blocks. Is that what your C-level executives mean by that? I’m curious how would they see a mini-waterfall enabling them to "be agile" (what do they mean by that?) and responsive to stakeholders too.

Mini-waterfalls will likely be an improvement over the long (1+ year) feedback loops of traditional waterfall, but often end up with a "frankenstein" process that is trying to be optimized for two different things. Some teams seem to have to pass through an intermediate "mini-waterfall" stage on their way to agile, but I try to avoid this happening with the teams I coach, as it is a painful, unnecessary stage.

I’ve been corresponding with someone else wanting to begin transitioning their waterfall-based company to agile. Last month she asked me: "Paul, wondering if you have slides/resources that talk about the pros and cons for both waterfall and agile methodologies. There is obviously a lot on Google, but wanted to ask in case you have something readily available." Here’s what I sent her way, so hopefully some of this will help you too…​

For the last 8 years the company Version One has done a "State of Agile" survey and compiled the results.

In 1970 Winston Royce wrote a whole paper describing why the iteration-less waterfall approach "is risky and invites failure" (Winston W. Royce (1970)^[1] I am not aware of a single case where waterfall has outperformed agile, but there is overwhelming evidence for the contrary. In 25 years of developing software myself I have never seen a waterfall project achieve its time, scope, budget and quality objectives. However, I have coded for, led, and coached numerous successful agile projects at a wide variety of companies.

A good resource to consult is The Business Value of Using Agile Project Management for New Products and Services by Dr. David F. Rico. It is a concise and well documented survey of data in support of Agile as a “study of studies” summarizing the agile impact. You might be able to draw on some of the charts in there.

See Iterative and Incremental Development: A Brief History. Page 9 says:

Further to my last email, is waterfall working so well for your organization that you would want to hold on to it? Are your projects are always successful because the business and customer outcomes are met with good cost management? Do employees feel energized and productive? Is the software high quality because the users love it and it’s easy to maintain, plus rework is minimal? And is your organization is continually getting more productive at delivering?

If all these conditions are being met then waterfall may be working for you as well as agile can. I’ve never yet to see an organization where this is the case. The opposite is always the case, waterfall is slow and inefficient to take software to the customers, customers and users are unhappy with the software and long lead times to get updates. Waterfall creates organizational silos, where the business and software people compete and argue over scope. Waterfall minimizes feedback loops, delaying the delivery of working software until extremely late in the process, maximizing the risk of building the wrong product. And waterfall imposes a heavy process burden with excessive and wasteful documentation standards.

I am working with teams doing agile that measure lead time (time from concept into users hands) in days rather than years. They collaborate together daily in teams across roles and specialties, focusing on delivery a feature or two at a time with high quality, and get fast feedback from their customers and stakeholders about whether they are on the right track or not based on actual usage of the software. They meet regularly to reflect on their process, run experiments and tweak their process, and are improving their productivity as a result. They are learning all the time and adjusting their direction as needed based on stakeholder needs, market changes, technology improvements and design insights. They keep ceremony and process to a minimum so they can focus on their primary job of delivering high quality, working software to customers as fast and frequently as possible. This is the reality of the teams I work with.

Is you would remember from what we covered on day one of your Agile for Teams class, the agile manifesto says in software development things like contracts, documentation, planning and processes are all important, but they are a means to an end, because there are other things that are even more important: Customer collaboration, people and interactions, responding to change and working software. Agile practices are geared towards these values. Does your organization want to be geared towards these same higher values? The same goes for the lean disciplines, which we also covered in class. Waterfall processes are typically not aligned well with the lean disciplines like "deliver fast" and "build quality in."

Images sourced from historical photos of Niagara Falls barrel riders at The Spirit of Blondin Lives in Wallenda!, published June 19, 2012 by dk Levick. Carlisle Graham - first to go through the Whirlpool Rapids in barrel 1886 from Niagara Falls Public Library