The Waterfall Accident
H. L. Mencken: “For every complex problem there is a solution that is simple, neat and wrong.”
For decades already, the waterfall model has been the prevalent approach when it comes to software engineering. Maybe the main advantage of this model is its simplicity and immediate appeal to common sense: First you find out what is needed, then you decide how you are going to do it, then you do it, after that you check if you really did it right, and finally you use it. But then, ...
On the other hand, the Agile community has carefully worked out that the waterfall approach doesn't work, and has proposed and successfully used alternative techniques for years. But despite all evidence, the waterfall process is a standard approach in many software projects even today.
So it is quite surprising that the omnipresence of the waterfall process is actually not much more than a quirk of history, a historical accident.
The waterfall model is not what we think it isFirst of all, the waterfall model is not what we think it is. What we today know as the waterfall model comes from a paper with the title "Managing the Development of Large Software Systems", written in 1970 by Winston W. Royce. On page 2 it contains the famous diagram with the cascade starting at "System requirements" on the upper left, continuing on through "Program Design" and "Coding" down to "Operations" on the lower right.
But nobody seemed to notice that Royce does not promote this model. On the contrary, directly below he writes “… the implementation described above is risky and invites failure.” He then goes on to promote a different process: He recommends to “do it twice” by building a throw-away “pilot model” first to explore novel elements and unknown factors. Furthermore, in the introduction Royce admits that he has no data to back-up his ideas, he calls them "personal views" and "prejudices".
Winston’s son Walker Royce described what his father felt about the public misconception of his paper, saying that the waterfall is described as the simplest description, but that it would not work for all but the most straightforward projects. He states that his father has always been a proponent of iterative, incremental, evolutionary development!
For any other idea this would not have been a good start. But not so for our waterfall. The big step towards popularity for the waterfall model came with DoD STD 2167, a military standard of the American Department of Defense for “Defense System Software Development”. It promoted a one-pass document-driven waterfall process. And it contained sentences like “The contractor shall establish the top-level design of each CSCI by allocating requirements from the SRS and, if applicable, IRS to the TLCSCs of each CSCI.” -- Quite a horror not just to supporters of Agile lightweight approaches.
Nobody can stop an avalanche…The principal author of STD 2167 later said he regretted the creation of the rigid single-pass waterfall standard. He said he was not familiar with the practice of timeboxd iterative development and evolutionary requirements at the time. He said that if he could write 2167 again, it would contain a strong recommendation for incremental iterative development!
But that was too late. 2167 had already influenced many other US and international software development standards such as the German V-Model. And even though the Dod started to warn people against 2167 and the waterfall as early as 1987, other governments and standards body did not update their standards but stuck to the old waterfall.
To sum it up: It was really a pure accident and a series of misunderstandings and speculations that led to the perception of the waterfall as the “obvious tried-and-true” process for long, complex and innovative projects. Wow!!
If you want to read more ...
- Have a look at a book that contains the complete story: Craig Larman’s book “Agile and Iterative Development: A Manager’s Guide” on page 102ff.
- Read Winston Royce's original paper on the Waterfall process.
- Enjoy the DoD military standard STD 2167