Two seemingly unrelated topics crossed paths while I was researching this post. A common theme in my writings is the questionable nature of statistics bandied about the Internet and presentations. Within the past few months, I heard again the myth that “90% of all communication is nonverbal,” which was thoroughly debunked in the 1980s—by the guy it’s attributed to! I thought I might prevent another myth now.
As for the other, intended topic of this post… To prove I am not an ideologue about Agile, I regularly point out I would still use traditional (“waterfall”) project management in certain circumstances. Sometimes, like last week during a talk I gave on Agile transformation, I get asked what those circumstances are.
By coincidence, I was already researching this post on that, which led to another questionable statistic. In a study directly comparing traditional versus agile projects, two professors included this line: “According to testimonial data gathered from 10,000 project managers (Wysocki, 2009, p. 328), no more than 20 per cent of all projects have the characteristics of traditional projects…” Not sure what “testimonial data” meant, I looked up the Wysocki source and found his book on reserve at the School of Information and Library Sciences at UNC-Chapel Hill. (I bragged to the head librarian that my sister was the first female Ph.D. graduate in library sciences there!)
Robert Wysocki is the author of an impressive multi-edition tome used as a textbook in many project management courses: Effective Project Management: Traditional, Agile, Extreme.[1] The origin of his statistic is a bit unclear, however. Wysocki cites no study on page 328. I suspect his claim there relates to a statement earlier in the book: “I make it a practice at all ‘rubber chicken’ dinner presentations to ask about the frequency with which the attendees encounter APM (Agile) projects. With very small variances in their responses, they say that at least 70 percent of all their projects are APM projects, 20 percent are TPM (traditional) projects, and the remaining 10 percent is split between” two other types I explain below. The “20 percent” figure lines up in the two statements, and informal surveys count as “testimonial data.”
However, his bio in the book says he has “trained over 10,000 project managers” (my emphasis). So that number does not align with the page 328 quote unless he considers a dinner presentation the same thing as a “training.” More critically, just because a project is run using waterfall doesn’t mean it “fit the characteristics” of a waterfall project—many companies continue to use waterfall that could be Agile. Indeed, his “little variance” phrase reads like a classic example of seeing and recalling what matters to you: Project Management Institute surveys[2] say far more than 20% of projects use waterfall (whether or not they should).
His statistic is far from scientifically valid, which is downright bizarre coming from a Ph.D. in mathematical statistics. Yet I can easily imagine some popular blogger quoting it as a fact without doing the legwork I did to understand the origin or check it against other sources. When that bloggers’ readers picked it up, many would only name the blogger as the source, if anyone—and a business myth would be born!
Moving on to my planned topic: How much waterfall is still valid? There are lots of general guidelines out there, mostly saying the more uncertainty there is, the more you should consider Agile. Unfortunately most projects have more uncertainty than practitioners admit. “In the case of construction,” for example, “research shows that, as late as the start of construction, significant uncertainty remains as to what is to be constructed.” So wrote two researchers at the intriguingly named Research Institute of the Built & Human Environment at the University of Salford (U.K.).[3] They include a list of studies showing little correlation between initial scoping and construction quality or project success. In other words, uncertainty eliminates the value of scope planning even on construction projects, the type most Agile doubters bring up to me as supposedly requiring waterfall.
Let us, then, get more specific on characteristics that call for Agile, to identify waterfall characteristics by elimination. In his book, Wysocki suggests a two-by-two matrix based on “Clear” vs. “Not Clear” and “Solution vs. Goal” to choose the appropriate project management methodology (PMM):
- Solution Clear and Goal Clear = Traditional
- Solution Not Clear but Goal Clear = Agile
- Solution Clear but Goal Not = “Emertxe” (his term, referring to projects to find new uses for existing technology)
- Solution Not and Goal Not = Extreme (referring to R&D projects)
I don’t buy his logic for differentiating among the last three. He claims, for example, that a difference between Agile and Extreme is that in the former, “scope is done once at the beginning of the project.” I’ve never heard or read that from any Agile expert, and the only organizations I’ve seen do it are those whose waterfall-blinded executives insisted. All of his non-Traditional PMMs are “incremental” or “iterative,” he says, words frequently associated with Agile projects. Certainly Full Scale agile™ covers all three situations. So I will combine those three, which leaves us with slightly more specific requirements: Waterfall is appropriate for projects where both the solution and the goal are clear, and Agile for all others. But this leaves out process and environmental factors.
Back to our two professors who quoted Wysocki. For a 2015 study they had performed a content analysis of 148 sources, both empirical and theoretical, to identify “critical success factors” (CSFs) for all project types based on how often factors were mentioned.[4] “CSFs are issues that if addressed appropriately, substantially increase the likelihood of project success,” they explain in the later study with the Wysocki quote.
In 2015 they found the following CSFs to be the most cited per PMM:
- Agile—”technological uncertainty, specification/requirement changes, organizational culture, change management, communication”
- Traditional—”project planning, monitoring/controlling, vision and mission, team expertise, project criticality”
Then for the 2017 study, they surveyed nearly 1,000 members of PM and Agile groups, asking which of the CSFs were important for outsourced traditional versus Agile projects.[5] They looked separately at “process success” on budget and cost adherence and “product success,” the quality of the output, noting there can be a conflict between the two. I have already dismissed the value of Triple Constraint success, so I’ll focus on product success. For that, the following CSFs were more important for the named PMM than the other PMM:
For Waterfall:
- “Specification changes”
- “User participation”
For Agile:
- “Change management”
- “Project team composition”
- “User support”
- “Technological uncertainty”
- “Technical complexity”
It seemed odd to me that making more “Specification changes” would improve product success in waterfall. But this is where the tension between the two types of success comes into play. Scope changes might kill process (Triple Constraint) success, but will deliver a product better matching what the customer needs.
Meanwhile, “User participation” is enshrined in the Agile Manifesto, so why wouldn’t it be more important for Agile instead of waterfall? In fact, the numbers show a shocking negative relationship between user participation and Agile product success, whereas for waterfall there was no correlation. The professors speculate there can be too much of a good thing, with users harming product quality by constantly changing the requirements.
I speculate this only happens when no controls are put around changes, a misunderstanding of Agile principles. In FuSS™ I insist that new user stories cannot be “officially” added mid-sprint or new epics mid-release (excluding true emergencies). That forces customers and/or Product Owners to put more thought into requirements because they can’t change them at will, and prevents teams from working on half-baked ideas that have not been thoroughly vetted by the team. This may be why “Change management” is more important in Agile than waterfall for product success—if you have no discipline, user participation can hurt the product.
Taking these sources together, we can specify that Agile is best used where there is high (“technological”) uncertainty about the solution or the goal; the project is complex; users will be actively engaged (“user support”), but their participation will be balanced with some discipline around changes (“change management”); and the “team composition” can fit the Agile recommendations (cross-functional, stable, etc.).
Waterfall is still valid in the few remaining situations. We may not know exactly its percentage of all projects, but given the description of Agile characteristics, most projects should be Agile. Wysocki’s claim that only 20% of projects fit waterfall is not built on good evidence. But it might be right anyway.
Please share this post at the bottom of the page.
[1] Robert Wysocki, Effective Project Management: Traditional, Agile, Extreme, 5th ed. (Indianapolis, IN: Wiley Publishing, Inc., 2009).
[2] Project Management Institute, “Research Highlights by Industry and Region,” Pulse of the Profession (Project Management Institute, 2018), https://www.pmi.org/-/media/pmi/documents/public/pdf/learning/thought-leadership/pulse/pulse_all-comparison-reports_final.pdf; Project Management Institute, “Capturing the Value of Project Management through Organizational Agility,” Pulse of the Profession (Newton Square, PA: Project Management Institute, 2015).
[3] R. L. Owen and L. Koskela, “Agile Construction Project Management,” n.d., 12.
[4] Arthur Ahimbisibwe, Robert Y. Cavana, and Urs Daellenbach, “A Contingency Fit Model of Critical Success Factors for Software Development Projects: A Comparison of Agile and Traditional Plan-Based Methodologies,” Journal of Enterprise Information Management 28, no. 1 (February 9, 2015): 7–33, https://doi.org/10.1108/JEIM-08-2013-0060.
[5] Arthur Ahimbisibwe, Urs Daellenbach, and Robert Y. Cavana, “Empirical Comparison of Traditional Plan-Based and Agile Methodologies: Critical Success Factors for Outsourced Software Development Projects from Vendors’ Perspective,” Journal of Enterprise Information Management 30, no. 3 (April 10, 2017): 400–453, https://doi.org/10.1108/JEIM-06-2015-0056.