The CHAOS Reports have been published every year since 1994 and are a snapshot of the state of the software development industry. The 2015 edition studied 50,000 projects around the world FY2011-2015, ranging from tiny enhancements to massive systems re-engineering implementations.
For the 2015 report it has been redefined what “successful” means. Earlier a project was considered successful if it was on budget, on time and on target (e.g. scope). Standish Group admits now that “on target” was never a good measure because it lacked any measure of customer outcome. A lot of projects could meet all these constraints, but still leave the customer unsatisfied – and this is noted by Standish Group. “On target” is now replaced with a measure of customer perceived value, and other additions has been done, so that in all there are 6 factors to consider: on time, on budget, on target, on goal, value and satisfaction. This has resulted in a 7% decrease in the rate of successful projects from the year before.
If we look at the agile vs waterfall metrics this is what the report gives us:
What is very clear is that agile projects are a lot more successful than waterfall projects. Considering all project sizes agile is about 350% more likely to be successful. For large size projects agile projects are 600% more likely to be successful. Another key takeaway is that regardless of approach you really should be doing small projects. If you are trying to scale your projects, then waterfall seems to not scale very well. In any way – choose an agile approach. These numbers have been consistent over time now – abandon your waterfall approach!
Although this report tells us that agile clearly is the way to go, there is still lots of room for improvement. The average number of successful agile projects is at 39%. This leaves 61% of the projects challenged or failed. While this is clearly better than 89% (waterfall), it is still a high number and an indication that there is a lot of bad agile out there.
I have also taken a look at The State of Agile Report 2015 and The 2015 State of Scrum Report. There is a lot of things to take from these surveys, and maybe I’ll address more of my takings on these reports in later posts. For this post however, I am going to focus on the success and failure rates of agile projects.
In The State of Agile Report it is stated that only 1% of the respondents admitted that their agile implementation was unsuccessful. In the email I got from VersionOne (yes, the survey is sponsored by a vendor) the wording is a little different: “only 1% had experienced agile failure”. Up until the 2013 version of this report one of the options for leading reasons for failed agile projects was “None of our agile projects failed”, and it’s a little confusing that this has been removed as an option now, because we don’t really get an indication of the number of failed agile projects anymore (except that only 1% has experienced agile failure). For 2012 and 2013 the numbers where 18% and 15% respectively for this option – so quite low numbers. The State of Scrum Report says the overall success rate of projects delivered using Scrum is 62%. If the Scrum project was deployed and managed through a PMO 93% of them were considered successful.
The numbers from the surveys seems to be miles away from the Chaos Report numbers with regards to the success rate. One explanation is of course that the criteria for calling a project a success is not the same, and probably a lot more stringent in the Chaos Report. The State of Scrum Report even has 52% of their respondents stating that the most common challenge is identifying and measuring the success of Scrum projects. Still, I am having a bit trouble believing that only 1% actually has experienced agile failure and that 93% of Scrum projects deployed and managed through a PMO are considered a success. Having said that, these surveys are by no means scientific and the objectivity of the respondents with regards to the projects they have been on might not be the best.
So despite the numbers in The State of Agile Report 2015 and The 2015 State of Scrum Report being very optimistic for everyone that loves agile, it seems the numbers might be a bit over-optimistic. The Chaos Report probably has a bit more valid numbers – but the overall takeaway should be that agile rocks and that waterfall should already have been thrown away. Having said that, it seems to be a lot of work left before we can be happy about the overall situation. Overall 61% of agile projects are not considered a success – and that is a lot of waste in the long run. We can always do better, and that is exactly what we must do through continuously improving ourselves and our agile practices.