April 29, 2008
An interesting article by Scott Ambler has been the recent subject of discussion within the development community of my current employer. In the article, Ambler suggests that the IT project failure rates frequently bemoaned by the likes of the Standish Chaos report in fact paint a distorted picture. Many so-called failures go on to deliver additional value to their organisations that far outweighs their total cost despite the fact they originally shipped over budget or schedule. In doing so, they render the traditional success criteria of “on-time, on-budget” pretty meaningless. As a result, he reasonably argues, project success is actually more frequent than the commonly held view suggests.
His article clearly highlights the elusive nature of project success as a directly observable (and therefore measurable) phenomenon. It is apparent that “on-time” and “on-schedule” are neither necessary or sufficient as benchmarks of success: many of us have worked both on a.) projects that shipped on-time/budget but delivered no long term value due to a flawed business case or an unforeseen changes in market conditions, and b.) projects that shipped late or over budget but that have transformed the profitability of the organisation that delivered them. Long-term revenue generation might seem a more reliable (but also less measurable) indicator, but even then many projects – e.g. regulatory compliance systems – have no direct bearing on revenue generation.
This has some significant implications for software project management. If on-time and on-budget are misleading benchmarks of project success and failure, then they must equally be unreliable indicators for risk management (as the risk we are managing is the risk of success/failure), which means that risk mitigation strategies based on those indicators should similarly be unreliable.
Which poses the question, if we can’t directly measure project success then how can we effectively manage it?
As a start to answering this, it seems reasonable to suggest that whilst we can’t measure success per se we might still be able to triangulate to at least a general understanding of its distribution using other markers that are directly measurable. By inducing a few plausible possibilities that to the best of our abilities we believe to be roughly equally likely, we can then perform a risk analysis by assessing the cumulative total risk for different management strategies across those scenarios:
So, what might we consider valid triangulation markers?
- Distribution of internet site traffic: business-to-consumer web sites are a subset of IT project deliveries that most commonly generate revenue by either CPM or CPC advertising models or else some form or e-commerce. In both instances, sites with the most page impressions will generate the greatest ad revenue or sales, and in short will be more successful (strictly speaking this is not entirely true as sites with better user segmentation data will be able to charge higher CPMs, but CPM % variations are negligable in comparison to variations in site traffic volumes so it remains a valid approximation).
- Technology stock price variations: technology companies have a business critical dependency on IT project delivery (something that most other sectors are also tending towards if not there already). Therefore we might expect some kind of correlation between technology company success, i.e. stock price, and the success of the underlying IT projects on which those organisations depend.
- Technology firm termination stats: not such an obvious choice, but firm termination stats can still tell us something indirectly about the nature of project success: a near-constant annual rate of firm terminations would imply some degree of predictability, whereas wide variations would indicate more chaotic/complex behaviour.
- Key performance indicators: the initially obvious choice. Almost equally obvious however is the question as to why projects currently remain judged against criteria of on-time/on-budget rather than KPIs. Answers might include the suggestion that we often use on-time/on-budget explicitly as our KPI, or else the more worrying possibility that on-time/on-budget is used as the standard default KPI in the frequent absence of better considered indicators and clearly defined business case acceptance tests. Add to this the facts that a.) where they exist, KPIs are normally used as an overly simplistic binary success/failure threshold thereby masking variations in the degree of success; b.) most firms do not (alas!) publish stats detailing their breakdown of IT project investments and KPI ratios where they exist; c.) as in the case of on-time/budget, the chosen KPIs might actually be bad indicators of real performance anyway, and we can conclude that they might not be so useful after all.
As it happens, all the above indicators exhibit power law distributions (for more info see Andriani and McKelvey – “Beyond Gaussian Averages: Redirecting Management Research Toward Extreme Events and Power Laws”; Barabasi and Frangos – “The New Science of Networks”; Paul Ormerod – “Why Most Things Fail”). What this means, and how power law behaviour differs from the classical mathematical/physical/economic world of normal distibutions will be the subject of the next post. In essence however, it suggests that while Ambler is correct in proposing that some projects over-schedule or over-budget may in fact be successes when measured against more meaningful criteria, overall there actually appears to be a higher rather than lower incidence of project “failure” in real terms.
As a concluding note it is worth highlighting that at this stage we are not making any claims about cause but only behaviour (for example, the power law distribution of internet site traffic could simply be a direct result of a 1:1 causal relationship with a power law distribution in web site investment/costs). All we are doing is a slight twist on the standard approach of starting from observable behaviour and then inferring the generality/causal explanation: because the behaviour (i.e. distribution of IT project success) is hidden in this instance, we must add another step first to triangulate it from related observables. Only then will be in any position to consider underlying causes.