April 13, 2010
Although I have been making use of them over the last 18 months in various presentations and the real options tutorial, I recently realised I’d omitted to publish graphs illustrating the optimal exercise point for real options on this blog. As a result, here they are:
The dotted blue line represents risk entailed by the agilist’s Cone of Uncertainty, and covers technical, user experience, operational, market and project risk – in short anything that could jeopardise your return on investment. The way this curve is negotiated is detailed below, by driving out mitigation spikes to address each risk.
The dotted brown line represents risk from late delivery. At a certain point this will start rising at a rate greater than your mitigation spike s are reducing risk, creating a minimum in the Cumulative Risk curve denoted in red. Remembering that
Feature Value = ((1 - Risk) x Generated Value) - Cost
this minimum identifies the Optimal Exercise Point.
One point worth exploring further is why Delayed Delivery is represented as risk rather than cost. The reason is because Cost of Delay is harder to model. For example, let’s say we are developing a new market-differentiating feature for a product. Let’s also say that there are X potential new customers for that feature in our target market, and that it would take Y weeks of marketing to convert 80% of those sales. Providing whenever we do launch, there remains Y weeks until a competitor launches a similar feature then then the cost of delay may be marginal. On the other hand, if we delay too long and a competitor launches their feature before us then there will be a massive spike in cost due to the loss of first mover advantage. However the timings of that cost spike will be unknown (unless we are spying on the competition), and therefore very hard to capture. What we can model though is the increasing risk is of that spike biting us the longer we delay.
I have found it very helpful to think about this using the insightful analysis David Anderson presented during his excellent risk management presentation at Agile 2009 last year. He divides features into four categories:
- Differentiator: drive customer choice and profits
- Spoiler: spoil a competitor’s differentiators
- Cost Reducer: reduce cost to produce, maintain or service and increase margin
- Table Stakes: “must have” commodities
Differentiators (whether on feature or cost) are what drive revenue generation. Spoilers are the features we need to implement to prevent loss of existing customers to a competitor, and are therefore more about revenue protection. And Table Stakes are the commodities we need to have an acceptable product at all. We can see how this maps clearly onto the example above. The cost of delay spike is incurred at the point when the feature we intended to be a differentiator becomes in fact only a spoiler.
This also has a nice symmetry with the meme lifecycle in product S-curve terms.
We can see how features start out as differentiators, become spoilers, then table stakes and are finally irrelevant – which ties closely to the Diversity, Dominance and Zombie phases. There are consequences here in terms of market maturity. If you are launching a product into a new market then everything you do is differentiating (as no-one else is doing anything). Over time, competitors join you, your differentiators become their spoilers (as they play catch-up), and then finally they end up as the table stakes features for anyone wishing to launch a rival product. In other words, the value of table stakes features in mature markets is that they represent barrier to entry.
More recently however, I have come to realise that these categories are as much about your customers as your product. They are a function of how a particular segment of your target market views a given feature. Google Docs is just one example that demonstrates how one person’s table stakes can be another person’s bloatware. Similarly, despite the vast profusion of text editors these days, I still used PFE until fairly recently because it did all the things I needed and did them really well. Its differentiators were the way it implemented my functional table stakes. The same is true of any product that excels primarily in terms of its user experience, most obviously the iPod or iPhone. Marketing clearly also plays a large part in convincing/co-ercing a market into believing what constitutes the table stakes for any new product, as witnessed in the era of Office software prior to Google Docs. The variation in the 20% of features used by 80% of users actually turned out not to be so great after all.
So what does this mean for anyone looking to launch a product into a mature market? Firstly, segment you target audience as accurately as possible. Then select the segment which requires the smallest number of table stakes features. Add the differentiator they most value, get the thing out the door as quickly as possible, and bootstrap from there.
April 23, 2009
(Firstly, a quick apology for the radio silence of late – regular updates here will now be resuming.)
So, in the last post we examined incentivisation in its broadest sense with the aim of shedding some light on the way selective pressures are created within organisations. We saw that the alignment of intra-organisational pressures with those of the external market is a fundamentally important factor in the health and efficiency of a business: put simply, it is critical to ensure that a.) employees really act in the long-term interests of the business and that b.) the business really acts in the long-term interests of the employees. We also highlighted the subtle and far-reaching consequences of how behaviour is rewarded (something the investment banks are now coming to terms with), in particular with respect to the conventional benchmarks of “on time, on budget”. We noted that this has driven the maturation of the IT project-centric world view, not because projects are fundamentally the best way of delivering business value but because they are the best vehicle for demonstrating benchmarks achieved and charging clients. More insidiously, such benchmarks can actually create selective pressures against innovation and new features (i.e. the very drivers of competitive advantage in the wider marketplace): anything innovative may be percieved as the risky option for a project team, whereas the safest way to deliver on time and on budget is to ensure they only ever undertake projects they have essentially delivered before. Indeed I have seen this happen: project after project shipped on time and budget whilst senior management sat around scratching their heads at the fact that they kept losing market share. As a wise man once said, “be careful what you ask for”.
The fundamental flaw with such approaches is obviously that they elevate the removal of uncertainty (i.e. perceived risk to timely delivery) to the primary objective, whereas the generation of new business value and innovation is normally all about working _with_ uncertainty. I think its enlightening to flip things round for a moment, and take a look at project timelines in reverse. So starting at the end point, we have the project being decommissioned at some point hopefully a number of years into the future. Roll back a bit and we might have major release upgrades interspersed with patch releases, users making use of x% of the available functionality, ongoing user training, the initial roll-out, iterations of analysis/test/build, etc. I find this “termination-oriented” perspective highly instructive. I haven’t been able to find stats concerning the lifespan of “successful” IT projects, but I’d be suprised if the average was more than five years. That implies nothing negative about the business value they generate. It simply reflects the fact that as long as a business case implementation is generating more value than it costs then it should be continued, and once that is no longer the case then it should be terminated.
Where the damage occurs is when projects/business cases despite no longer being cost effective or financially viable. In other words it not project failure that causes the worst problems but business cases that are perpetuated beyond their useful lifespan. These become the living-dead “zombie” memes, that drain the financial, staffing and morale lifeblood of an organisation. Interestingly, a former Clinton aide has just published The Tyranny of Dead Ideas about exactly this subject. What matters most is identifying such instances as early as possible and terminating them. Unlike their gamekeeping equivalent however, such project culls should also have a life-giving compliment, whereby dormant “before their time” business cases are activated in response to changing favourable market conditions. In this way, both ends of the useful lifespan of a business case are managed effectively. This is the essence of working in acceptance of uncertainty.
There is an obvious parallel here in software design. Unchecked failures/exceptions escalate. Good architecture treats error handling as an integral part of the design, and promotes defensive programming practices in order to create failure isolation boundaries that isolate and contain the consequences of the intrinsic instabilities of complex production environments. Similarly good program management should treat project failure handling as an integral part of the methodology, and promote defensive program management practices that create failure isolation boundaries that isolate and contain the consequences of the intrinsic instabilities of complex market environments. If a project failure escalates and causes significant problems to an organisation then that is not a project problem – it is a fundamental failing of program management methodology. This is why 37signals are correct in stating that organisation failure amongst start-ups is overrated. Any company that allows failure to escalate unchecked to the point where higher-level market isolation boundaries are invoked and it collapses, would signify to me a lack of management understanding concerning the essential nature of business value generation in unpredictable market conditions.
As a final thought, this idea also creates an additional viewpoint in the debate about regulation and state intervention in free-market economies. An evolutionary virtue of the free-market economy is that it entails a failure isolation boundary/ceiling between organisation and economic sector such that organisational failure is always culled before it escalates higher (the collapse of Soviet communism arguably being an instance of unchecked failure that bubbled up beyond through economic spheres and into the political). However this begs the question of whether such culling could possibly be enforced by some other means (e.g. corporate taxation structures??). If so then regulatory interventionist policies may not intrinsically be a bad thing…
May 5, 2008
In the previous post we argued that the starting point for managing risk in IT project delivery should be a description of the distribution and frequency of project success: you can’t manage something if you don’t know what it looks like. However, we saw that project success in real terms – i.e. of maintaining or increasing the long-term viability of the organisation – is not obviously measurable. We therefore proposed a triangulation approach to infer its distribution from a number of key indicators. These indicators all display power law behaviour. We will now examine what this means..
First however, some historical context. The history of ideas within our culture has its roots in the Renaissance and before that Persia and Ancient Greece. And as we should expect of any people starting to explore the unknown workings of the world they inhabit, the first relationships they discovered were the simplest. Mathematical descriptions of simple, independent observable events were formulated in the natural philosophy of Newton and Descartes, out of which evolved the classical physical sciences. The apparently objective, predictive and repeatable nature of these relationships was hailed as a sign of their exactitude (as opposed to their simplicity) and as a result the physical sciences became the benchmark by which the validity of other areas of inquiry were judged. At the same time, their core tenets of predictability and causal interaction were used as the foundations on which fields ranging from financial mathematics to the social sciences and management theory have been built.
This world of classical physics is one of Bell Curves (also known as the Normal or Gaussian distribution), stable averages and meaningful standard deviations. It is easily demonstrated by example of a coin toss: if I repeatedly toss 10 unbiased coins then the distribution of heads will tend towards a bell curve with an average/peak at 5 heads.
Fig 1. example bell curves (courtesy of Wikipedia):
The first challenge to this world view came from quantum mechanics at the turn of the last century, where discrete causal interaction was replaced by the fuzziness of probability distribution functions and the uncertainty principle. More recently it was then challenged at the macro level by the study of the chaotic behaviour of complex systems. These systems are characterised by interdependence between events which can result in both positive and negative feedback loops. On the one hand seemingly large causal triggers can be absorbed without apparent impact whilst on the other, large effects can be spun up from trivial and essentially untraceable root causes. The result is pseudo-random behaviour, and something that follows the same mathematical description the economist Pareto discovered eighty years earlier in his studies of income distribution (succintly summarised as the 80:20 rule) and that Bradford discovered thirty years earlier in textual index analysis: namely the power law. Since then examples have been found everywhere from epidemiology, stock price variations, fractals and premature birth frequencies through to coastline structure, word usage in language, movie profits and job vacancies.
Fig 2. example Power Law Curves (courtesy of Wikipedia):
The power law derives its name from the dependence or inverse dependence of one variable on the squared, cubed, etc power of the other. (Plot the log of one against the other, and the gradient of the straight line will give you the exponent – i.e. whether it is a square or cube relationship). For example, Pareto discovered that income distributions across populations often followed a roughly inverse square law: for a given income band, roughly one quarter of the amount of people will receive double that income and one ninth will receive triple. The fact that this holds true whether you are looking at the lowest or highest income brackets denotes a signature characteristic of power law phenomena. It is known as scale-invariance or self-similarity, and is most widely recognised in another power law field: fractals.
Other key characteristics of power laws are an unstable mean and variance (i.e. they are statistically irregular, hence unpredictable), and they have a fat/long tail in comparison to bell curves (i.e. extreme events are a lot more frequent):
“The dream of social science [JE: project methodologies??], of building robust frameworks that allow prediction, is shattered by the absence of statistical regularity in phenomena dominated by persistent interconnectivity.” (Sornette, 2003)
“Paretian tails decay more slowly than those of normal distributions. These fat tails affect system behaviour in significant ways. Extreme events, that in a Gaussian world could be safely ignored, are not only more common than expected but also of vastly larger magnitude and consequence. For instance, standard theory suggests that over that time [JE: 1916 - 2003] there should be 58 days when the Dow moved more than 3.4 percent; in fact there were 1001″ (Mandelbrot and Hudson, 2004)
The fundamental message here can be read as follows. The apparently objective world of simple, independent events, normal distributions and classical physical/economic sciences is not actually the norm. Being the domain of the most simple events, it’s just that we discovered it earlier than everything else. In fact it is the limiting edge case along a sliding scale of much more commonly occurring complex and/or chaotic systems through to truly random or stochastic processes, all of which exhibit intrinsically unpredictable and more extreme power law behaviour. And the critically important point as it affects us in the delivery of IT projects? - that we need a risk management model tailored to the complex world of generating business value rather than the vastly over-simplistic world of basic mechanics. The most spectacular/shocking example of what happens when someone attempts to model such power law systems using the normal distributions of classical methodologies is given by the collapse of the Long Term Capital Management hedge fund. As regards the implications for us within the realms of risk management of IT project deliveries, that will be the subject of the next post.
April 29, 2008
An interesting article by Scott Ambler has been the recent subject of discussion within the development community of my current employer. In the article, Ambler suggests that the IT project failure rates frequently bemoaned by the likes of the Standish Chaos report in fact paint a distorted picture. Many so-called failures go on to deliver additional value to their organisations that far outweighs their total cost despite the fact they originally shipped over budget or schedule. In doing so, they render the traditional success criteria of “on-time, on-budget” pretty meaningless. As a result, he reasonably argues, project success is actually more frequent than the commonly held view suggests.
His article clearly highlights the elusive nature of project success as a directly observable (and therefore measurable) phenomenon. It is apparent that “on-time” and “on-schedule” are neither necessary or sufficient as benchmarks of success: many of us have worked both on a.) projects that shipped on-time/budget but delivered no long term value due to a flawed business case or an unforeseen changes in market conditions, and b.) projects that shipped late or over budget but that have transformed the profitability of the organisation that delivered them. Long-term revenue generation might seem a more reliable (but also less measurable) indicator, but even then many projects – e.g. regulatory compliance systems – have no direct bearing on revenue generation.
This has some significant implications for software project management. If on-time and on-budget are misleading benchmarks of project success and failure, then they must equally be unreliable indicators for risk management (as the risk we are managing is the risk of success/failure), which means that risk mitigation strategies based on those indicators should similarly be unreliable.
Which poses the question, if we can’t directly measure project success then how can we effectively manage it?
As a start to answering this, it seems reasonable to suggest that whilst we can’t measure success per se we might still be able to triangulate to at least a general understanding of its distribution using other markers that are directly measurable. By inducing a few plausible possibilities that to the best of our abilities we believe to be roughly equally likely, we can then perform a risk analysis by assessing the cumulative total risk for different management strategies across those scenarios:
So, what might we consider valid triangulation markers?
- Distribution of internet site traffic: business-to-consumer web sites are a subset of IT project deliveries that most commonly generate revenue by either CPM or CPC advertising models or else some form or e-commerce. In both instances, sites with the most page impressions will generate the greatest ad revenue or sales, and in short will be more successful (strictly speaking this is not entirely true as sites with better user segmentation data will be able to charge higher CPMs, but CPM % variations are negligable in comparison to variations in site traffic volumes so it remains a valid approximation).
- Technology stock price variations: technology companies have a business critical dependency on IT project delivery (something that most other sectors are also tending towards if not there already). Therefore we might expect some kind of correlation between technology company success, i.e. stock price, and the success of the underlying IT projects on which those organisations depend.
- Technology firm termination stats: not such an obvious choice, but firm termination stats can still tell us something indirectly about the nature of project success: a near-constant annual rate of firm terminations would imply some degree of predictability, whereas wide variations would indicate more chaotic/complex behaviour.
- Key performance indicators: the initially obvious choice. Almost equally obvious however is the question as to why projects currently remain judged against criteria of on-time/on-budget rather than KPIs. Answers might include the suggestion that we often use on-time/on-budget explicitly as our KPI, or else the more worrying possibility that on-time/on-budget is used as the standard default KPI in the frequent absence of better considered indicators and clearly defined business case acceptance tests. Add to this the facts that a.) where they exist, KPIs are normally used as an overly simplistic binary success/failure threshold thereby masking variations in the degree of success; b.) most firms do not (alas!) publish stats detailing their breakdown of IT project investments and KPI ratios where they exist; c.) as in the case of on-time/budget, the chosen KPIs might actually be bad indicators of real performance anyway, and we can conclude that they might not be so useful after all.
As it happens, all the above indicators exhibit power law distributions (for more info see Andriani and McKelvey – “Beyond Gaussian Averages: Redirecting Management Research Toward Extreme Events and Power Laws”; Barabasi and Frangos – “The New Science of Networks”; Paul Ormerod – “Why Most Things Fail”). What this means, and how power law behaviour differs from the classical mathematical/physical/economic world of normal distibutions will be the subject of the next post. In essence however, it suggests that while Ambler is correct in proposing that some projects over-schedule or over-budget may in fact be successes when measured against more meaningful criteria, overall there actually appears to be a higher rather than lower incidence of project “failure” in real terms.
As a concluding note it is worth highlighting that at this stage we are not making any claims about cause but only behaviour (for example, the power law distribution of internet site traffic could simply be a direct result of a 1:1 causal relationship with a power law distribution in web site investment/costs). All we are doing is a slight twist on the standard approach of starting from observable behaviour and then inferring the generality/causal explanation: because the behaviour (i.e. distribution of IT project success) is hidden in this instance, we must add another step first to triangulate it from related observables. Only then will be in any position to consider underlying causes.