A guide to putting into practice a number of ideas that have been discussed on this blog over the last 18 months, especially with regards to MMF valuation (already incorporating a number of feedback iterations from different projects).

Download: Options Analysis User Guide 1.2

If you’d like a copy of the spreadsheet then drop me a line or else join the Real Options discussion group and download from there.

[update: uploaded latest version of PDF]

Advertisements

In the last post we argued for a more rigourous, quantitative approach to featureset valuation over the conventional, implicit and overly blunt mechanisms of product backlog prioritisation. We borrowed a simple valuation equation from decision tree analysis to give us a more powerful tool for both managing risk and determining the optimal exercise point for any MMF:

Value = (Estimated Generated Value * Estimated Risk) - Estimated Cost

where 

Estimated Risk = Estimated Project Risk * Estimated Market Risk

A few comments are worth noting about this equation:
1.) It contains no time-dependent variable. The equation simply assumes a standard amortisation period to be agreed with stakeholders (typically 12 to 24 months). Market payback functions and similar are ignored as they introduce complexity and hence risk (as described in more detail below). We are not seeking accuracy per se, but simply enough accuracy to enable us to make the correct implementation decisions.
2.) It is very simplistic. Risk management must be reflexive: at the most basic level, project risk can be divided into two fundamental groupings:

  • Model-independent risks
  • Model-related risk

The former includes typical factors such as new technologies, staff quality and training, external project dependecies, etc. The latter includes two components: the inaccuracy of the risk model and the incomprehensibility of the risk model. We start incurring inaccuracy risk as soon as the simplicity of our model is so great that it leads us to make bad decisions or else provides no guidance. MoSCoW prioritisation is a good example of this. On the other hand we start incurring incomprehensibility risk as soon as the risk model is so complex that it is no longer comprehensible by everyone in the delivery team (which will clearly be relative across different teams). The current financial crisis is a large-scale example of a collapse in incomprehensibility risk management. If financial risk models had been reflexive and taken their own complexity into account as a risk factor, then there is no way we would have ended up with situations where cumulative liabilities were only even vaguely understood by financial maths PhDs: if we take a team of twenty people, it is clear that a sophisticated and accurate model that is only understood by one person entails vastly more risk than a simplistic, less accurate model that everyone can follow. We can generalise this in our estimation process as follows:

Total Risk = Project Risk * Market Risk * Model Incomprehensibility Risk * Model Inaccuracy Risk

or as functions:

Total Risk = Risk(Project) * Risk(Market) * Risk(Incomprehensibility(Model)) * Risk(Inaccuracy(Model))

Furthermore, given our general human tendency towards overcomplexity for most situations this can be approximated to

Total Risk = Risk(Project) * Risk(Market) * Risk(Incomprehensibility(Model))

3.) All risk is assigned as a multiplier against Generated Value, rather than treating delivery risk as an inverse multiplier of Cost. I have had very interesting conversations about this recently with both Chris Matts and some of the product managers I am working with. They have suggested a more accurate valuation might be some variation of:

Value = (Estimated Generated Value * Estimated Realisation Risk) - (Estimated Cost / Estimated Delivery Risk)

In other words, risks affecting technical delivery should result in a greater risk-adjusted cost rather than a lesser risk-adjusted revenue. This is probably more accurate. However is that level of accuracy necessary? In my opinion at least, no. Firstly it creates a degree of confusion as regards how to differentiate revenue realisation risk and delivery risk: is your marketing campaign launch really manifestly different in risk terms from your software release? If either fails it is going to blow the return on investment model, so I would say fundamentally no. Secondly, I might be wrong but I got the feeling that part of the reticence to accept the simpler equation from our product management was a preference against their revenue forecasts being infected by a thing over which they had no control: namely delivery risk (perhaps a reflection of our general psychological tendency to perceive greater risk in situations where we have no control). However that is a major added benefit in my opinion: it helps break down the traditional divides between “the business” and “IT”. As the technology staff of Lehman Brothers will now no doubt attest, the only people who aren’t part of “the business” are the people who work for someone else.

For me, this approach creates the missing link between high-level project business cases and the MMF backlog. We start with a high-level return on investment model in the business case, that then gets factored down into componentised return on investement models as part of the MMF valuation process. These ROI components effectively comprise the business level acceptance tests for the business case. The componentised ROI models then drive out the MMF acceptance tests, from which we define our unit tests and then start development. In this way, we complete the chain of red-gree-refactor cycles from the highest level of commercial strategy down to unit testing a few lines of code. The scale invariance of this approach I find particularly aestheticly pleasing: it is red-green-refactor for complex systems…

fractal-red-green-refactor

In the previous post we started an examination of software delivery from the perspective of evolutionary biology. In that context we saw that business cases can be viewed as memes and software projects as a certain class of phenotype (or way in which that business case gets expressed). Following on from that, an organisation’s slate of new commercial development proposals can be seen as its meme-equivalent DNA, where at any given moment a subset of those replicators will be activated/ratified and then express an extended range of intra- and extra-organisational phenotypes including marketing campaigns, IT projects, industry bodies, etc. The success of these phenotypes will in turn determine the degree to which those memes/business cases are then perpetuated via further iterations of investment and development. To understand more about how this happens, we now need to look at the nature of selective environments.

An examination of most companies today will reveal multiple concurrent levels of collaboration and competition: individuals compete and collaborate within the environment of their team, with their peers in separate teams and business divisions, and very often with other people in the industry within which they work (IT news groups being an obvious collaborative example). Teams compete and collaborate within the enviroment of their business division, across business divisions, and quite often across company boundaries with similar teams in competitor organisations. Similarly organisations compete and collaborate within industry sectors, and again sectors quite often compete within the wider economy (e.g. online music sharing services competing with the traditional record industry).

The first point of great interest about this is its symmetry with the scale-invariance of power law systems. Whether we are looking at the level of individual team members or the global economy, we can see the same thing happening: namely different environmental factors applying selective pressure in favour of certain key characteristics. Secondly, when we more closely examine those environmental factors within a business context we can see they are nothing other than what micro-economists refer to as incentives. Incentives are the features of economic environments that determine adaptive advantage: they create the selective pressure. (It is worth highlighting at this point that we are not making any claims about human nature: incentives can promote altruistic, enlightened behaviour as much as greed/self-interest). Along the scale described above from individuals to the global economy, different incentives will create different selective pressures. Those pressures may act in the same direction or else they may act in conflict. For example, the impending credit crunch clearly suggests that recent city trader incentive/bonus structures were in conflict with the interests of the wider economy. 

Incentives can be specified either explicitly or implicitly. Explicit incentivisation takes the form of sales targets, call centre response times, unit test coverage targets or any other published quality metrics. Implicit incentivisation fills in the remaining gaps, and is normally adopted as a result of unreflective organisational behaviour (for example, inexperienced IT management rewarding anti-collaborative “rock star coder” behaviour with more kudos or the most interesting project work). It is frequently the underlying cause of unexpected or undesirable behaviour, and the first step towards effectively addressing such situations is normally the identification of those rogue incentives so they can be removed or else explicitly overridden.

In this way, we can see that the health of a business environment or any other complex system depends on the alignment of its incentives (i.e. success criteria) across the different tiers of selective pressure (something Jim Shore has recently aluded to in slightly different terms as the multiple aspects of project success). This in turn reflects the interdependencies characteristic of such power law systems. Where incentives get out of alignment, those interdependencies are no longer accommodated and malignancy is the result (quite literally in the biological world: cancerous cells compete and replicate very successfully at the cellular level, but at the overall expensive of other levels i.e. the organism).

When we consider the project-centric world view currently prevalent across the IT industry from this perspective, a few things come to light. We begin to understand that a programme management culture of on-time/on-budget project incentivisation has created selective pressure in favour of IT projects simply because they are an easy vehicle for meeting that target. Part of this is related to the misguided insistence by so many IT divisions today of referring to the other parts of their organisation as “the business” (frequently this is in turn symptomatic of an over-the-wall software release mentality and ultimately a basic lack of care about the real value of what is being delivered: “the project shipped on time and on budget, beyond that it’s not my problem”). A project does not just deliver within the IT division environment: we are part of “the business” too and we need continual reminding of that fact. As we’ve seen previously, on-time/on-budget has no direct correlation with organisation-level pressures to deliver added value. When we align selective pressures across the delivery environment and incentivise software delivery more meaningfully in terms of generating business value, IT projects are demoted to their rightful position as incidental artefacts – artefacts that frequently just get in the way.

A final key point to note about the scale-invariance of selective pressure is that it also emphasises the holistic nature of organisational health. It’s not just about the organisation: unless the needs of every interdependent adaptive tier are being met – from job satisfaction of team members up to healthy competition across your industry sector – then your organisation is ultimately going to end up in trouble.

In previous posts we saw that the generation of business value via IT projects essentially follows a power law distribution. By examining the nature of power law systems, we went on to conclude that adaptive strategies offer the most effective way of managing risk in such environments. We will now begin to explore what a fully adaptive risk management strategy might look like, using as our starting point an overview of the key principles underlying nature’s great adaptive risk management engine: Natural Selection..

Evolutionary ideas have recently been gaining prominence in studies of organisational behaviour and efficiency from two directions:

  • Evolutionary Micro-economics (top down), in response to the limitations of traditional rationalist supply/demand models based on Game Theory.
  • Adaptive Project Methodologies (bottom up), focussing on evolutionary design and iterative delivery to mitigate the inherent unpredictability of requirements and market conditions.

The most fundamental principle on which these ideas are based is the notion of a replicator. A replicator can be defined as any entity of which copies are made, where that entity has some causal influence on its own probability of being propagated. The classic biological example is a gene, which is copied during cell division and which influences its probability of being propagated via the environmental effects of the proteins it encodes (and in turn, the effects of the composite structures out of which those proteins are built). The specific DNA sequence of a gene is known as its genotype, and the corresponding expression of that genotype is its phenotype.

In the Extended Phenotype, Richard Dawkins switched the primary focus of evolutionary studies away from the organism. He showed that “organism” is ultimately just an arbitrary point along the scale of phenotypes: from specific proteins at one end, up through more complex protein structures to organs, organisms and social groups at the other. The fundamental unit driving natural selection forwards across the generations is the replicator or Selfish Gene – everything else from protein to social group is just artefactual byproduct (that impacts the probability of further replicator propagation).

Other instances of replicators include memes. A meme is “any unit of cultural information, such as a practice or idea, that gets transmitted verbally or by repeated action from one mind to another. Examples include thoughts, ideas, theories, practices..” When we consider the field of IT project delivery within this context, we can spot obvious correlations. Business cases are memes which, when ratified, result in the generation of a suite of phenotypic artefacts ranging from marketing strategies to IT delivery teams to unit tests, SCM repositories and deployed production systems to new revenue streams. These artefacts end up shaping their business division, organisation and industry sector, and in doing so determine the probability of the business case propagating and spawning further system releases, new marketing campaigns, etc.

There is a key lesson for us as IT practioners to take from this, one that evolutionary biologists have already learnt. It is that artefacts (be they organisms, social groups, IT projects or marketing campaigns) don’t ultimately matter. The thing that matters is the replicator: the business case or gene. We need to follow evolutionary biology’s re-orientation towards the gene, and shift our focus away from IT projects and create practices centred solely on the business case. I now believe that “projects” can actually be an impediment to the efficient generation of real business value from IT. They act as an inflexible body of emotional and financial investment that creates resistance to both a.) change and b.) termination where such change makes the business case no longer viable in real terms (which is when real damage is then inflicted). We will discuss more on this topic in subsequent posts. Before that however, we need to examine the nature of selective environments – which will be the subject of the next post. In doing so we will hopefully shed some light on the factors that have led to our current project-orientated IT world view.

 

In the previous post, we explored the behavioural differences of simple and complex systems. We saw that complex systems display power law distributions, the key characteristics of which are increased unpredictability and an increased likelihood of extreme events when compared to simple Gaussian systems. Additionally, the existence of positive and negative feedback loops makes them more resistant to causal analaysis: the potential for repeated amplification of trivial trigger events can make it very difficult to understand what is going on (see the 1987 stock market crash as an example). We will now examine the implications of those differences for risk management, focussing in particular on IT project delivery.

Conventional business management practices are based on the implicit assumptions we have inheritted from our cultural past, that ultimately have their roots in the scientific tradition: we use specific instances or case studies to infer a generalised understanding of a domain; that understanding then allows us to predict it, and once we can predict it we can then define an effective strategy for managing it. On the other hand, in our everyday lives and throughout the natural world reactive risk management is the norm. For example, to avoid being run over by a car when crossing the road we do not need to understand how a car works but only what it looks like (i.e. fast moving metallic thing on wheels). Similarly to avoid being eaten by a lion, a deer does not need to understand big cat physiology but only what one looks like (i.e. fast moving furry thing on legs).

From this, we can see that risk management strategies can be grouped at the most basic level into one of two categories:

  1. Cause based:
    • Standard business practice
    • Analyse cause, then define strategy
    • Predictive/Pro-active
  2. Observation based:
    • Normal practice in daily life and natural world
    • Adaptive/Reactive

In situations where they both work, the latter is obviously inferior as it affords no potential for proactivity and forward planning. However the former is critically dependent on the predictability of the thing being managed.

Now previously we have seen that IT project success in real terms appears to display power law behaviour. Possible explanations for this might include:

  1. There is a simple causal relationship with an underlying pseudo-power law phenomenon. It might just be that the size of investment in IT projects follows a roughly power law distribution and that the returns generated are directly proportionate to that investment. Most projects receive small to moderate investment whilst a few get massive investment and that is what results in the correlated power law distribution of generated business value.
  2. The world of IT project delivery is a complex but deterministic system, hence it displays power law behaviour.
  3. IT project delivery has dependecies on truly random phenomena, hence the generation of delivered business value displays power law behaviour.

Which of these is most accurate is a matter of conjecture: some people might argue for the first explanation, whilst others might stand by the second. We are going to stand back from that debate. Instead we will only assume this: that to the best of our knowledge, all of the explanations sound to some extent reasonable and one of them actually happens to be true. As discussed in the first post of this series, this then allows us to assess each strategy against possible explanation/scenario as follows:

 

This demonstrates that in the absence of certain knowledge, adaptive metholodies clearly represent the lowest risk approach to IT project delivery as they are effective for every explanation. More generally, we can summarise this by stating:

  • Simple, independent processes that are described by normal distributions are best managed by predictive strategies
  • Complex, interdependent processes that are described by power law distributions are best managed by adaptive strategies.

In the next post we will start exploring what a fully adaptive IT risk management strategy might look like, within the context of lessons we can learn from other areas including evolutionary biology.

The Power Law

May 5, 2008

In the previous post we argued that the starting point for managing risk in IT project delivery should be a description of the distribution and frequency of project success: you can’t manage something if you don’t know what it looks like. However, we saw that project success in real terms – i.e. of maintaining or increasing the long-term viability of the organisation – is not obviously measurable. We therefore proposed a triangulation approach to infer its distribution from a number of key indicators. These indicators all display power law behaviour. We will now examine what this means..

First however, some historical context. The history of ideas within our culture has its roots in the Renaissance and before that Persia and Ancient Greece. And as we should expect of any people starting to explore the unknown workings of the world they inhabit, the first relationships they discovered were the simplest. Mathematical descriptions of simple, independent observable events were formulated in the natural philosophy of Newton and Descartes, out of which evolved the classical physical sciences. The apparently objective, predictive and repeatable nature of these relationships was hailed as a sign of their exactitude (as opposed to their simplicity) and as a result the physical sciences became the benchmark by which the validity of other areas of inquiry were judged. At the same time, their core tenets of predictability and causal interaction were used as the foundations on which fields ranging from financial mathematics to the social sciences and management theory have been built.

This world of classical physics is one of Bell Curves (also known as the Normal or Gaussian distribution), stable averages and meaningful standard deviations. It is easily demonstrated by example of a coin toss: if I repeatedly toss 10 unbiased coins then the distribution of heads will tend towards a bell curve with an average/peak at 5 heads.

Fig 1. example bell curves (courtesy of Wikipedia):

example Bell Curves

The first challenge to this world view came from quantum mechanics at the turn of the last century, where discrete causal interaction was replaced by the fuzziness of probability distribution functions and the uncertainty principle. More recently it was then challenged at the macro level by the study of the chaotic behaviour of complex systems. These systems are characterised by interdependence between events which can result in both positive and negative feedback loops. On the one hand seemingly large causal triggers can be absorbed without apparent impact whilst on the other, large effects can be spun up from trivial and essentially untraceable root causes. The result is pseudo-random behaviour, and something that follows the same mathematical description the economist Pareto discovered eighty years earlier in his studies of income distribution (succintly summarised as the 80:20 rule) and that Bradford discovered thirty years earlier in textual index analysis: namely the power law. Since then examples have been found everywhere from epidemiology, stock price variations, fractals and premature birth frequencies through to coastline structure, word usage in language, movie profits and job vacancies. 

Fig 2. example Power Law Curves (courtesy of Wikipedia):

example Bell Curves

The power law derives its name from the dependence or inverse dependence of one variable on the squared, cubed, etc power of the other. (Plot the log of one against the other, and the gradient of the straight line will give you the exponent – i.e. whether it is a square or cube relationship). For example, Pareto discovered that income distributions across populations often followed a roughly inverse square law: for a given income band, roughly one quarter of the amount of people will receive double that income and one ninth will receive triple. The fact that this holds true whether you are looking at the lowest or highest income brackets denotes a signature characteristic of power law phenomena. It is known as scale-invariance or self-similarity, and is most widely recognised in another power law field: fractals.

Other key characteristics of power laws are an unstable mean and variance (i.e. they are statistically irregular, hence unpredictable), and they have a fat/long tail in comparison to bell curves (i.e. extreme events are a lot more frequent):

“The dream of social science [JE: project methodologies??], of building robust frameworks that allow prediction, is shattered by the absence of statistical regularity in phenomena dominated by persistent interconnectivity.” (Sornette, 2003)

“Paretian tails decay more slowly than those of normal distributions. These fat tails affect system behaviour in significant ways. Extreme events, that in a Gaussian world could be safely ignored, are not only more common than expected but also of vastly larger magnitude and consequence. For instance, standard theory suggests that over that time [JE: 1916 – 2003] there should be 58 days when the Dow moved more than 3.4 percent; in fact there were 1001″ (Mandelbrot and Hudson, 2004)

The fundamental message here can be read as follows. The apparently objective world of simple, independent events, normal distributions and classical physical/economic sciences is not actually the norm. Being the domain of the most simple events, it’s just that we discovered it earlier than everything else. In fact it is the limiting edge case along a sliding scale of much more commonly occurring complex and/or chaotic systems through to truly random or stochastic processes, all of which exhibit intrinsically unpredictable and more extreme power law behaviour. And the critically important point as it affects us in the delivery of IT projects? – that we need a risk management model tailored to the complex world of generating business value rather than the vastly over-simplistic world of basic mechanics.  The most spectacular/shocking example of what happens when someone attempts to model such power law systems using the normal distributions of classical methodologies is given by the collapse of the Long Term Capital Management hedge fund. As regards the implications for us within the realms of risk management of IT project deliveries, that will be the subject of the next post.