InfoQ article with Chris Matts

Towards the end of last year I was lucky enough to meet up with Chris Matts for lunch on a fairly regular basis. To my great surprise, he did me the amazing honour of capturing some of what we discussed concerning memes in one of his legendary comics. They have just been published on InfoQ together with a discussion piece here:

On Failure

(Firstly, a quick apology for the radio silence of late – regular updates here will now be resuming.)

So, in the last post we examined incentivisation in its broadest sense with the aim of shedding some light on the way selective pressures are created within organisations. We saw that the alignment of intra-organisational pressures with those of the external market is a fundamentally important factor in the health and efficiency of a business: put simply, it is critical to ensure that a.) employees really act in the long-term interests of the business and that b.) the business really acts in the long-term interests of the employees. We also highlighted the subtle and far-reaching consequences of how behaviour is rewarded (something the investment banks are now coming to terms with), in particular with respect to the conventional benchmarks of “on time, on budget”. We noted that this has driven the maturation of the IT project-centric world view, not because projects are fundamentally the best way of delivering business value but because they are the best vehicle for demonstrating benchmarks achieved and charging clients. More insidiously, such benchmarks can actually create selective pressures against innovation and new features (i.e. the very drivers of competitive advantage in the wider marketplace): anything innovative may be percieved as the risky option for a project team, whereas the safest way to deliver on time and on budget is to ensure they only ever undertake projects they have essentially delivered before. Indeed I have seen this happen: project after project shipped on time and budget whilst senior management sat around scratching their heads at the fact that they kept losing market share. As a wise man once said, “be careful what you ask for”.

The fundamental flaw with such approaches is obviously that they elevate the removal of uncertainty (i.e. perceived risk to timely delivery) to the primary objective, whereas the generation of new business value and innovation is normally all about working _with_ uncertainty. I think its enlightening to flip things round for a moment, and take a look at project timelines in reverse. So starting at the end point, we have the project being decommissioned at some point hopefully a number of years into the future. Roll back a bit and we might have major release upgrades interspersed with patch releases, users making use of x% of the available functionality, ongoing user training, the initial roll-out, iterations of analysis/test/build, etc. I find this “termination-oriented” perspective highly instructive. I haven’t been able to find stats concerning the lifespan of “successful” IT projects, but I’d be suprised if the average was more than five years. That implies nothing negative about the business value they generate. It simply reflects the fact that as long as a business case implementation is generating more value than it costs then it should be continued, and once that is no longer the case then it should be terminated. 

Where the damage occurs is when projects/business cases despite no longer being cost effective or financially viable. In other words it not project failure that causes the worst problems but business cases that are perpetuated beyond their useful lifespan. These become the living-dead “zombie” memes, that drain the financial, staffing and morale lifeblood of an organisation. Interestingly, a former Clinton aide has just published The Tyranny of Dead Ideas about exactly this subject. What matters most is identifying such instances as early as possible and terminating them. Unlike their gamekeeping equivalent however, such project culls should also have a life-giving compliment, whereby dormant “before their time” business cases are activated in response to changing favourable market conditions. In this way, both ends of the useful lifespan of a business case are managed effectively. This is the essence of working in acceptance of uncertainty.

There is an obvious parallel here in software design. Unchecked failures/exceptions escalate. Good architecture treats error handling as an integral part of the design, and promotes defensive programming practices in order to create failure isolation boundaries that isolate and contain the consequences of the intrinsic instabilities of complex production environments. Similarly good program management should treat project failure handling as an integral part of the methodology, and promote defensive program management practices that create failure isolation boundaries that isolate and contain the consequences of the intrinsic instabilities of complex market environments. If a project failure escalates and causes significant problems to an organisation then that is not a project problem – it is a fundamental failing of program management methodology. This is why 37signals are correct in stating that organisation failure amongst start-ups is overrated. Any company that allows failure to escalate unchecked to the point where higher-level market isolation boundaries are invoked and it collapses, would signify to me a lack of management understanding concerning the essential nature of business value generation in unpredictable market conditions.

As a final thought, this idea also creates an additional viewpoint in the debate about regulation and state intervention in free-market economies. An evolutionary virtue of the free-market economy is that it entails a failure isolation boundary/ceiling between organisation and economic sector such that organisational failure is always culled before it escalates higher (the collapse of Soviet communism arguably being an instance of unchecked failure that bubbled up beyond through economic spheres and into the political). However this begs the question of whether such culling could possibly be enforced by some other means (e.g. corporate taxation structures??). If so then regulatory interventionist policies may not intrinsically be a bad thing…

Environmental Selective Pressure

In the previous post we started an examination of software delivery from the perspective of evolutionary biology. In that context we saw that business cases can be viewed as memes and software projects as a certain class of phenotype (or way in which that business case gets expressed). Following on from that, an organisation’s slate of new commercial development proposals can be seen as its meme-equivalent DNA, where at any given moment a subset of those replicators will be activated/ratified and then express an extended range of intra- and extra-organisational phenotypes including marketing campaigns, IT projects, industry bodies, etc. The success of these phenotypes will in turn determine the degree to which those memes/business cases are then perpetuated via further iterations of investment and development. To understand more about how this happens, we now need to look at the nature of selective environments.

An examination of most companies today will reveal multiple concurrent levels of collaboration and competition: individuals compete and collaborate within the environment of their team, with their peers in separate teams and business divisions, and very often with other people in the industry within which they work (IT news groups being an obvious collaborative example). Teams compete and collaborate within the enviroment of their business division, across business divisions, and quite often across company boundaries with similar teams in competitor organisations. Similarly organisations compete and collaborate within industry sectors, and again sectors quite often compete within the wider economy (e.g. online music sharing services competing with the traditional record industry).

The first point of great interest about this is its symmetry with the scale-invariance of power law systems. Whether we are looking at the level of individual team members or the global economy, we can see the same thing happening: namely different environmental factors applying selective pressure in favour of certain key characteristics. Secondly, when we more closely examine those environmental factors within a business context we can see they are nothing other than what micro-economists refer to as incentives. Incentives are the features of economic environments that determine adaptive advantage: they create the selective pressure. (It is worth highlighting at this point that we are not making any claims about human nature: incentives can promote altruistic, enlightened behaviour as much as greed/self-interest). Along the scale described above from individuals to the global economy, different incentives will create different selective pressures. Those pressures may act in the same direction or else they may act in conflict. For example, the impending credit crunch clearly suggests that recent city trader incentive/bonus structures were in conflict with the interests of the wider economy. 

Incentives can be specified either explicitly or implicitly. Explicit incentivisation takes the form of sales targets, call centre response times, unit test coverage targets or any other published quality metrics. Implicit incentivisation fills in the remaining gaps, and is normally adopted as a result of unreflective organisational behaviour (for example, inexperienced IT management rewarding anti-collaborative “rock star coder” behaviour with more kudos or the most interesting project work). It is frequently the underlying cause of unexpected or undesirable behaviour, and the first step towards effectively addressing such situations is normally the identification of those rogue incentives so they can be removed or else explicitly overridden.

In this way, we can see that the health of a business environment or any other complex system depends on the alignment of its incentives (i.e. success criteria) across the different tiers of selective pressure (something Jim Shore has recently aluded to in slightly different terms as the multiple aspects of project success). This in turn reflects the interdependencies characteristic of such power law systems. Where incentives get out of alignment, those interdependencies are no longer accommodated and malignancy is the result (quite literally in the biological world: cancerous cells compete and replicate very successfully at the cellular level, but at the overall expensive of other levels i.e. the organism).

When we consider the project-centric world view currently prevalent across the IT industry from this perspective, a few things come to light. We begin to understand that a programme management culture of on-time/on-budget project incentivisation has created selective pressure in favour of IT projects simply because they are an easy vehicle for meeting that target. Part of this is related to the misguided insistence by so many IT divisions today of referring to the other parts of their organisation as “the business” (frequently this is in turn symptomatic of an over-the-wall software release mentality and ultimately a basic lack of care about the real value of what is being delivered: “the project shipped on time and on budget, beyond that it’s not my problem”). A project does not just deliver within the IT division environment: we are part of “the business” too and we need continual reminding of that fact. As we’ve seen previously, on-time/on-budget has no direct correlation with organisation-level pressures to deliver added value. When we align selective pressures across the delivery environment and incentivise software delivery more meaningfully in terms of generating business value, IT projects are demoted to their rightful position as incidental artefacts – artefacts that frequently just get in the way.

A final key point to note about the scale-invariance of selective pressure is that it also emphasises the holistic nature of organisational health. It’s not just about the organisation: unless the needs of every interdependent adaptive tier are being met – from job satisfaction of team members up to healthy competition across your industry sector – then your organisation is ultimately going to end up in trouble.

The Selfish Business Case

In previous posts we saw that the generation of business value via IT projects essentially follows a power law distribution. By examining the nature of power law systems, we went on to conclude that adaptive strategies offer the most effective way of managing risk in such environments. We will now begin to explore what a fully adaptive risk management strategy might look like, using as our starting point an overview of the key principles underlying nature’s great adaptive risk management engine: Natural Selection..

Evolutionary ideas have recently been gaining prominence in studies of organisational behaviour and efficiency from two directions:

  • Evolutionary Micro-economics (top down), in response to the limitations of traditional rationalist supply/demand models based on Game Theory.
  • Adaptive Project Methodologies (bottom up), focussing on evolutionary design and iterative delivery to mitigate the inherent unpredictability of requirements and market conditions.

The most fundamental principle on which these ideas are based is the notion of a replicator. A replicator can be defined as any entity of which copies are made, where that entity has some causal influence on its own probability of being propagated. The classic biological example is a gene, which is copied during cell division and which influences its probability of being propagated via the environmental effects of the proteins it encodes (and in turn, the effects of the composite structures out of which those proteins are built). The specific DNA sequence of a gene is known as its genotype, and the corresponding expression of that genotype is its phenotype.

In the Extended Phenotype, Richard Dawkins switched the primary focus of evolutionary studies away from the organism. He showed that “organism” is ultimately just an arbitrary point along the scale of phenotypes: from specific proteins at one end, up through more complex protein structures to organs, organisms and social groups at the other. The fundamental unit driving natural selection forwards across the generations is the replicator or Selfish Gene – everything else from protein to social group is just artefactual byproduct (that impacts the probability of further replicator propagation).

Other instances of replicators include memes. A meme is “any unit of cultural information, such as a practice or idea, that gets transmitted verbally or by repeated action from one mind to another. Examples include thoughts, ideas, theories, practices..” When we consider the field of IT project delivery within this context, we can spot obvious correlations. Business cases are memes which, when ratified, result in the generation of a suite of phenotypic artefacts ranging from marketing strategies to IT delivery teams to unit tests, SCM repositories and deployed production systems to new revenue streams. These artefacts end up shaping their business division, organisation and industry sector, and in doing so determine the probability of the business case propagating and spawning further system releases, new marketing campaigns, etc.

There is a key lesson for us as IT practioners to take from this, one that evolutionary biologists have already learnt. It is that artefacts (be they organisms, social groups, IT projects or marketing campaigns) don’t ultimately matter. The thing that matters is the replicator: the business case or gene. We need to follow evolutionary biology’s re-orientation towards the gene, and shift our focus away from IT projects and create practices centred solely on the business case. I now believe that “projects” can actually be an impediment to the efficient generation of real business value from IT. They act as an inflexible body of emotional and financial investment that creates resistance to both a.) change and b.) termination where such change makes the business case no longer viable in real terms (which is when real damage is then inflicted). We will discuss more on this topic in subsequent posts. Before that however, we need to examine the nature of selective environments – which will be the subject of the next post. In doing so we will hopefully shed some light on the factors that have led to our current project-orientated IT world view.