dotShift: Is Predictability Impossible?

We have had a certain amount of pain recently centered around the age-old conflict between the ideal and the real. Specifically, the difficulties in balancing the needs of the business for predictability with the realities of software development. Predictability is essential in making business decisions regarding resources, finance, marketing, etc. At the same time, developing software has proven to be a difficult enterprise, and once that seems to historically defy all attempts at estimation and scheduling.

So the question at the forefront of a lot of people's mind is an important one to consider, "is predictability impossible when developing software"?

Martin Fowler has been around the software development world since the 80's, and is considered a pioneer in any fields such as object-oriented technology, refactoring, patterns, agile methodologies, etc. He has been thinking and writing about this sort of issue since the 90's, and one essay in particular caught my eye. The following is a pertinent excerpt (see the full text here).

Is Predictability Impossible?

In general, no. There are some software developments where predictability is possible. Organizations such as NASA's space shuttle software group are a prime example of where software development can be predictable. It requires a lot of ceremony, plenty of time, a large team, and stable requirements. There are projects out there that are space shuttles. However I don't think much business software fits into that category. For this you need a different kind of process.

One of the big dangers is to pretend that you can follow a predictable process when you can't. People who work on methodology are not very good at identifying boundary conditions: the places where the methodology passes from appropriate in inappropriate. Most methodologists want their methodologies to be usable by everyone, so they don't understand nor publicize their boundary conditions. This leads to people using a methodology in the wrong circumstances, such as using a predictable methodology in a unpredictable situation.

There's a strong temptation to do that. Predictability is a very desirable property. However if you believe you can be predictable when you can't, it leads to situations where people build a plan early on, then don't properly handle the situation where the plan falls apart. You see the plan and reality slowly drifting apart. For a long time you can pretend that the plan is still valid. But at some point the drift becomes too much and the plan falls apart. Usually the fall is painful.

So if you are in a situation that isn't predictable you can't use a predictive methodology. That's a hard blow. It means that many of the models for controlling projects, many of the models for the whole customer relationship, just aren't true any-more. The benefits of predictability are so great, it's difficult to let them go. Like so many problems the hardest part is simply realizing that the problem exists.

However letting go of predictability doesn't mean you have to revert to uncontrollable chaos. Instead you need a process that can give you control over an unpredictability. That's what adaptivity is all about.