Monday, February 21, 2005

Iterations - more than just risk management

Classic software engineering tells us that iterations and incremental delivery are all about risk management. With each iteration of working code, risk is reduced for the whole release. This is a key theme in Barry Boehm and Richard Turner’s Balancing Agility and Discipline. It’s oft repeated by the establishment and by the SEI. So when you are selecting a feature mix for an iteration, you should be selecting the riskiest things first. That would be good risk management and good use of iterations. A fail early strategy!

I’d like to suggest that there are other reasons why we do iterative development. Reasons which are aligned with the Declaration of Interdependence and with Lean and agile concepts of value flow. We iterate to keep batch sizes small. Small batch transfers to system or user acceptance test, keep flow smooth through test and keep WIP inventory in test departments manageable. Small batch sizes facilitate short lead times. Short lead times mean knowledge is created quickly with minimum atrophy from concept to realization. This in turn implies that quality is higher. All of this applies regardless of how risky the individual features in an iteration mix may have been.

Naturally, it’s academically easy to argue that all of that smooth flow is simply a risk management technique. Reducing everything to risk management is academically convenient as an abstraction. However, I feel it is a model which has outlived it’s usefulness. I’m not saying that risk management as a practice is outmoded. It’s not! Risk management - the practice of anticipating special cause events and mitigating them or forming contingency plans to recover from them if they occur is still good practice. However, we can widen our view of why iterations are useful. Iterations facilitate the smooth flow of value and consequently an improved quality assurance.

Sunday, February 20, 2005

MSF for Agile Software Development

My colleague Randy Miller finally got the second draft of his new agile process out the door. We’ve created a new space at MSDN to house it and the debate around it. Go check it out.  I’d encourage you to download it. Take a look. Please provide all the feedback you have the energy to supply. We’ll have another rev of this before the product ships later this year. So here is your chance to have an influence on Microsoft’s agile offering.

Today, I presented the newly named MSF for Agile Software Development at Web Services Edge. At this conference, Visual Studio Team System was being shown everywhere. Before my presentation, there was a two-hour .Net mini-tutorial where the latest modeling capabilities were shown in a live demo. We also had machines set up in the exhibit hall so that conference goers could try it out for themselves.

Our new modeling diagrams, the logical datacenter, system, application, and class diagrams, really make building web services much easier. Not only are the code and the models synchronized, the logical datacenter diagram validates new web services against the deployment environment. The folks in the Whitehorse group, I mean, Visual Studio Team Architect group have really done a nice job.

Classifying Uncertainty Revisited

This new entry replaces an earlier one Sometimes I just get it wrong. Here is an attempt to get it right second time around. This is a supplement to the content of Chapter 4 - Dealing with Uncertainty. The diagram reflects the 4-types of variation identified by De Meyer et al in their article from Winter 2002 edition of MIT Sloan Management Review, Managing Project Uncertainty from Variation to Chaos [PDF].

We can use this model to better understand a system for software engineering in a given market.  Variation is where assigned cause has been eliminated and only chance cause exists within a known and understood system. Foreseen Uncertainty is where there are identifiable risks and understood issues which affect the delivery of the project but the basic market for the deliverable is understood and the business model or go-to-market strategy is understood. However, there is sufficient uncertainty that assignable cause variation will be observed and must be dealt with through aggressive issue log management.  Unforeseen Uncertainty will feel out of control most of the time and what gets delivered won’t be exactly what the customer wanted or when they wanted it. This could be because the software development is happening with a new paradigm of tools or method - when teams start using MDA or DSLs for example - but it may also occur in new markets where the business model is not understood and the degree of variation cannot be predicted. The answer is to iterate frequently and plan adaptively. It is primarily this class of project which Doug DeCarlo addresses with his Extreme Project Management method. Finally, there is Chaos, the land where we don’t know what we don’t know and we are trying to find out - neither the market, the business model, the customer base, the product features, or the technology are understood. It’s the land of research projects.

From a project management perspective, knowing where our project lies in this continuum is important for setting expectations. With Variation we can simply create a common cause buffer based on existing process control limits. With Foreseen Uncertainty, we create a risk management plan which may contain some mitigation approaches which amount to further buffer. In some industries this is called insurance. Insurance works just like common cause buffering but the aggregation is at a higher level. Say for example, a risk mitigation was to bring on more staff to a project, then those staff must be coming from a pool of people elsewhere in the organization. In other words, the wider organization is underwriting the insurance for the project. Unforeseen Uncertainty is a land of high risk - we don’t know what we know, or our system is not understood. Playing in this categorycould also be called gambling. As every gambler will tell you - you win some and you lose some. Many VC’s will tell you that they lose up to 19 for every one they win. The Unforeseen Uncertainty category is the land of venture capital. Finally, Chaos is a land where only the research budget should venture. It’s extreme gambling. It’s adventure rather than merely venture. It’s a world where you assume you lost your money and nothing got delivered. Delivery is a bonus. It’s like winning the lottery. The project management objectives for Unforeseen Uncertainty are simple - try to move the project into the Foreseen Uncertainty quadrant before the money runs out - then ask for more. With Chaos it is similar - try to move the project into either the Unforeseen Uncertainty or Foreseen Uncertainty quadrants before the money runs out. It is unreasonable to buffer a plan in the lower half of the diagram - a not understood market. It simply costs too much. Better not to buffer, but to get as far as you can with what you’ve got and demonstrate that you’ve moved to a world of greater certainty before asking for more resources. Iterative projects with adaptive planning is the name of the game.

It’s easy to see from this chart that the more uncertain, the greater agility is needed to react to change. In the lower half, the iteration cycle must be short, the feedback loop very tight. By understanding where our project lies in this uncertainty continuum we can make decisions about what represents the best iteration cycle and make informed guesses about how much we want to invest in requirements and analysis versus code, test and refactor.

Saturday, February 19, 2005

Thoughts on DOI #6 - Situationally Specific

And finally, I conclude my explanation of the Declaration of Interdependence with the sixth statement about situationally specific strategies and tactics.

#6 - Situationally Specific

• We improve effectiveness and reliability through situationally specific strategies, processes and practices

For me this was the hardest of the six statements to accept. I had wanted something stronger - a recognition that the most effective solution couples the engineering discipline with the project management discipline and allows them to feed off each other intelligently. I’ve learned this from FDD. The project management aspects of FDD are different because the Coad Method of software engineering enables a different way to think about project management. Project management in FDD isn’t some generic task-centric formula. It’s based on the architecture and analysis method. The reporting isn’t standard earned value rather it’s based on the production of features. I believe that when project management and engineering disciplines are combined a more effective process is delivered. I didn’t get that. So I settled for this message about situationally specific opportunities which gets us most of the way there.

So for me, the DOI is embracing the idea that it is OK to argue for different approaches and to go ahead and customize project management techniques and to couple them to knowledge of how the engineering technique works. It’s OK if you can show that it is leading to a more economically effective solution.

Friday, February 18, 2005

Thoughts on DOI #5 - Team

And yet more explanation of the Declaration of Interdependence with the fifth statement focused on teams.

#5 - Team

• We boost performance through group accountability for results and shared responsibility for team effectiveness

Taylorism in the 20th Century was all about the cost accounting notion that you optimized the whole by focusing on efficiency at the individual level. With the DOI we are calling this out to be wrong. We’re taking a Peter Drucker / Michael Porter style view that business is done in a value chain. The optimal performance is where everyone in the value chain partners together and has a vested interest in the success of the whole chain. It is this notion which led us to the statement “shared responsibility for team effectiveness”.

“Group accountability” captures the notion of “personal safety”. Personal safety is the flip-side of courage. Why should you need to be courageous? Because you do not live in an environment of personal safety! Hence, group accountability says, “we are all in this together. Together we succeed or together we accept responsibility for our failure.” By creating an environment of personal safety, we believe that individuals are motivated to do the right thing for the optimal performance of the whole system and not take a local decision to protect themselves which is sub-optimal to the global performance.