Tuesday, June 28, 2005

It’s Time to Declare Victory

Recently, we’ve seen a growing number of the agile community worrying about new topics. Guys who used to be talking about the value of standup meetings or monthly commitments are suddenly talking about corporate governance or portfolio management or product mix selection or enterprise scalability. Why? Because the agile development movement has won the debate - the battle is over. This past year, the analyst groups like Meta, Gartner and co have been around the C level executives asking them “what’s your strategy for going agile?” Yep, the jury is in and the verdict is that agile development delivers a better return for the stockholder. If you want to be a thought leader and top dollar consultant in this space now, you need a new trick. The old trick is a commodity already.

Regular readers of this site will know that they’ve been ahead of this curve for at least a couple of years. There is some pretty detailed material on product mix selection and portfolio management either in my book or in the papers that I’ve presented over the past couple of years. I’m adding to that with some solid corporate governance material in the new MSF which coupled to a tool like Team System delivers trustworthy transparency. I’m going to be saying a lot more about trustworthy transparency this coming fall. Look out for my keynote at the EuroSUN event in Copenhagen this November.

It’s actually pretty simple to explain why I’ve been ahead of the curve on this new trend - we were using transparent reporting tools with FDD in 1998. We had end-to-end traceability from requirements to tests via an intranet application. We had automated reporting and charting. We had what are now known as information radiators. In fact, the great big chart, was pasted outside the wall of the CIO’s office once a week. He had to walk past it to get to his door. He had complete transparency into all 2000 or so features. He could see how many were started, how many were finished, how many were blocked or late and of those in progress at what stage they had reached. He could see the names of the team members responsible for different sections. If the wall chart piqued his interest, then he could go pull up his home page and start clicking. He could keep clicking and see issues, see designs, object models, java doc, source code and test results. In fact everyone at the bank could technically have done the same - including the Chairman. With FDD, we were doing good governance with trustworthy transparency 7 years ago. Enterprise level objectivity in agile methods isn’t new.

The agile manifesto has had its catalytic affect. Agile is a flywheel building momentum and changing the industry - changing the way people think about delivering value from software engineering activities. Improving quality through defect reduction techniques like pair programming and test first development is all very well but to really deliver bottom line changing results, we must now look to solve harder problems. Problems that involve coordination across organizations. For that reason, the race is now on to deliver an agile enterprise with good governance, agile portfolio management and product mix selection. Expect to be hearing a lot more about trustworthy transparency.

Monday, June 27, 2005

Not So Formal

Yesterday, I talked about trust as the essence of agile software development. Today, I want to talk about a lack of trust as the scourge of traditional heavyweight software engineering methods.

When we started out to create MSF for CMMI Process Improvement, it was known by the codename “MSF Formal”. The business plan called for two methods - one agile, one CMMI Level 3. These had been identified by market research as areas of interest to our core customers for Visual Studio products - IT departments and CIO’s. It was assumed (we now know wrongly) that they were two ends of a spectrum - opposites. Agile wasn’t formal and formal wasn’t agile. That of course is correct. We associated formal with American aerospace and defense contracting and government contracting. We associated it with existing CMMI adopters. And hence assumed “formal” and CMMI were synonymous. Of course, we don’t see aerospace and defense as a big market for Visual Studio Team System this time around - v2.0 in 2007 maybe . Sure there is Microsoft platform development in that world but as a percentage of market share for Team System, it is a small market. Our core market is IT departments and outsource vendors.

What we assumed was that those who wanted to adopt formal quality assurance, needed a formal method like those used in American defense and government contracting. We even had internal evangelization presentations that described “MSF Formal” as having “all the knobs on 11” - where the knobs were documentation, gates and signoffs, verification steps and so forth. We identified these as aspects of a trustless environment. We talked of “MSF Formal” as a “low trust method”. Where there is low trust, there must be documentation, verification, audit and potentially punishment. Where there is a lack of trust, there is fear. Where there is fear, there is a lot of people taking cover and protecting themselves. Such protection is waste - muda. It takes the form of overly long schedules, overly detailed specifications, precise plans and elaborate change procedures to track all of the changes, audit trails and assigned accountability and responsibility - there must be a scapegoat.

We all know that waterfall is bad. I’ve been preaching that large batch sizes are bad for years now on this site. However, not all formal development is waterfall in nature but it can still be bad from an economic perspective because it is riddled with fear and fear breeds lots of waste. We anecdotally describe these processes as “heavyweight”. What we really mean is “lacking in trust.” Methodologies can be iterative and have lots of feedback and understand people and their psychology but if they encourage and breed fear and leave out trust then there are still likely to be heavyweight and wasteful. By holding a mirror to the problem, we see in reflection, trust as the essence of agile.

Luckily, we had an epiphany and gained a deeper, better understanding of the CMMI before it was too late. We didn’t go and create a heavyweight, formal process with all the trust taken out. That wouldn’t have been doing our market any favors. We wouldn’t have been adding value. No CIO in his right mind wants a wasteful heavyweight process even if it does gain him a CMMI Level 3 badge to brag about on the golf course on Saturday mornings. What we are delivering shows that formal quality assurance and agility are compatible because trust can be part of it. But we should have known that already. It was Deming’s mantra - drive out fear! Drive out fear! DRIVE OUT FEAR!

Sunday, June 26, 2005

PMInAction 2005

I’m going to be speaking at a PMI event this summer. PMInAction in Anaheim, California, on Saturday August 13th. I almost fell off my chair when I got that phone call! This site doesn’t exactly carry too much PMBOK material.

I’ve been asked to do 2 sessions. I will be presenting a bunch of my standard stuff but each of the two session will take a different slant. In one I’ll be focusing on how to apply the thinking of W. Edwards Deming and statistical process control to project management, planning, estimating and tracking, and in the other I’ll be focusing on the Theory of Constraints and why bottlenecks and the drum-buffer-rope solution are important for project managers worried about delivering software projects on time.

In short - two topics the attendees won’t be hearing from anyone else! So I’m differentiated value add. Hopefully I can be entertaining too. I hope to see some of you there. As Orange County CA is a big aerospace and defense area there might be a few CMMI enthusiasts who want to find out more about our new MSF for CMMI Process Improvement method for Visual Studio Team System. I’ll try to make myself available throughout the day. Please come and introduce yourself if you see me wondering the halls.

[Dowload the advertising flyer in PDF]

Here is the full text of the abstracts I submitted…

#1 The influence of W. Edwards Deming on Project Planning and Tracking in MSF v4.0

Traditional project management success is measured as conformance against plan. W. Edwards Deming taught that conformance to process and his theory of Profound Knowledge were preferable to conformance to plan or specification. He believed that focusing on process, productivity and variation created a culture of continuous improvement and ultimately led to better economic results, whilst conformance to plan encouraged heroic effort and a lack of repeatability. Deming’s work is widely admired and implemented in manufacturing and production processes. How might it be adopted into the project management body of knowledge and how would it affect the way we manage and run projects. Microsoft has adopted Deming’s thinking into its Microsoft Solutions Framework (MSF) for CMMI Process Improvement methodology. This talk will explain how conformance to process is reconciled with iterative project planning, tracking and reporting and how project managers can avoid making what Deming called Mistake #1 and Mistake #2.

#2 Focusing on Bottlenecks: How flow, variation and constraints affect project scheduling

Eli Goldratt has long since argued that traditional Gantt or PERT project scheduling is flawed. His Critical Chain scheduling method is based on his Theory of Constraints which first grew to prominence in alleviating bottlenecks in manufacturing operations. It’s poorly understood in project management circles that Goldratt’s manufacturing solution, Drum-Buffer-Rope is a pre-requisite to Critical Chain and that modeling a project a flow through a set of functional operations is essential to understanding the power of Critical Chain. This presentation will explain the theory of Agile Management for Software Engineering and show how through the incorporation of Lean queuing theory to software engineering projects, it was possible to enable both the Drum-Buffer-Rope and Critical Chain solutions for use in software projects. Microsoft has adopted the use of Lean cumulative flow tracking and reporting into its new Visual Studio 2005 Team System Product and the Microsoft Solutions Framework v4.0 methodology enabling the use of Goldratt’s theories, for better economic results in software projects.

Trust is the Essence of Agile

This past week I was putting the finishing touches to my paper for next month’s Agile Conference in Denver. [You can hear me during the experience reports on Wednesday morning.] As I read the reviewers comments [I asked people inside Microsoft and members of the Yahoo! group to review the paper] I realized that there are two schools of thought on the topic of what agile is all about and neither of them match with the world view that we, on the MSF team at Microsoft, hold.

The first school thinks that agile is all about feedback loops. It’s all small iterations in different shells from 15 minutes continuous integration to monthly sprint retrospectives. The second school of thought thinks that agile is all about people and treating people as human rather than fungible interchangeable engineering work stations. Both of these are important aspects of agile but for me they do not identify agile as unique.

Iterative and incremental development life cycle ideas have been around since the 1970’s. Agile may have added some new tooling twists in recent years such as continuous integration but I don’t believe this is enough to cite a unique differentiator. First of all, continuous integration isn’t a facet of every agile method and it uses tooling and automation which some of the agile community at other times will tell you that you don’t need. There is a bit of a double standard - tooling for testing and integration is OK, but tooling for project management is to some people, unacceptable - apparently it should all be on index cards and white boards.

Meanwhile, treating people as people isn’t new either. It’s 35 years since Jerry Weinberg published The Psychology of Computer Programming and 30 years since Fred Brooks’ original Mythical Man Month article. So, much as the work of Alistair Cockburn and others, has highlighted the importance of treating people as people and realizing that psychology is the big factor in motivation and productivity, I don’t feel that the people factor uniquely differentiates agile software development.

The factor that I feel is unique in comparison to earlier approaches is TRUST. Trust is the grease that takes the friction out of the software engineering economy. It’s well documented that economies with high levels of trust generate the highest levels of growth and the greatest wealth for their participants. For example, the trust amongst Dutch merchants 2 and 3 hundred years ago, led to the greatest levels of trade going through cities like Amsterdam. This brought great wealth to Holland and established the Dutch Guilder as the World’s reserve currency. Although there was some literature about the importance of trust in software engineering - Luke Homann’s Journey of the Software Professional comes to mind - it wasn’t a recognized aspect of the software engineering economy until agile methods put it front and center.

In developing MSF v4.0 in both its Agile Software Development and CMMI Process Improvement flavors, we’ve focused on the idea that high levels of trust were important for high productivity. We’ve focused on eliminating the waste in software engineering which comes from a lack of trust. [With the CMMI method, we still need to facilitate audits and appraisals but we’ve tried to automate as much of the evidence gathering using the tool as is possible within the specification.] Agile software development brought the idea of trust to the forefront. When there is trust, there is less waste, less extra work, less verification, less auditing, less paperwork, less meetings, less finger pointing, less blame-storming. Building trust between the engineering group and the customers is the first goal for any agile manager. Equally building trust with and amongst the engineering team is also essential. So many aspects of agile methods are about building trust - frequent delivery, focus on working code rather than documentation, face-to-face communication, pair programming, peer reviewing, stand-up meetings, shared responsibility and joint accountability, direct customer collaboration including on-site customers and customer involvement in modeling sessions, estimating, tracking and reporting based on customer valued functionality, information radiators, big visible charts, burndown, cumulative flow and ultimately complete transparency into entire engineering process. Agile for the first time, enables us to run software development like other parts of a business. It clears away the fog. It lifts the veil of secrecy. It blows away the opaque clouds and reveals the naked truth of what is really going on.

During the development of MSF v4.0, we’ve faced criticism from some quarters that Microsoft was misusing the “agile” term or misappropriating it, or seeking to reinvent it in its own image - Simon Evans provides one recent example. Nothing could be further from the truth. What we have done is create an agile method which addresses previously unsolved problems for typical IT departments. It’s a full lifecycle agile method that gives responsibilities to a wider range of roles - roles typically found in IT departments. And with our CMMI method, we have stuck closely to our core agile process definition and stretched it to fit. We’ll be releasing the Beta of MSF for CMMI Process Improvement on MSDN next weekend. Meanwhile, you can get the story of how we invented a full lifecycle agile method that is compatible with the CMMI, by coming to hear my presentation at the Agile conference. I hope to see you there.

Friday, June 17, 2005

1275 Balloons

That’s how many balloons it takes to fill the volume of an office on the Microsoft campus so that the door won’t open from outside!

How do I know this?

I arrived early to the office one Monday morning recently, to find the corridor full of balloons. A manager colleague who sits about 60 feet from me was pulling them out of his office and fighting his way in to his room. Another colleague brimming with smiles explained to me that it took 1275 balloons to fill an office and soon they would be floating all around the 3rd floor of building 25. Why? It was my colleagues 10th anniversary with the company. Some of his staff, and a few long Microsoft serving manager colleagues had got together and created the balloon stunt. They had hired two air compressors and worked late on Friday and most of Sunday to fill all the balloons.

How did they know 1275 balloons was the right number? Well it’s Microsoft tribal folklore. It has apparently been done often enough that the empirical observation has concluded that 1275 is the right number. Yes, it’s a ritual and it’s a very effective way of signaling individual value within the tribe. For a tribe that hires for intelligence, it values experience. As the office building filled with balloons which got everywhere - in my office, in the kitchen, in the toilets, in the elevators, throughout all the corridors, everyone was asking, “why is the office filled with balloons?” By the end of the first day, everyone knew why and everyone was aware of the passing of a 10 year anniversary, and the increased individual value of this long serving member of the tribe.