Thursday, December 29, 2005

You are What You Read

I get asked often, “what are you reading at the moment?” I find that the asker often rushes out to buy and read whatever I’m reading. This provides the possibility that I can hold a deeper conversation with the person, however, I actually find it curious. Perhaps its my background in OO analysis and design and structured methods generally. Why duplicate what someone else is doing? Why create homogeneity in this fashion? Isn’t it better for my colleagues to read a different set of books? [Anyway, enough of the rhetorical questions.] Here is the list of books I’ve read this past year…

Great Boss, Dead Boss by Ray Immelman. I’ve talked about this before. Arguably one of the most important books in management and leadership to appear this decade. I wonder how long it will take mainstream management scientists to wake up to it? Darn, there I go rhetoricalizing again. Next, I read The Wee Book of Calvin (thanks, Keith.) I’ve been getting in touch with my Calvinist cultural heritage and trying to understand better how it affects my attitudes to life and work. I read a clutch of books about trust. The most important is Francis Fukuyama’s Trust, which I picked up at the Agile 2005 conference bookshop. Hal Macomber recommended the other two, though I didn’t much like The Trusted Leader by Galford and Seibold Drapeau. Flores and Solomon’s Building Trust in business, politics, relationships and life is much better though not as useful in my opinion as Fukuyama’s work which I feel is more readily applied to the world of software engineering.

I also got through a diverse set of business and related books. Steven Johnson’s Everything Bad is Good for You is amusing and stimulating and guilt relieving. All those evenings spent in front of CSI:Miami and Boston Legal as I try to chill and relax after a hard day’s toil inside the borg are indeed not wasted. Malcolm Gladwell’s Blink was equally entertaining but hard to see how it might be applied to our field. I’ve discussed Seth Godin’s All Marketers Are Liars here before. The Power of Full Engagement is on the other hand very applicable to the world of agile software development. I haven’t figured out how to articulate it yet. I took the precaution of recommending it to Alistair Cockburn whose work is more people-centric than mine and he might be able to make more use of it. If only Thomas Friedman’s The World is Flat took after its title. It’s a verbose tomb. Avoid it if you can. Lee Eisenberg’s The Number is an astonishing book that Ben Bernanke will be hoping doesn’t catch on as a best seller. Suggesting that Americans save for their retirement. Geesh! What is the world coming to? Talk like that could cause a global depression. The Rules of the Red Rubber Ball is cute and inspirational. A great little book to give as a present. You’ll read it in half an hour. My vote for best business book and second most influential book (on me) of the year is The Wisdom of Crowds by James Surowiecki. I also greatly enjoyed Managing with Aloha by Rosa Say. She’s clearly a manager after my own heart. Management is about building great teams. It’s like gardening with people. Plant, nurture, train, feed and watch and enjoy the growth.

Very recently I read Kent Beck’s Extreme Programming Explained 2nd Edition. For those who took the time to read the Acknowledgements section, you’ll know that I was one of the manuscript reviewers along with a very esteemed group including Joshua Kerievsky, Steve McConnell and Mike Cohn. I guess when you are Kent, you can attract a top rated review group. Kent made quite a few changes from the draft to the final book and I was pleasantly surprised. I’ll be saying more about this in a future post. I can also highly recommend Sanjiv Augustine’s Managing Agile Projects. Sanjiv is one of my buddies in the Agile Project Leadership Network (APLN) and though I’d got to know him well through APLN board meetings, I hadn’t had time to read his book until this summer. Ironically, I was a reviewer (3 years ago) for his proposal. He wrote a very different book than his original proposal. I think you’ll like what he has to say. I also got around to the 2nd edition of Larry Leach’s Critical Chain Project Management. It’s a text book and it reminds me of my own book. Larry is a great thinker and his book contains great thoughts. I’m not so hot on Michael A. Cusumano’s The Business of Software. Writing here as a guy who has spent his life since the age of 14 in the business of software, I didn’t recognize my business from Cusumano’s telling. Another book that didn’t resonate for me was Artful Making by Robert Austin and Lee Devin. This book is really popular in the agile community, particularly the XP and Lean crowd but it seemed simplistic to me. It polarizes the debate into mass production versus artistry and that seemed wrong. Meanwhile, just like the Cusumano book, I felt that an academic was trying to persuade me that he really understood the software business and I just wasn’t convinced.

For my day job, I read CMMI SCAMPI Distilled which I can highly recommend and I was never far from my copy of CMMI: Guidelines for process integration and product improvement by my friend Mike Konrad and his colleagues Mary Beth Chrissis and Sandy Shrum, which I read last year but had to frequently revisit throughout 2005.

And finally, I’ve been thinking about writing another book. To limber up, for this, I decided to read some novels. I want my writing to have more flourish. I want to regain the style I had five years or more ago. Try this for example, RTFM, whose time is it anyway?. Or this review of Jakob Nielsen’s Designing Web Usability. Some great writing. I wonder what it was that killed that instinct? But here I am rhetoricalizing again. Indeed, some of my web writing is much better than that in my book. The Prentice Hall style requirements don’t help. It’s a text book. Next time I’m going to do something more prose like, more right brained, less structured - at least it will look less structured to the passing reader. So to exercise my artistic bent over this holiday season I’ve read Tibor Fischer’s The Voyage to the End of the Room, and David Lodge’s Thinks. BTW. Fischer is my favorite writer in the English (as opposed to American) language.

All-in-all a fairly light literary year for me! Suggest what I ought to be reading next year! Leave me a comment… Or tell what your favorite post has been. What do you think was my best writing?

Tuesday, December 27, 2005

Extreme Brandy Butter

Traditions are important. Family traditions, held as tacit knowledge, passed from father to son, equally so. This Christmas Day I reprised an Anderson family tradition - the making of the brandy butter to be used later with Christmas pudding and mince pies as part of our traditional British Christmas feast. The Anderson family takes their dessert after Christmas turkey very seriously indeed. We’re extreme brandy butter makers.

Having a cancer patient in the family is difficult enough, when it is known to be terminal and the patient is a nine hour flight by 747 from insurance covered health care, it is all the more difficult. However, two years ago, my father was given the opportunity to spend Christmas with us here in Seattle. My mother had died that past September and for the first time since a business trip to Frankfurt in 1975, he was able to travel abroad. My mother wasn’t a traveler. Her idea of a vacation was 4 days at a bed and breakfast within 3 hours drive from home. She just didn’t like to be away. Nor did she like to be alone.

It’s an ill wind that doesn’t blow some good and that year the good that blew was the opportunity to have 3 generations of the family together, here in Seattle. Both my father and I were well aware that it might be his last. We were determined to make it work, despite the health difficulties and equally we were determined to make it fun. It was therefore necessary to pass on the tradition of making the brandy butter on Christmas Day to me - the eldest son. It seems strange that for all those years I had enjoyed the eating but had never enquired about the making nor had I ventured to expirement with it. It was implicitly the job of the head of household. So with my dad looking over my shoulder (do we call this pairing?) I prepared the ultimate condiment for the pudding. After all, it is basically fat, sugar and alcohol, what more could a body need to be happy?

To follow along you will need (at least) 8oz of unsalted butter - the kind of butter that if consulted by a hypnotist could still regress back to a memory of cows in a green pasture from its former life as milk. You’ll need a bag of fine-grained baker’s sugar. No need for measurements - this is extreme brandy butter making. For those who know anyone in the food preparation trade, you’ll know that they can tell you that cooking is an art, whilst baking is a science. Baking is chemistry. It is all about precise measurements, and precise techniques of preparation - folding, not beating for example. In theory brandy butter making is chemistry. I’m sure it is possible to create the ultimate six sigma brandy butter. And indeed, the manufacturer who produces the small jars that can be had from Cost Plus World Market in downtown Seattle for seven bucks a pop must indeed have some chemistry and some quality assurance on their side. However, for the Andersons brandy butter making is a craft - indeed an art form. So, just a bag of sugar will do fine. You’ll also need some brandy. Again, no measurements required - any old bottle of very special (VS) port or Cognac will do. You’ll need a small baking bowl, a knife, a fork (not for the extreme b-b maker any modern-fangled tools like whisks or hand-blenders, we prefer individuals and their interaction with the ingredients over processes and tools) and a spatula.

Cut the butter into smaller blocks with the knife, about 4 to 8 blocks will do - no need to be precise. We’re going to practice ECUF (enough creaming up front) also known as JECI (just enough creaming initially), so add all the blocks to the bowl. Open the bag of sugar and shake a handsome portion on to the butter. It should probably cover the butter cubes but not drown them. Squeeze gently and stir with the fork and gradually work the sugar into the butter. Gradually, it will cream and become a smoother single mass. At this point, repeat the pouring of the sugar and mix as vigorously as possible with the fork. It’s now time to sample the first feedback. Use the spatula and take a small sample from the bowl. Sweet enough? Does it taste like mildly buttered sugar? If not repeat the sugar pouring and beating and sample some more feedback with the spatula frequently. When the creamed sugar is prepared to your satisfaction, you are now ready for the highly iterative process of fortifying to create the finished product.

Plan on a large first iteration. Liberally, pour the brandy into the bowl - no 6th gil measures here. Think well left of Bill Clinton with your definition of liberal (for those reading from the archipelago just North West of Europe insert Charles Kennedy or your favorite soft politician instead.) Gradually stir the brandy into the creamed sugar. It will begin to turn a champagne brown in color. When the brandy is completely absorbed, use the spatula to sample some feedback. The second iteration can probably be of a similar length. Be generous, tell the Cognac bottle it is better to give than to receive. Mix it in again. It will take a little longer to be fully absorbed. The color will darken slightly. When sampling the feedback this time around, you should be able to detect the brandy flavor. From this point on, smaller iterations will be required. More feedback sampling will be needed to blend the perfect batch of brandy butter.

How do you know when you’re done, Dad?

When you’ve added as much brandy as you can without it separating later into its constituent parts, then you’re done! You remember that year when I over did it slightly? Yes. It separated in the fridge. Physics. The lightest stuff floats to the top. Beautiful set of layers. Well, I learned my lesson.

So small iterations, rapid feedback. Obtaining that perfect mixture for ultimate taste (and effect) needs an empirical process with oft sampled feedback on the output.

Add a thimbleful (approximately, no thimbles are actually used) of brandy. Stir. Taste. Again. Stir. Taste. At a point very soon, the creamed butter will become loose from the bowl and slide around on a thin ice of brandy. To the uninitiated, it looks like your done. Wrong! You are about half way there. Stir more vigorously. The brandy will be absorbed. Repeat. Another thimble. Stir vigorously. Taste. Repeat. Again. And yet again. Are you getting a taste for this?

If suddenly you realize that maybe, just maybe you don’t have enough product remaining in the bowl, then you are a craftsman, an artist, a scholar of the brandy butter creaming process. You’ll need to make a second bowl later. Don’t stop to get more butter from the fridge. If on the other hand, you find that you’ve had to make a mad dash to the drinks cabinet in the sitting room and raided the 5 star VSOP in order to provide raw material for future iterations then I salute you!

By this time, your eyes should be watering from the fumes rising from the bowl. If they are not, you might like to rethink your regular weekly alcohol consumption.

How does it taste now? Does it skid around the bowl for tens of seconds before absorption of each thimble of brandy? Is your feedback interpretation, “pretty darn good, potent stuff?” If so then just one more iteration should do it. Just for luck!

Decant the remaining finished product into a serving dish, cover and place in the fridge until after dinner. It should harden up just enough to be pleasant to cut and serve and should then melt into the pudding within a minute. So eat quickly for maximum pleasure. 

Monday, December 26, 2005

Burning down the Christmas card list

It’s been a recurring theme for me in recent years - the idea that we (humans) are naturally programmed to look for transaction cost optimization that naturally leads to large batch sizes and “waterfall” thinking.

This year like any other year, we spend many an evening in December working on our Christmas card list. The first task is to update the information. The data is perishable. People move. Relationships change. Names change. People we haven’t heard from for years get dropped from the list and new names added. My wife and I have lived around the World in the past 15 years. Between us, we’ve managed to live in Oxford, Toronto, Hong Kong, Singapore, Dublin, Dallas (Tx), Overland Park (Kansas), and Seattle, besides our native Scotland and Japan. We’ve picked up a lot of friends and colleagues with whom we keep in touch mostly just that one time a year.

It is interesting to compare my process against my wife’s. After gathering an updated list, together with a set of cards, a set of photos of the kids, and stamps, I dutifully write a single card, address the envelope, insert the photo, seal the envelope and stamp it for sending. On a single sitting I might write 10 cards. Cards seldom depart without a personalized hand-written note inside. It’s an effort. I manage to find the energy about two nights per week. My cards are sent off in batches of about 10, perhaps twice a week, starting in early December. I concentrate on the overseas ones and those that require the most detailed notes first. My wife has the same labor input and same quality of finish but she first addresses all the envelopes, then she writes every single card, then she adds the photos, then she stamps them and finally sends them in one single batch. Her large batch of cards is dispatched towards the 3rd week in December with some doubt as to whether the longer traveled cards will make it before Christmas. The saving grace being that few of the recipients are from a Christian tradition and hence they have a tolerance to receive the cards before or around New Year rather than Dec. 25th.

My wife’s “efficiency” driven preference - you have to know her and her focus on economics - leads to a pure “waterfall” large batch transfer system with all the value delivered in one large batch at the end, and additional risk of late delivery to some of our friends. My iterative, flow of value approach, gets some cards on their way as early as the first week in December. Risk is mitigated. However, it is possible that my iterative small batch size approach does involve more energy expenditure. Regardless, all my cards made it to their destinations before December 25th. And had I run out of energy mid-Month (my job at Microsoft has been busy recently and the commute long and tiresome) then at least half my list (those furthest away and generally my more important family and old friends) would have received their cards.

Is my approach counter-intuitive? Or is the large batch size, more efficient (batch size over transaction cost) approach the natural way for the human race?

Tuesday, December 20, 2005

Is there room for professional estimating?

It is a core tenet of eXtreme Programming, as Kent Beck reminded us in his 2nd edition, that developers do there own estimates for tasks and user stories. The idea is that over time, developers will learn to recognize the type of a story or a task and have built a body of experience which enables them to estimate it with accuracy.

Or alternatively, we could record the data in a tool!

Kent assumes with XP that tools for management are not used. Whiteboards and story cards suffice. It’s a core tenet of Agile Management that I assume you are using a tool for work item tracking. We used such a tool, known as Knowledge Management System (KMS) on the first FDD project in Singapore, and I’ve been using one ever since. Microsoft now pays me to help develop their version, Visual Studio Team Foundation Server. Once you have a tool, you don’t need to rely on developers memory, you can use the historical data.

In my book, I argued that developers should not do their own estimating, rather the (functional) manager should do it, based on the analysis of the work to be done. The developers should focus on the analysis because that analysis directly contributes to the knowledge that needs to be created.

Comments on my post Sunday emphasizing the difference between analysis and estimation, suggest that I haven’t seen professional estimators at work in other industries. This may be true. Some other comments choose to ignore my qualification that I’m talking only about the software application development business. That’s right folks, everything on this site is about management of software application development in IT shops and product or service companies. I only talk about what I know about. And unlike much of the other advice on project management out there, I never even suggest that what I teach can be used in these other domains. So, if you’re in the business of sending probes to Mars, building military aircraft, designing cars or constructing bridges, tower blocks or freeway systems, then the advice on this site probably doesn’t apply to your situation.

Glen Alleman suggests that I need a better definition for analysis and estimation. Glen might want to take that up with the folks at Webster’s because I used the dictionary definitions. As a modeler, analyst, designer, architect and engineer in the software business, I’m fussy about definitions. Getting everyone on the same page with a glossary of agreed understanding is important. If that glossary can be in the form of a model even better. I’m also not one to reinvent the wheel. When someone else has already created the knowledge, I follow a simple rule: re-use, don’t re-invent. That’s one big thing that is wrong with our industry - too much re-invention in craft shops hand building everything from scratch - not enough reuse. If a smart man learns from his mistakes (the widely advocated feedback and learning in the agile community) then a wise man learns from others mistakes. The wise software engineers are the ones who seek to reuse first and invent second. The patterns movement in the 1990’s totally got that idea.

So, it is a funny thing that the comments from Sunday are so negative when I am in fact arguing for their position. My book argued that the manager be placed in the position of the professional estimator, that all estimates include a margin for variation and that variation is measured from the historical data recorded in a work item tracking tool. Where there is subtlety in my argument, it relates to what analysis really is. It is a breakdown of what is to be built. An outline of the knowledge to be created. It is part of software systems engineering.

If we compare that notion to estimation in XP, we see something quite different. Wet we can estimate based on user stories - a requirements method that until Mike Cohn came along and tried to standardize the approach for creating stories was based on  vague prose descriptions subject to extreme variation - or developer tasks. Tasks define activities to be performed. They are developer focused and quantify time and effort to be expended. They aren’t, generally speaking, deliverable focused and don’t help to define the detail of what is to be built or the knowledge that is to be created.

At the end of the day, the customer takes delivery of a piece of working software, that has an architecture and a set of functionality. They don’t take delivery of the plan that created that software, or the estimates that enabled the construction of the plan. Estimation is “muda.” It isn’t something the customer values and will use (beyond the concept that they might care about risk management during development.) By definition then, estimation is muda! The outstanding question is whether estimation is necessary waste or pure waste. It is almost certainly necessary waste. It is hard to be effective without estimates and a workable plan. Nevertheless, even necessary waste should be minimized in a process. It’s an overhead! My argument then is simple - if you are going to spend any time on planning, let that planning focus on analysis and the creation of knowledge that can be delivered to the customer. Use as lightweight a technique as possible to generate the estimates from the analysis. That technique is agile estimating.

Assuming that developers are a capacity constrained resource, it is desirable to keep them focused on creating deliverable knowledge that can be turned in to working code. Using others (call them ‘professional’ if you want) to do the estimating, is a classic exploitation step that maximizes throughput from the capacity constrained developer resources. I presented a degenerate and extreme example of this technique in my XIT SE case study. Other situations won’t be so extreme but the principle applies. Why waste precious developer resources having them generate estimates when another non-constraint can do the job equally well or better?

Sunday, December 18, 2005

Estimation versus Analysis

I’ve noticed some confusion over the difference between estimation and analysis. When I’ve suggested stop estimating and focus on analysis some people have struggled to understand the difference. Again when I suggested all estimates are muda,  but continued to encouraged the use of analysis, some people struggled to understand the difference. So I decided to look up the dictionary meaning for clarification.

Verbs first…

to estimate - to calculate approximately (the amount, extent, magnitude, position, or value of something) [Look it up]
to analyze - to examine methodically by separating into parts and studying their interrelations [Look it up]

Now for the nouns,...

estimate - a tentative evaluation or rough calculation, as of worth, quantity, or size; a statement of the approximate cost of work to be done, such as a building project or car repairs [Look it up]
analysis - the separation of an intellectual or material whole into its constituent parts for individual study; the study of such constituent parts and their interrelationships in making up a whole. [Look it up]

It is clear that estimation is about quantification of cost while analysis is about the decomposition and integration of a set of parts. Analysis adds value by creating knowledge about what components we need to create and how they integrate together to synthesize a whole solution. Estimation merely quantifies the cost. I believe that in our industry people are lousy at estimating. And that this may indeed be true of all knowledge work. Estimating knowledge work is hard. It’s even harder without proper analysis. However, reliable and repeatable analysis techniques that, though they exhibit variation and are seldom used perfectly, significantly contribute to our understanding and planning of software development and project delivery, do exist, and should be used. Analysis reduces variation and makes plans more accurate. Estimation does neither of these things. Agile estimating on the other hand uses historical productivity data and uses the output of analysis to provide a reliable estimate with a buffer for variation. The result is a plan that is more accurate with time spent on value adding, knowledge creating, decomposition and integration rather than inaccurate, non-value added cost quantification.