Friday, April 15, 2005

Driving Variation Out

I’ve been asked by email to publish some more material on driving variation out of the constraint. Specifically I was asked for software development and FDD examples but I’ll get to those in a future post. First of all I’d like to look at one of Seattle’s famous bottlenecks - the SR-520 freeway bridge across Lake Washington linking Seattle with its “east-side” suburbs of Bellevue, Kirkland and Redmond.

Anyone who works at Microsoft’s Redmond campus and lives in the city of Seattle commutes across the lake on the SR-520. Equally suburban dwellers from the north east-side take it in the opposite direction to commute to Seattle. On an evening when there is a baseball game or a basketball game in the city, it can be jammed for 6 miles from the lake back to the Redmond campus. It’s a notorious bottleneck every day morning or evening. So how would we drive variation out of this bottleneck?

The throughput of the SR-520 has a maximum of 60 cars per minute. That number is limited by the takt time (the time difference between moving cars) of 2 seconds. There are 2 lanes. So the maximum throughput is 30 cars per lane per minute. The 2 seconds comes from a well buffered “thinking time” for braking. It’s the minimum safe time between vehicles. Empirical study of traffic has shown that maximum throughput is usually achieved at about 40 miles per hour. Faster speeds add extra variation to the system because not everyone can react fast enough and the result is that at faster speeds, a longer takt time between vehicles or greater variation in the gaps occurs. Below 40 miles per hour, maintaining a 2 second gap is difficult because the forward velocity is so slow. Below 20 miles per hour, it becomes impossible.

It’s worth observing that takt time controls throughput and not velocity! Velocity affects lead time. At 40 miles per hour, the 4 miles of the narrowest part of the SR-520 on the east-side would take 6 minutes to navigate. Equally, the work-in-process inventory of cars on the freeway does not affect throughput directly. The goal to maximize throughput, therefore, is to find a way to maintain precisely 2 seconds between vehicles.

So we must look at what causes variation in the gaps between vehicles. Some causes will be - driver skill, attention span, eyesight, weather conditions, speed combined with reaction time of individual drivers, and lane switching. It’s clearly hard for the traffic management people to control many of these but some they could affect. For example, lane switching. Why not simply ban it for the whole 4 mile stretch? Place double white lines between the lanes for the whole 4 mile stretch! This would have a significant effect. When the freeway becomes overloaded with too much inventory and speeds fall to a crawl, lane switching as people try to find the faster moving lane, actually causes things to get much worse. Eliminating lane switching is a way to drive variation out of the constraint.

What could be done to overcome some of the more human variations such as reaction time, eyesight and skill level? Well technology exists and has been tested by Mercedes on German autobahns to use ultrasonic or laser technology to measure the distance between vehicles. Such technology could be used like cruise control to maintain the 2 second gap between vehicles. Naturally, every vehicle using the bridge would need to be fitted with it. The combination of these two ideas would probably treble the throughput on the SR-520 at peak times. It’s so compelling, it’s a wonder that the employers on the east-side don’t sponsor it

However, such technology is not full proof and accidents will still happen and an accident (or special cause variation) is devastating to throughput. What else could be tried to reduce the likelihood of a special cause variation?

Well ultimately, you take people out of their vehicles and put them in larger ones with a scheduled timetable running on rails. Anyone for a monorail extension across the lake?

Tuesday, April 12, 2005

Samurai or Artisan?

In traditional Japanese society, there are 4 classes of people: samurai; farmers; artisans; and merchants; in that order of importance or rank. Actually there is a fifth class - the untouchables who are also unmentionable. This group includes shoemakers and leather tanners who are not classed as artisans. The Japanese are very particular about anything to do with feet. Feet after all walk the dirty ground and the shoes covering them are therefore unclean.

[A brief tribal aside: one of Ray Immelman’s 22 tribal attributes is that a strong tribe knows how it compares to the untouchables. To make the unified Japanese tribe of Tokugawa Shogunate strong, it was therefore necessary to have an untouchable class in society.]

I’ve been wondering how the Japanese view software engineers. Are they samurai or artisans? Technically, any knowledge worker makes the samurai grade. Software engineers have a university degree. They are educated. They are part of a profession. All professional occupations are samurai jobs. By the 19th Century, Japan was so peaceful and civilized that the samurai were no longer needed as warriors. The modern samurai of his day was a doctor or other professional. Meanwhile, in todsay’s world, those ceremonial swords - the short and the long - would both come in handy for hacking up source code and refactoring it However, the reality of much software development and a key argument from some of the agile community has been that software development is a craft. That would put it clearly in the artisan category.

So, if the Von Neumann architecture had been around in the 1840’s before the American forced regime change in Japan, would the Shogun have viewed the programmers as members of the samurai? Would they have been rich, well dressed, middle class land owners with servants, horses and vacation homes in the country, licensed to carry swords in public, and revered by ordinary members of society as their betters? It’s interesting to compare which of these attributes apply today and which don’t - “So, you work at Microsoft. Tell me, what is it about software engineering that has got anything to do with being a professional engineer?”, a Boeing employee.

Monday, April 11, 2005

Imperial Palace Card System

When you visit the National Garden in the Imperial Palace grounds in Tokyo, they give you a plastic entrance card on entry. You return the card on exit at either the North or East gates of the palace. On first observation, it seems like a security scheme to insure that no visitor is not  accidentally forgotten and remains in the grounds after 4.30pm. However, I wonder how well this scheme would work. Let’s consider the variation in it.

On entry our guard couldn’t decide if we were a party/family of 5 or 6. The 3 month old in the carrier around my neck ought to count but was it really necessary. As we shuffled through so did several other individuals and families. Had he really counted everyone accurately? On exit, it might have been easy to explain that the 3 month old didn’t have a card whilst we secretly kept it as a souvenir (which we didn’t, everyone in the party except me being a good upstanding Japanese citizen). I’m left wondering what the shrinkage of cards is each day? And how much search effort around the National Garden is required to be sure that the missing cards walked out the facility and aren’t hiding in the bushes.

As a security system, I think the card system has more variability than would ideally be desired.

There is a second, simpler but less obvious reason for the cards - it’s a Kanban system. The cards simply limit the inventory of people visiting the gardens to keep them from getting overcrowded. I strongly suspect this is the real reason. When the cards at the entrance run out, then new visitors need to wait until someone leaves. This is only likely to happen on very peak times such as New Year, the Emperor’s Birthday and so forth. The Kanban solution has much lower susceptibility to variability. It doesn’t matter if there is a small amount of shrinkage on an individual daily basis because it’s a fail safe arrangement. So what if a few visitors have to wait a little longer to gain admission and the overall maximum inventory is reduced from shrinkage. The Kanban idea is much more variation tolerant.

I’m currently thinking a lot about Kanban systems and software engineering - after Don Reinertsen challenged me with it recently. He believes that the work I’ve been doing with cumulative flow diagrams will enable a Kanban system in tooling. The question is, what would be the benefit of such a system. It would certainly be easy to extend Visual Studio Team System to report inventory levels at different work item states and to limit the creation of new work items or movement of work items from state-to-state according to some Kanban limit. BUT, and this is the real question, for what purpose? I need to find a team willing to experiment with this idea manually and compare it against the drum-buffer-rope solution I have been evangelizing to see whether it is a flyer or not. Don’t expect an answer to this question anytime soon.

Sunday, April 10, 2005

Sushi Lunch

Saturday 9th April was my birthday. One that put me fairly and squarely in the “late thirties” category (san-ju-hatuchi to be precise). In honor of this occasion, my in-laws treated the whole family to a sushi lunch at a serious but not overly pricey sushi restaurant in the Hibiya district of Tokyo. Hibiya is a financial office district to the North East corner of the Imperial Palace. As it was hanami season and the palace gardens were busy with visitors, the restaurant which was no more than a sweet 5 iron over the moat into the palace grounds, was also busier than usual for a Saturday. However, this seasonal variation must be considered as common cause rather than special cause because it is entirely possible to anticipate that any Saturday during hanami, the restaurant will be busier.

On arrival we were told that we could not be seated - there was a constraint . The constraint was being protected by a capacity buffer of seats outside the door. We sat down. The menu was brought to us but strangely no one came back to take our order. Eventually we were seated inside at our table and a waiter arrived to take our order. My brother-in-law had placed some serious effort into selecting the pieces to be ordered and the numbers of each piece including some special treats for me as the birthday boy. The order was placed and we waited and waited. Meanwhile, the almost 3 year old in the corner seat was getting hungry and was eager with anticipation for the arrival of her California roll.

A plate of sushi arrived. No California roll!

On further inspection, it appeared that only half (han-bun) of our order had been fulfilled. An enquiry was made. A very polite waitress explained that the sushi chefs were fully loaded (a capacity constrained resource) and that in order to keep all the guests happy and provide food in a reasonable lead time, they were filling only half the order initially. The sushi chefs were self-organizing. They were presented with the order list and could burn it down as they saw fit. So an essentially randomly chosen selection representing half of our order had been delivered. Fair enough! Small batch sizes. Incremental delivery. Shorter lead times. It all made sense.

However, the vocal toddler was annoyed and frustrated. She started to chant, in English thankfully, I WANT SUSHI! I WANT SUSHI!

This was a serious sushi restaurant. You could tell this because the only condiment on the table was soy sauce. The sushi was served with ginger but no wasabi. This is to prevent uncouth foreigners from insulting the sushi chef by adding extra wasabi. The chef has expertly selected just the right amount of wasabi to enhance the flavor of each piece. Why add more? However, the uncouth do have an outlet, they can always dip pre-prepared pieces such as unagi (grilled eel which already has soy sauce) into the soy sauce - henna gaijin!

So we offered our unhappy customer the tamago (egg) sushi. She wrongly identified the egg as cheese and said, “I don’t want cheese!” So she dismantled the sushi, offered the “cheese” to daddy and ate the rice!

Eventually, the second plate of sushi arrived. No California roll!

This was a serious sushi restaurant! They could not or would not make us Kariforunia maki! A lot of bowing came with this explanation. The toddler burst into tears and wept uncontrollably to the consternation of the waitress who could not comprehend what had just happened. My brother-in-law skillfully negotiated an alternative order and asked that they expedite it.

So now other customers order were being delayed whilst they processed our expedite order.

Serious sushi eaters, eat their rolls (maki) after the individual pieces. The chefs had chosen to make any rolls we ordered last. As they were self-organizing and burning down our order list, the information that they could not or would not make the California roll had not been forthcoming until the last moment. Hence, the surprise!

So what is the moral of this story? Iterative incremental delivery and self-organizing burn-down aren’t enough! There has to be some analysis of the requirements and an attempt to understand the true customer needs - to understand the customer’s definition of quality. Based on this, the priorities should be set for the incremental deliveries. There should be a commitment to the customer - a promise made - and it should be honored.

In addition to the quality problem, this restaurant had a broken organizational structure and poor separation of responsibilities. The “Anderson lunch project” should rightly have been the responsibility of the waiter who should have been playing the project manager role (and maybe the program manager role). The waiter should have analyzed our requirements and understood our priorities. This should have been communicated to the chefs and the order of production of our sushi should have been negotiated against the competing orders at the time. The sushi chefs should have been purely responsible for the production of sushi. They should not have had any project management, program management or scheduling responsibility. After all, they had no direct contact with the customer and as the system’s capacity constrained resource, they should not have been wasting sushi making capacity trying to do anything else. (Actually, they weren’t wasting capacity, they simply weren’t doing a job that needed to be done. This was the root cause of the quality problem.)

[We will see when I get back to discussing Ray Immelman’s work that he agrees with my separation of functional capacity against project flow responsibilities but for tribal reason rather than variation and constraint reasons].

All of this goes to show that quality is not a given in Japan. They might be great at making cars but many of the management lessons have not transferred elsewhere. No end of bowing can make amends to a 2 year old whose expectations were disappointed.

Hanami is non-deterministic

Before I get back to describing Immelman’s model for organization and management based on tribal behavior, I thought I’d make a few observations from my trip to Japan.

The Japanese season of Hanami (cherry blossom viewing) is taking place in Tokyo this week. From a management perspective, this means an excuse to organize a morale boosting party for your staff. Here’s how to do it. Select the junior-man (the Japanese have a word for this) from the office. At this time of year, he (or she) will be a fresh college graduate with a new suit, shiny shoes, and a stunned, deer-in-the-headlights, look about him. Dispatch this individual with a large tarpaulin to Ueno Koen (or another nearby park open after dark) with orders to stake out a territory and under no circumstances should he leave his guard post. If you’re feeling generous then send a couple of his friends to keep him company, fetch water and sandwiches. It will be a long day of guard duty. Now have your assistant source a karaoke machine - probably from the cupboard under the printer and insure the batteries are charged and send someone trustworthy to buy the beer. Show leadership by leaving the office before 6.30pm and ask everyone to follow you to the park. Maybe let your hair down - loosen that tie a little - heck, maybe even take it off. [They’re snappy dressers in Tokyo]. There will be a fun evening ahead for everyone - and the Japanese really know how to party and enjoy themselves when you put a microphone in their hands. Lastly, as a manager, it would be responsible to wind things up in time to insure that everyone can catch their train home.

So here is the problem with Hanami. It’s not like the Emperor’s Birthday - you just can’t schedule it in your calendar. The cherry trees bloom when it suits them dependent on the winter weather conditions and the arrival of Spring. Yes, there is variation! The thing is, that to really be enjoying the spirit you have to hold the party under the trees whilst the cherry blossom is falling around you - not before and not after. The whole idea is to celebrate the fragility and shortness of life and its beauties. The window of opportunity is only a few days long each year.

So when the Anderson’s (yep, that’s me in the Asakusa Temple gardens last week with the junior carrier of the Agile Management gene code) go to Japan for Hanami, we have to make a guess - airline schedules are deterministic (more or less). So we book for the first week in April. And then we set our expectation that we will probably get to see some cherry blossom either blooming or falling but probably not both and maybe neither.

In our real home lives, we know how to deal with variation. We know how to set expectations appropriately. We accommodate variation like it was natural because it is. Why do we struggle to do so in our professional lives? So, did we get to see cherry blossom. Yes, we did! Did we see it falling? No, we didn’t. Did we get to party under the stars, petals falling around us. Nope! Did our vacation meet specification? Yes, it did! Because that specification had variable expectations built into it.