Wednesday, October 12, 2011
Understanding the Process of Knowledge Discovery
Since 2007 when I first published on the Kanban Method as we recognize it today, I have struggled to articulate how to get started and describing the thing around which we hang a kanban (WIP limited pull) system. The first of the five core practices was described as “visualize workflow” in book. In some subsequent articles and online publications I and Janice Linden-Reed played around with alternatives. We tried borrowing the term “value stream” from Lean, and later got seduced by the intellectual objection that knowledge work isn’t a linear process and we should borrow Clayton Christensen’s term “value creation network.” The publication of that particular article on Kanban (which I refuse to link as I don’t want its search engine ranking improved) represented, perhaps, an all-time low in the struggle to articulate how to do Kanban. Recently, I attended Lean Kanban Benelux 2011 and was inspired by Michael Kennedy, an expert in Lean Product Development, to adopt his language of a “knowledge discovery process.”
What exactly is the problem?
I’ve perceived for more than a decade that specialization in software development was often an emergent property of 20th Century management approaches. Specialization into analysts, designers, coders, and testers would lend itself to higher utilization metrics, greater efficiency, and perhaps economy scale through efficient analysis, design, coding or testing of large batch sizes of knowledge work-in-progress. It’s this large batch size, efficiency-oriented, software development life cycle process that is often referred to as “waterfall.” My interest was in focusing on flow through the development process and reducing lead times. I wanted to decouple the lifecycle of the knowledge work(-in-progress) from the people performing the work. I also wanted to find language that removed any implications of a batch transfer such as “phase.” Finding suitable language has been challenging.
In truth I’ve struggled with the language to describe how software emerges from an idea to working code since I authored my first book in 2002. Page 18 of Agile Management describes stages of transformation. I was never happen with that terminology but it was the best I could do back then. Terms such as “phases” already carried baggage in software development.
More recently Chris Matts had suggested that software development was an information arrival process. This was inspired by his focus on Real Option Theory and the need for information to arrive in order for decisions to be made. I liked Chris’ ideas and felt it offered some improvements. However, I found that many people didn’t connect with the concept of information arriving. It seemed to them they had to make the information appear, it didn’t arrive of its own accord. However, the idea that software development was a process where we went from 0% information about a feature or function to 100% information which represented a fully working piece of code, was attractive.
Recently in my classes I’ve been asking people to map workflow by identifying state changes in the work by asking themselves “what is the dominant activity for discovering information (about the work item)?” And to consider that creation of software is an “punctuated information arrival process molded by the dominant information discovery activity at any given time.” When we find that a particular activity is no longer yielding new information, we tend to switch to a new means of generating new information, the inflection point when we switch activities indicates a state change in the work, for example, it is analyzed, or designed, or coded.
Then I watched Michael Kennedy use the term “knowledge discovery process,” and go on to describe the notion that product development was the process of going from 0% knowledge about an idea to 100% knowledge of a finished product ready for delivery to a customer. Given the historic background and my search for a simple way of expressing the flow of software development, Kennedy’s language seems like the perfect fit. Software development, for any one work item (user story, use case, feature, function, requirements) is a process of knowledge discovery, punctuated by changes in the dominant knowledge discovery activity. Hence, we can “analyze a problem” to discover knowledge, and when we reach a point of diminishing returns we can do something else, such as “design tests to validate working functionality against our understanding of the problem”, until we reach a point of diminishing returns and perhaps we choose a new activity such as “writing and compiling code.” Any software development process, no matter how iterative and incremental in its nature, can be described in these terms. Such a description is devoid of any implications of specialization of labor, hand-offs between individuals or batch transfers.
New Guidance on Visualizing Workflow
So now we can articulate a very clean and clear definition for mapping workflow when starting a Kanban initiative and designing the initial kanban system. Treat the process of creating a finished piece of work, of a given type, as a process of knowledge discovery, where we go from no knowledge (idea without substance) to complete knowledge (ready for delivery). Ask “what is the sequence of dominant knowledge discovery activities?” and draw that sequence. Use this to define the states within the workflow. Each of these states will then be visualized on our board or electronic tool. We can then define WIP limits for individual states or sets of states within the workflow.
And what of Value Streams and Value Creation Networks?
While I understand a number of Lean and Agile consulting firms are having some success with “value stream mapping” and are actively selling this to their clients, I personally never intended for this to creep into Kanban or to be part of Lean initiatives with software development and IT groups. I am naturally leery of borrowing any specific practices and tools from manufacturing industry. Such tools are often context specific and rooted in the physical nature of the work. Our work is invisible, virtual and revolves around knowledge and information not physical things. While Lean principles make sense for us, I am skeptical about tools.
My observation is that those mapping value streams focus on the hand-offs between people and departments and not on the natural progression of the work. I worry that this will lead to a focus on managing people (or groups of people such as departments) and not on managing the work. The great paradigm shift is that in Lean we are managing the work and allowing the people to self-organize to deliver against customer demand. I noticed that some consultant have picked up the Lean tools and put them to direct use with software teams, creating diagrams for value streams that look like the bones of fish picked clean on the plate. The truth is that value streams are perhaps the one area where physical systems are actually much more complicated that knowledge work. Mapping the value-chain delivery of components, sub-systems, assemblies, and the economic batch sizes for such deliveries, for a production system is often very complicated while analysis->design->code->test will generally approximate whatever we might be doing in a software development context.
Some consultants have argued with me that Value Stream Mapping has enabled them to work with clients to find improvement opportunities that have been immediately actionable. This is great news. It seems that any activity that encourages collaborative analysis of process problems and drives consensus opinions about changes, is a good thing. If this is working for you I wouldn’t suggest you stop doing it. However, David J. Anderson & Associates are unlikely to be offering Value Stream Mapping as a service nor will it be included in any of our training material.
The push for adoption of the term “value creation network” came from those, rightly, objecting to the idea that a stream implied a linear flow between people and that knowledge work flowed through a complex network of people before completion. That is we wanted to capture back flows and loops and iterative development then a stream was the wrong metaphor, a network was a stronger match. This is indeed correct. However, I feel that a process of knowledge discovery, mapping the dominant knowledge discovery activities as a sequence, complete avoids the issue.
Hence, I believe that value stream mapping and value creation network mapping are a potential source of waste. Mapping workflow doesn’t need to be difficult and it shouldn’t take too long. Mapping workflow should be one of the simplest and fastest parts of getting started with Kanban. For each work item type ask the simple question, “What activities do we perform to discover knowledge to create working software?” Then ask, “Which activity dominates at any given stage in the development of the software?” and sequence these to provide a framework around which to design a board and kanban system. Going forward, we’ll be aligning our language with Michael Kennedy’s. This change will also be reflected in a 2nd edition of the Kanban book. [Note: I have no firm plans or schedule for a 2nd edition at this time.]