Kanban, Scrum/XP and the Paradox of Constraints

Building and maintaining software is about confronting limits:

• Limits on the time that you have to get your work done.
  
• Limits on the money that you have to spend, on the talent that you can get.
  
• The limits of your technology platform, on its performance and reliability and cost of operations.
  
• Limits imposed by decisions that have already been made, decisions on architecture and design that limit how much you can change, how safely and how quickly.
  
• And you must maintain compatibility with legacy systems and interfaces to partners.
  
• You may be forced to work with certain suppliers and partners, even if they can’t meet all of your needs.
  
• You will have to meet hard, unmovable deadlines.
  
• You will have to deal with regulatory oversight and legal compliance dictates, and follow change control and change management controls that will slow you down and make your work more expensive.
  
• And in maintenance you have a running system, people rely on it to get their jobs done, and they expect it and need it to work a certain way.
  

All of these constraints box you in. They make your work more difficult and expensive. Project Managers are taught to identify and understand constraints and manage them as sources of risk and cost.

But there’s a wonderful paradox in constraints and limits that I enjoy thinking about:

Constraints take control away from you: by dictating, by forcing you to think and work a certain way, by limiting your options.

But…

Constraints help you to take control: by dictating, by forcing you to think and work a certain way, by limiting your options.

The story of Chandler in Dreaming in Code made a strong case for building software under constraints, as you follow the agonizing unraveling of a project that set out to solve too much, with too much money, too much time, and too much talent. By contrast, the book also profiled 37signals, the small team behind Ruby on Rails, which set out to build simple software with almost no resources. Their ethos is to “Embrace Constraints”:

“Constraints drive innovation and force focus. Instead of trying to remove them, use them to your advantage.”

This is the point of software development management frameworks like Scrum, Extreme Programming and Kanban: to place practical limits on the way that the team works, to force the team into a box, and so control the way that work is done.

Constraints in Scrum and XP

Scrum and XP force the team to work in fixed, short time boxes. Time boxes limit how much work that you can consider and take on at one time. You have to come up with a solution in a short time period, and while it may be a sub-optional solution (you probably would have thought of something smarter if you had more time), at least you get something done and can test the results. And there is a good chance that you will be able to build on what you have done so far, fill it in some more in later steps.

Time boxes serve as a hedge against perfectionism and gold plating and procrastination. Forcing people to work under time constraints helps fight off Parkinson’s Law: that work expands to fill the time available for its completion. It blows the whistle on people who don’t want to let go, who continue to redesign, rewrite, refactor, tune, polish and fiddle with code longer than they need to. And being pushed forward from one deadline to the next keeps you from over-thinking, from letting the problem space or solution space expand.

Doing work in small batches with fixed time limits helps to manage delivery risk. You can only get so far off track in a short time, and if it turns out that you got it wrong and you have to throw it all away, you aren’t wasting too much. And it shouldn’t take too long to understand why you failed, how to get back on track. Small decisions are easier and safer to make, and to unmake. And frequent planning gives you the flexibility to adapt to changing circumstances and take new information into account, to reset and re-prioritize.

With time boxes you are forced to work "in the small”, to think, really think, about how to get work done. It’s all about execution: who, what, when, what happens first, second. It creates a sense of urgency. And a sense of satisfaction, in seeing work done, in the feedback that you get.

In his book Succeeding with Agile Development Using Scrum, Mike Cohn makes the point that the pressure to deliver working software on a regular basis drives teams to get better at their work, to become more efficient, to adopt better technical practices and disciplines like automated testing and Continuous Integration, to get or build better tools.

And the time box becomes a commitment to management and to the customer, it forces your work to be more predictable and transparent. Your customers can plan around a regular delivery cycle, and make their own commitments.

Lean and Kanban: Constraining Work in Progress

I have spent the last few months looking into Kanban for software development, to see what we can learn from and use. Kanban, based on Lean Manufacturing principles, also constrains the amount of work that the team does, but in different ways. Rather than limiting the amount of time that the team has available to get their work done, you explicitly limit the number of things that the team can do at any point in time: the amount of Work in Progress. Each person works on one thing at a time (with maybe some small allowance for issues that could get blocked), following the idea that if small batches are good, then a batch size of one is optimal.

The team uses input queues and buffers between steps (for example: analysis, design, coding, testing, deployment) to smooth out work as it is being done:

• to minimize slack (over capacity) and stalls (under capacity);
  
• to account for variability in the work to be done (a fundamental problem with applying Kanban to software development: work that is easy to code, but hard to test; or hard to code but easy to test, and so on); and
  
• to adjust as the workload or the team changes, and as you learn more about how you are doing and how to do it better.
  

The goal is to keep the input queues and the buffers as small as possible, just big enough to keep the work constantly moving (a constant state of flow).

In Kanban, like in Scrum and XP, planning and prioritization is done just-in-time. The difference is that in a time boxed method, planning is done at the beginning of each time box; in Kanban, planning and prioritization are decoupled from the rest of the work. Prioritization can be scheduled at a regular interval, or done on demand, just before the work starts – again with a small amount of buffering to make sure that team members don’t stall waiting for work. You decide with the customer what to do next and qualify it, put it into the queue, and then work is pulled based on priority and the team member’s ability to do the job. This forces you and the customer to always be ready, to focus on what is important.

Planning further out is considered waste, since the situation may change and make your plans (like most plans) out dated.

You don’t accept more work than you can take on immediately. If you run into problems downstream, the queues will fill up, and the pipeline of work upstream will stall – there’s no point taking on more analysis and estimation work if the team is falling behind on building and testing and deploying code. This shines a bright light on inefficiencies and bottlenecks. As your work changes, and the bottleneck moves, you recognize this and adapt. And you constantly look for ways to eliminate unnecessary work, rework, delays and blocks – anything that stops you from getting work done as efficiently as possible.

Constraints as a Management Tool

Limiting the amount of work that the team can do at one time, either through time boxing or setting limits on work in progress queues, is a coarse management tool but an effective one.
I’ve worked with time boxing methods at different companies and come to rely heavily on this approach. Time boxing works, and today there are good tools and supporting practices that make time boxing easier to implement and sustain.

If you find it difficult to break your work into time boxes, if your work is highly interrupt-driven like in many maintenance and support shops, then Kanban offers an alternative control structure. I am not sure that I buy into its manufacturing control-line metaphor, and that all of these ideas map well from manufacturing to the highly variable work involved in building and supporting software. I think that there is more work that still needs to be done by the community, to make the fit cleaner and clearer. More people need to try it out, see what works, and whether it sticks. But I understand why Kanban could work. It makes explicit what many well-run teams already do: track and control and buffer work demands across different people on the team.

Whether you follow any of these methods is not important. What is important is that you understand constraints. Understand how constraints take control away, and how they can be used to take control back - how use them to your advantage.