Design Doesn't Emerge from Code

I know a lot of people who are transitioning to Agile or already following Agile development methods. Almost all of them are using something based on Scrum at the core, mixed with common XP practices like Continuous Integration and refactoring and automated unit testing – pretty much how Mike Cohn says things should be done in his book Succeeding with Agile.

Emergent Design in Scrum and XP

But none of them are doing emergent design as Cohn describes it, or as Kent Beck explains how design is done in Extreme Programming: trying to get away without any upfront design and architecture work, coding features right away and relying on test-first development, refactoring and technical spikes to work out a design on the fly, one week or two weeks at a time.

“For the first iteration, pick a set of simple, basic stories that you expect will force you to create the whole architecture. Then narrow your horizon and implement the stories in the simplest way that can possibly work. At the end of this exercise you will have your architecture. It may not be the architecture you expected, but then you will have learned something.” Kent Beck

You don’t need upfront architecture and design?

Maybe it’s because everyone I know is working at scale – building big enterprise systems and online systems used by lots of customers, systems that have a lot of constraints and dependencies. Many of them are working on brownfield projects where you need to understand the existing system’s design and implementation first, before you can come up with a new design and before you can make any changes. Performance-critical, mission-critical systems in highly-regulated environments.

Emergent, incremental design doesn’t work for these cases. And it doesn’t scale to large projects or any project that has to be delivered along with other projects and that has specified integration points and dependencies – which is pretty much every project that I've ever worked on.

Bob Martin, another one of the people who helped define how Agile development should be done, thinks that this incremental approach to design is, well…

“One of the more insidious and persistent myths of agile development is that up-front architecture and design are bad; that you should never spend time up front making architectural decisions. That instead you should evolve your architecture and design from nothing, one test-case at a time. Pardon me, but that’s Horse Shit.”

Martin goes on to say that

“there are architectural issues that need to be resolved up front. There are design decisions that must be made early. It is possible to code yourself into a very nasty cul-de-sac that you might avoid with a little forethought.”

Architecture and Design in Disciplined Agile Delivery

The way that most people that I know approach Agile development is better described by Scott Ambler in Disciplined Agile Delivery, a model for scaling Agile to larger systems, projects and organizations. As Ambler’s research shows, almost all teams (86%) spend at least some time (on average a month or more) on upfront on planning, scoping and architecture envisioning – what he calls the “Inception Phase” (borrowing from Rational’s Unified Process) or what most others call “Sprint 0” or “Iteration 0”.

This is time spent to understand the scope of the system at a high-level at least, and the constraints and dependencies that the project needs to work within. Time to model the main chunks of the system and their interfaces, and to choose a technical direction to start with.

Upfront architectural and design work doesn't have to take a lot of time. As Ambler points out, for many teams (except for some startups), a lot of architectural decisions have already been made for you:

“In practice, it’s likely you won’t need to do much initial architectural modeling: a large majority of project teams work with technical architecture decisions that were made years earlier. Your organization will most likely already have chosen a network infrastructure, an application-server platform, a database platform, and so on. In these situations your team will need to invest the time to understand those decisions and to consider whether the existing infrastructure build-out is sufficient (and if not, identify potential improvements).”

It’s when you have a real greenfield development project, when you don’t have anything to leverage and you’re doing something completely new, that you should spend more time on upfront thinking about design – not less.

Can you “be Agile” without Emergent Design?

Of course you can. Bob Martin points out that there’s nothing in “Agile Development” that says that you shouldn't do design upfront – as much design as you need to for the size of the system that you are building and the environment that you are working in.

You can and should do iterative, incremental design and development starting with a plan of where you are going and how you think that you are going to get there. As you go along and prove out your design and respond to feedback and deal with changes in requirements, this is where incremental design actually does come into play – handling changes in direction, filling in gaps, correcting misunderstandings. The design will change and maybe become something that you didn't expect. But you need a place to start from – designs don’t just emerge from code.