Architecture-Breaking Bugs – when a Dreamliner becomes a Nightmare

The history of computer systems is also the history of bugs, including epic, disastrous bugs that have caused millions of $ in damage and destruction and even death, as well as many other less spectacular but expensive system and project failures. Some of these appear to be small and stupid mistakes, like the infamous Ariane 5 rocket crash, caused by a one-line programming error. But a one-line programming error, or any other isolated mistake or failure, cannot cause serious damage to a large system, without fundamental failures in architecture and design, and failures in management.

Boeing's 787 Dreamliner Going Nowhere
The Economist, Feb 26 2013

These kinds of problems are what Barry Boehm calls “Architecture Breakers”: where a system’s design doesn't hold up in the real world, when you run face-first into a fundamental weakness or a hard limit on what is possible with the approach that you took or the technology platform that you selected.

Architecture Breakers happen at the edges – or beyond the edges – of the design, off of the normal, nominal, happy paths. The system works, except for a “one in a million” exceptional error, which nobody takes seriously until a “once in a million” problem starts happening every few days. Or the system crumples under an unexpected surge in demand, demand that isn't going to go away unless you can’t find a way to quickly scale the system to keep up – and if you can’t, you won’t have a demand problem any more because those customers won’t be coming back. Or what looks like a minor operational problem turns out to be the first sign of a fundamental reliability or safety problem in the system.

Dreamliner is Troubled by Questions about Safety
NY Times, Jan 10, 2013

Finding Architecture Breakers

It starts off with a nasty bug or an isolated operational issue or a security incident. As you investigate and start to look deeper you find more cases, gaping holes in the design, hard limits to what the system can do, or failures that can’t be explained and can’t be stopped. The design starts to unravel as each problem opens up to another problem. Fixing it right is going to take time and money, maybe even going back to the drawing board and revisiting foundational architectural decisions and technology choices. What looked like a random failure or an ugly bug just turned into something much uglier, and much much more expensive.

Deepening Crisis for the Boeing 787
NY Times, Jan 17 2013

What makes these problems especially bad is that they are found late, way past design and even past acceptance testing, usually when the system is already in production and you have a lot of real customers using it to get real work done. This is when you can least afford to encounter a serious problem. When something does go wrong, it can be difficult to recognize how serious it is right away. It can take two or three or more failures before you realize – and accept – how bad things really are and before you see enough of a pattern to understand where the problem might be.

Boeing Batteries Said to Fail 10 Times Before Incident
Bloomberg, Jan 30 2013

By then you may be losing customers and losing money and you’re under extreme pressure to come up with a fix, and nobody wants to hear that you have to stop and go back and rewrite a piece of the system, or re-architect it and start again – or that you need more time to think and test and understand what’s wrong and what your options are before you can even tell them how long it might take and how much it could cost to fix things.

Regulators Around the Globe Ground Boeing 787s
NY Times, Jan 18 2013

What can Break your Architecture?

Most Architecture Breakers are fundamental problems in important non-functional aspects of a system:

• Stability and data integrity: some piece of the system won’t stay up under load or fails intermittently after the system has been running for hours or days or weeks, or you lost critical customer data or you can’t recover and restore service fast enough after an operational failure.
• Scalability and throughput: the platform (language or container or communications fabric or database – or all of them) are beautiful to work with, but can’t keep up as more customers come in, even if you throw more hardware at it. Ask Twitter about trying to scale-out Ruby or Facebook about scaling PHP or anyone who has ever tried to scale-out Oracle RAC.
• Latency – requirements for real-time response-time/deadline satisfaction escalate, or you run into unacceptable jitter and variability (you chose Java as your run-time platform, what happens when GC kicks in?).
• Security: you just got hacked and you find out that the one bug that an attacker exploited is only the first of hundreds or thousands of bugs that will need to be found and fixed, because your design or the language and the framework that you picked (or the way that you used it) is as full of security holes as Swiss cheese.

These problems can come from misunderstanding what an underlying platform technology or framework can actually do – what the design tolerances for that architecture or technology are. Or from completely missing, overlooking, ignoring or misunderstanding an important aspect of the design.

These aren’t problems that you can code your way out of, at least not easily. Sometimes the problem isn't in your code any ways: it’s in a third party platform technology that can’t keep up or won’t stay up. The language itself, or an important part of the stack like the container, database, or communications fabric, or whatever you are depending on for clustering and failover or to do some other magic. At high scale in the real world, almost any piece of software that somebody else wrote can and will fall short of what you really need, or what the vendor promised.

Boeing, 787 Battery Supplier at Odds over Fixes
Wall Street Journal, Feb 27 2013

You’ll have to spend time working with a vendor (or sometimes with more than one vendor) and help them understand your problem, and get them to agree that it’s really their problem, and that they have to fix it, and if they can’t fix it, or can’t fix it quickly enough, you’ll need to come up with a Plan B quickly, and hope that your new choice won’t run into other problems that may be just as bad or even worse.

How to Avoid Architecture Breakers

Architecture Breakers are caused by decisions that you made early and got wrong – or that you didn't make early enough, or didn't make at all. Boehm talks about Architecture Breakers as part of an argument against Simple Design – that many teams, especially Agile teams, spend too much time focused on the happy path, building new features to make the customer happy, and not enough time on upfront architecture and thinking about what could go wrong. But Architecture Breakers have been around a lot longer than Agile and simple design: in Making Software (Chapter 10 Architecting: How Much and When), Boehm goes back to the 1980s when he first recognized these kinds of problems, when Structured Programming and later Waterfall were the “right way” to do things.

Boehm’s solution is more and better architecture definition and technical risk management through Spiral software development: a lifecycle with architecture upfront to identify risk areas, which are then explored through iterative, risk-driven design, prototyping and development in multiple stages. Spiral development is like today’s iterative, incremental development methods on steroids, using risk-based architectural spikes, but with much longer iterative development and technical prototyping cycles, more formal risk management, more planning, more paperwork, and much higher costs.

Bugs like these can’t all be solved by spending more time on architecture and technical risk management upfront – whether through Spiral development or a beefed up, disciplined Agile development approach. More time spent upfront won`t help if you make naïve assumptions about scalability, responsiveness or reliability or security; or if you don’t understand these problems well enough to identify the risks. Architecture Breakers won’t be found in design reviews – because you won’t be looking for something that you don’t know could a problem – unless maybe you are running through structured failure modelling exercises like FMEA (Failure mode and effect analysis) or FMECA (Failure mode, effects and criticality analysis), which force you to ask hard questions, but which few people outside of regulated industries have even heard about.

And Architecture Breakers can't all be caught in testing, even extended longevity/soak testing and extensive fuzzing and simulated failures and fault injection and destructive testing and stress testing – even if all the bugs that are found this way are taken seriously (because these kinds of extreme tests are often considered unrealistic).

You have to be prepared to deal with Architecture Breakers. Anticipating problems and partitioning your architecture using something like the Stability Patterns in Michael Nygard’s excellent book Release It! will at least keep serious run-time errors from spreading and taking an entire system out (these strategies will also help with scaling and with containing security attacks). And if and when you do see a “once in a million” error in reviews or testing or production, understand how serious it can be, and act right away – before a Dreamliner turns into a nightmare.

Moving forward from Failure

System failures at scale are inescapable, as I have talked about before in the context of designing systems for failure in high-scale computing and how to apply these ideas to enterprise systems. Failures are wasted if you don’t learn enough from them; if the way that you design and deliver application systems, and the way that you deploy and run these systems, don’t get better as a result. You have to find lessons in each failure and constantly move forward, and make the system, and the team, more resilient.

I’ve read and heard a lot of good stuff over the last year or so from the DevOps community on handling system failures: how to minimize the risk of failures through deployment planning, how complex systems fail, how to communicate failures effectively to stakeholders so that you don’t destroy trust, and how to understand failures through postmortem analysis, including Jacob Loomis’ essay “How to Make Failure Beautiful” in the Web Operations book and John Allspaw’s keynote on Advanced Postmortem Fu and Human Error 101 at this year’s Velocity conference.

Many of these ideas fit with my own experience, and fill in some important gaps – they extend beyond the concerns of operating high-volume Web sites to broader, general problems in system operations and systems engineering, and deserve a wider audience outside of the Web operations and Web startup worlds.

Postmortems and Root Cause Analysis

The keys to successful postmortems and Root Cause Analysis are deceptively simple, and therefore difficult to get right:

• get the right people together in a room.


• create the right environment for blameless problem solving: make it safe for people to be open and honest, don’t point fingers, focus on facts and solving problems and how to get better.


• postmortems are expensive and painful, and people want to get out of them as soon as possible. Don’t stop too soon, before the team has really understood the problems, and the solutions.


• don’t be satisfied with a single root cause – complex system failures usually have multiple causes if you dig deep enough.


• human error isn’t a root cause – it’s a symptom of something else that you’ve done wrong.

You can’t stop with Root Cause Analysis

Root Cause Analysis is important, but it is just the first step. Once you’ve reviewed a failure as a team and found your way to the root causes or at least identified some real problems, now you have to do something about it, beyond the immediate fixes and workarounds that the team made recovering from the incident.

There are straightforward things that you can do now – detective and corrective actions, quick fixes, low cost and low risk – so you do them. Fix a bug, plug a hole, add defensive coding, add some tests. Better logging and diagnostics and alerting, better error handling, better metrics – you can always do this stuff better – so that ops can see the problem coming or at least recognize it when it happens again. Better troubleshooting tools and training for ops.

Don’t stop with corrective actions either

But it’s still not enough to recover from a failure and patch the holes and tighten things up, or even build a better incident management capability. You have to make sure to find the lessons in each failure, really learn about why the failure happened, how it relates to other failures or problems that you have; and about how you dealt with the failure when it happened, and what you need to do to do a better job the next time. Then you have to act to reduce the risk and cost of failures, take steps to make sure that failures like this one, and even failures that aren’t like this one, don’t happen again. And that when the next failure does happen, that the team is more prepared for it, that you can recover faster and with less stress and impact to the business.

Some of these preventative actions are straightforward: training for developers and testers and ops, and better communications in and especially between teams, checklists, health checks and dependency checks, more disciplined change controls and deployment controls, and other safeties.

But most preventative changes, the ones that make a real difference, are deeper, more fundamental: fixing architecture, culture, organization, or management weaknesses. Fixing these kinds of problems takes longer, involves more people, and costs more. Making changes to the architecture – switching out a core technology or implementing a partitioning strategy to contain failures and to scale up – can take months to work out and implement. Organizational and management changes are hard because this directly impacts people’s jobs. Changing culture is even harder and takes longer, especially to make the changes stick.

You can’t improve what you can’t see – you need data

These problems are also hard to see and understand. It’s naïve to think that you can recognize or prove the need for fundamental changes to architecture or organization or management controls or culture from a postmortem meeting – it takes time and perspective and experience with more than one failure to see this kind of problem. And because the fixes are fundamental and expensive, they can be hard to justify, hard to get management and the business and your own people to buy-in.

You need data to help you understand what you’re doing wrong, what you need to change or what you need to stop doing – and what you’re doing right, what you need to keep doing, and to build a case for change. At last year’s Velocity conference, John Allspaw explained how to use metrics to find patterns and trends in failures, to understand what’s hurting you the most, or hurting you the most often, what’s working well and what’s not. To find out what changes are safe, and what changes aren’t; what failures are cheap and easy to recover from, and what failures can take the company down.

Metrics are important to help you see problems, and to help show if you are learning and getting better. Track the types and severity of failures, frequency of failures, your response to failures – time to detect and time to recover from failures; and the frequency and type and size of changes, and the correlation between changes and failures (by type and size of changes, by type and severity of failures). And make sure to track regressions – the number of times that you take a step backward.

Deciding what to fix, and what not to

Metrics can also help you to decide what you can fix and change, and what’s not worth fixing or changing – at least not for now. You need to recognize when a problem is a true outlier, an isolated case that is unlikely to happen again and that you can accept and move forward. It’s important that you can see the difference between a problem that is a one-of, and a first-of –an indicator of something deeply wrong, a fundamental weakness in the way that the system was built or the way that you work. You don’t want to over-react to outliers but you also don’t can’t treat every problem as unique and miss seeing the patterns and connections, the underlying root causes.

There is diminishing returns in preventing (or even reducing the probability) of some problems, trying to stop the unstoppable. Some problems are too rare, or too expensive or difficult to prevent – and trying to prevent these problems can introduce new complexities and new risks into the system. I agree with John Allspaw that there are situations when it makes more sense to focus on how to react and recover faster, on improving MTTR rather than MTBF, but you need to know when to make that case.

Moving forward from Failures is real work and needs to be managed

People don’t want to think about failures for long – they want to put the mistakes and confusion and badness behind them and move on. And the business and management want priorities back on day-to-day delivery as soon as possible. When failures happen, you need to act fast, get the most that you can out of the moment, make decisions and commitments and reinforce changes while the pain is still fresh.

But fixing real problems in the system or the way that people are organized or the way that they work or the way that they think, can’t be done right away. Building reliability and resilience and operability into how you work and how you think and how you build and test software and how you solve problems takes time and commitment and continuous reinforcement. Management, and the team, have to be made accountable for longer-term preventative actions, for making bigger changes and improvements.

This work needs to be recognized by the business and management and the team and built in to the backlog, and actively managed. You need to remind people of the risks when you see them missing steps or trying to cut corners, or falling back into old patterns of thinking and acting. You’ll need to use metrics and cost data to drive behaviour and to drive change, and to decide how much to push and how often: are you changing too much too often, running too loose; or is change costing you too much, are you overcompensating?

A Resilient system is a DevOps problem

Development can’t solve all of these problems on its own, and neither can Operations. It needs the commitment of development (and by development, I mean everyone responsible for designing, developing, testing, securing and releasing the software) and operations (everyone responsible for building and securing and running the infrastructure, deploying the code, and running and monitoring the system). This is a real DevOps problem: development and operations have to be aligned and work together, communicating and collaborating, sharing responsibilities and technology. This is one of the places where I see DevOps adding real value in any organization.

Dreaming in Code - A Failure in Leadership

Reading Scott Rosenberg’s Dreaming in Code gives you a sick feeling, the same sick feeling that you have watching a movie where the hero’s life is coming unraveled, or when you are involved in a project that is going nowhere fast, facing certain failure, and there is nothing that you can do to change the outcome. I made myself read it twice – there are some hard lessons for managing software development in this book.

Dreaming in Code tells the story of the failed Chandler project started by Mitch Kapor, software industry visionary and founder of Lotus Development Corp, currently Chairman of the Mozilla Foundation. Chandler began as an ambitious, “change the world” project to design and build a radically different kind of personal information manager, an Outlook-and-Exchange-killer that would flatten silos between types of data and offer people new ways of thinking about information; provide programmers a cross-platform, extensible open source platform for new development; and create new ways to share data safely, securely, cheaply and democratically, in a peer-to-peer model.

The project started in the spring of 2001. Because of Mitch Kapor’s reputation and the project’s ambitions, he was able to assemble an impressive team of alpha geeks. Smart people, led by a business visionary who had experienced success, interesting problems to solve, lots of money, lots of time to play with new technology and chart a new course: a “dream project”.

But Dreaming in Code is a story of wasted opportunities. Scott Rosenberg, the book’s author, followed the project for 3 years starting in 2003, but he gave up before the team had built something that worked – he had a book to publish, whether the story was finished or not. By January 2008, Mitch Kapor had called it quits and left the company that he founded. Finally, in August 2008, the surviving team released version 1.0, scaled back to a shadow of its original goals and of course no longer relevant to the market.

It is interesting, but sad, to map this project against Construx’s Classic Mistakes list, to see the mistakes that were made:

Unclear project vision. Lack of project sponsorship. Requirements gold-plating. Feature creep. Research-oriented development. Developer gold-plating. Silver bullet syndrome. Insufficient planning. Adding people to a late project. Overly optimistic schedules. Wishful thinking. Unrealistic expectations. Insufficient risk management. Wasted time in the “fuzzy front end”: the team spent years wondering and wandering around, playing with technology, building tools, exploring candidate designs, trying to figure out what the requirements were - and never understood the priorities. Shortchanged quality assurance… hold on, quality was important to this project. It wasn’t going to play out like other Silicon Valley startups. Then why did they wait almost 3 years before hiring a test team (of one person), while they faced continual problems from the start with unstable releases, unacceptable performance, developers spending days or weeks bug hunting.

The book begins with a meeting in July 2003, when the team should be well into design and development, where the project manager announces that the team is doomed, that they cannot hope to deliver to their commitments – and not for the first time. This is met with…. well, nothing. Nobody, not even Mitch Kapor, takes action, makes any decisions. It doesn't get any better from there.

This project was going to be different, it was going to be done on a “design-first” basis. But months, even years into the project, the team is still struggling to come up with a useful design. The team makes one attempt at time-boxed development, a single iteration, and then gives up. Senior team members leave because nothing is getting done. Volunteers stop showing up. People in the community stop downloading the software because it doesn’t work, or it does less than earlier versions – the team is not just standing still, they are moving backwards.

The project manager, after only a few months, quits. Mitch Kapor takes over. A few months later, the “fun draining out of his job”, he asks the team to redesign his job, and come up with a new management structure. OK, to be fair, he is rich and doesn’t have to work hard on this, but why start it all in the first place, why put everyone involved, even those of us who are going to read the book, through all of this?

The new management team works through a real planning exercise for this first time, making real trade-offs, real decisions based on data. It doesn’t help. They fail to deliver – again. And again with an alpha release. They then spend another 21 months putting together a beta release, significantly cutting back on features. By the time they deliver 1.0 nobody cares.

It’s a sad story. It’s also boring at times – the team spends hours, days in fruitless, philosophical bull sessions; in meaningless, abstract, circular arguments; asking the same questions, confronting the same problems, facing (or avoiding) the same decisions again and again. As the book says, “it’s Groundhog Day, again”. I hope that the writer may have not fully captured what happened in the design sessions and planning meetings – that, like the Survivor tv show, we only see what the camera wants us too, or happens to, an incomplete picture. That these people were actually more focused, more serious, more responsible than they come across.

The project failed on so many levels. A failure to set understandable, achievable goals. A failure to understand, or even attempt to articulate requirements. A failure to take advantage of talent. A failure to engage, to establish and maintain momentum. A failure to manage risks. A failure to learn from mistakes - their own, or others.

Most fundamentally, it was a failure of leadership. Mitch Kapor failed to hold himself and the team accountable. He failed to provide the team with meaningful direction, failed to understand and explain what was important. He failed to create an organization where people understood what it took to deliver something real and useful, where people cared about results, about the people who would, hopefully, someday, use what they were supposed to be building. And he gave up: he gave up well before 2008 when he left the company; he gave up almost from the start, he gave up when the hard, real work of building out his vision had actually begun.