Failure Isolation and Recovery: Learning from High-Scale and Extreme-Scale Computing

While I have been building business-critical enterprise systems for a long time, I haven't worked on high-scale cloud computing or Internet-scale architectures, with tens of thousands or hundreds of thousands of servers. There are some fascinating, hard problems that need to be solved in engineering systems at high-scale, but the most interesting to me are problems in deployment and operations management, and especially how to deal with failure.

With so many moving parts in high-scale systems, failures of one kind or another are common: disks (especially), servers and multiple servers, racks, networks, data center outages and geographic disasters, and database failures, middleware and application software failures, and of course human error – mistakes in configuration and operations. As you scale up, you will also encounter multiple simultaneous failures and failure combinations, normal accidents, more heisenbugs and mandelbugs, and data corruption problems and other silent failures.

These challenges are taken to extremes in petascale and exascale HPC platforms. The Mean Time to Failure (MTTF) on petascale systems (the largest super computers running today) can be as low as 1 day. According to Michael Heroux at Sandia National Laboratories, exascale systems, consisting of millions of processors and capable of handling 1 million trillion calculations per second (these computers don’t exist yet, but are expected in the next 10 years)

“will have very high fault rates and will in fact be in a constant state of decay. ‘All nodes up and running’, our current sense of a well-functioning scalable system, will not be feasible. Instead we will always have a portion of the machine that is dead, a portion that is dying and perhaps producing faulty results, another that is coming back to life and a final, hopefully large, portion that is computing fast and accurate results.”

For enterprise systems, of course, rates of component or combination failures will be much lower, but the same risks exist, and the principles still hold. Recognizing that failures can and will happen, it is important to:

    Identify failures as quickly as possible.

    Minimize and contain the impact of failures.

    Recover as quickly as possible.

At QCON SF 2009, Jason McHugh described how Amazon’s S3 high-scale cloud computing service is architected for resiliency in the face of so many failure conditions. He lists 7 key principles for system survivability at high-scale:

1. Decouple upstream and downstream processes, and protection yourself from dependencies upstream and downstream when problems occur: overload and spikes from upstream, failures and slow-downs downstream.

2. Design for large failures.

3. Don’t trust data on the wire or on disk.

4. Elasticity – resources can be brought online at any time.

5. Monitor, extrapolate and react: instrument, and create feedback loops.

6. Design for frequent single system failures.

7. Hold “Game Days”: shutdown a set of servers or a data center in production, and prove that your recovery strategy works in the real world.

Failure management needs to be architected in, and handled at the implementation level: you have to expect, and handle, failures for every API call. This is one of the key ideas in Michael Nygard’s Release It!, that developers of data center-ready, large-scale systems cannot afford to be satisfied dealing with abstractions, that they must understand the environment that the system runs in, and that they have to learn not to trust it. This book offers some effective advice and ideas (“stability patterns”) to protect against horizontal failures (Chain Reactions) and vertical failures (Cascading Failures).

For vertical failures between tiers, use timeouts and delayed retry mechanisms to prevent hangs and timeouts – protect request handling threads on all remote calls (and resource pool checkouts), and never wait forever. Release It! introduces a Circuit Breaker pattern to manage this: after too many retries, close-off the connection, back away, then try again – if the problem has not been corrected, close-off and back away again. And remember to fail fast – don’t get stuck blocking or deadlock if a resource is not available or a remote call cannot be completed.

In horizontal Chain Reactions within a tier, the failure of one component raises the possibility of failure in its peers, as the workload rebalances to overwhelm the remaining services. Protect the system from Chain Reactions with partitions and Bulkheads – build internal firewalls to isolate workloads on different servers or services or resource pools. Virtual servers are one way to partition and isolate workloads, although we chose not to use virtual servers in production because of the extra complexity in management – we have had a lot of success with virtualization for provisioning test environments and management services, but for high-volume, low-latency transaction processing we’ve found that separate physical servers and application partitioning is faster and more reliable.

Application and data partitioning is a fundamental design idea in many systems, for example exchange trading engines, where the market is broken down into many different trading products or groups of products, isolated in different partitions for scalability and workload balancing purposes. James Hamilton, formerly with Microsoft and now a Distinguished Engineer on the Amazon Web Services team, hilights the importance of partitioning for application scaling as well as fault isolation and multi-tenancy in On Designing and Deploying Internet-Scale Services, an analysis of key issues in operations management and architecture of systems at high-scale, with a focus on efficiency and resiliency. He talks about the importance of minimizing cross-partition operations, and managing partitions intelligently, at a fine-grained level. Some of the other valuable ideas here include:

    Design for failure

    Zero trust of underlying components

    At scale, even unlikely, unusual combinations can become commonplace

    Version everything

    Minimize dependencies

    Practice your failover operations regularly - in production

Read this paper.

All of these ideas build on research work in Recovery Oriented Computing at Berkeley and Stanford, which set out to solve reliability problems in systems delivered “in Internet time” during the dot com boom. The basics of Recovery Oriented Computing are:

Failures will happen, and you cannot predict or avoid failures. Confront this fact and accept it.

Avoiding failures and maximizing Mean Time to Failure by taking careful steps in architecture and engineering and maintenance, and minimizing opportunities for human error in operations through training, procedures and tools – all of this is necessary, but it is not enough. For “always-on” availability, you also need to minimize the Mean Time to Recovery (MTTR).

Recovery consists of 3 steps:

1. Detect the problem
2. Diagnose the cause of the problem
3. Repair the problem and restore service.

Identify failures as quickly as possible. Minimize and contain the impact of failures. Provide tested and reliable tools for system administrators to recover the system quickly and safely.

One of the key enablers of Recovery Oriented Computing is, again, partitioning: to isolate faults and contain damage; to simplify diagnosis; to enable rapid online repair, recovery and restart at the component level; and to support dynamic provisioning – elastic, or “on demand” capacity upgrades.

As continuing reports of cloud computing failures and other large scale SaaS problems show, these problems have not been solved, and as the sheer scale of these systems continues to increase, they may never be solved. But there is still a lot that we can learn from the work that has been done so far.

Secure Software Development: Awareness Training

I’ve been looking for a good basic awareness course on software security as an introduction for new developers and as a refresher for the rest of the team. A course that covers the important bases, and reinforces fundamental issues in building secure software. I prefer online training for something like this: if you have the discipline, online and on-demand training works because it is easier to schedule, people can go as fast or as slow as they like, and of course it is less expensive.

Stanford Software Security Foundations

Last year I took the Software Security Foundations course, the first course in Stanford University’s Advanced Computer Security certificate program:6 courses, all offered online, on-demand.

Foundations is a basic introduction to software security for developers and technical managers. This course was designed by (and principally delivered by)
Neil Daswani a former security program manager at Google, a Stanford alumni, and now a principal at Dasient, a web computer security startup.

The course is based on Mr. Daswani’s book Foundations of Computer Security: What Every Programmer Needs to Know: it is useful to have a copy of the book handy while going through the course. Foundations is a day’s worth of lectures made available as online videos for a 3-month period, with slides that can be downloaded for printing, and an exam at the end to ensure that you aren’t lazy about following the material. It covers the fundamentals of software security, including:

Security Principles

Authentication, authorization, access control, confidentiality, data integrity, non-repudiation, risk management, and secure system design principles including least privilege, fail-safe and defense-in-depth. Good, but no real surprises here.

Secure Programming

Buffer overflows, SQL injection, password management, cross site request forgery, cross site scripting, and common mistakes in crypto. There was good coverage of SQL injection and cross domain security threats, particularly XSRF. However I found the explanation of Cross Site Scripting confusing, even after reading through the section on cross domain security in the book – it’s not a straightforward problem, ok, but it can be explained in a simpler way. The examples and problems are biased slightly towards C/C++ programmers, but this should not present a problem for programmers working in other environments. It covers most of the important bases with the exception of input validation, which needs more attention.

Introduction to Cryptography

A walkthrough of symmetric encryption and public key cryptography, and then a brief discussion of advanced research in cryptography from Stanford Professor Dan Boneh, an expert in applied cryptography. The explanation of cryptographic primitives was especially lucid, and, well beautiful: I didn’t know that cipher block chaining could be beautiful, but it is. Some of the more advanced issues in cryptography are covered at a cursory level only (not a lot of it will stick with you if you aren’t already familiar with this subject), and you are referred to other courses offered in the program if you want to get a good, basic understanding of how to use cryptography and secure protocols.

SANS Software Security Awareness

In January I checked out the On Demand Software Security Awareness course from the SANS Institute.

I have had success with other courses from SANS before, classroom and On Demand. In particular the course SANS Security Leadership Essentials for Managers is excellent: a challenging and exhaustive 6-day course covering pretty much everything a technical manager would need to know about IT security from technical, project management, risk management, strategic, and legal and ethical perspectives.

Software Security Awareness is a short, 3-hour survey course, offered online as recorded audio lectures and a slide show. SANS prints and binds a copy of the slides and couriers a copy to you shortly after you register for the course. The course covers:

- vulnerability/patch cycle
- security architecture
- principles of software security
- security design
- implementation (coding and deployment)
- input issues
- code review and security testing.

Unfortunately, the course does not start out strong: the walkthrough of the vulnerability/patch cycle is supposed to help build a case for secure software development, but it takes too long to make too fine a point, and it’s hard to stay interested.

The next sections on architecture, principles of software security, and design are confused by a weakness in organization. It’s not clear where architecture ends and design begins, and there is unnecessary repetition between the sections. It would flow better if the discussion started with foundational principles of software security (which are well covered in the course), then proceeding through architecture and then design. Architecture should cover security requirements, risk analysis and threat modeling to determine “how much security is enough”, defense-in-depth, attack surface analysis and minimizing attack surface, complexity and economy of mechanism, cost considerations, and layering and defining trust zones. Some of these issues are covered in the discussion on architecture, some in design, some in both, and some not at all.

The risk assessment discussion surveys different risk modeling techniques, including Microsoft’s STRIDE and DREAD, OCTAVE and others. It includes a recommendation for SALSA, an initiative which SANS seems to have started together with another vendor back in 2007 and which has gone nowhere since. SALSA doesn’t belong with the other established models.

The section on design covers security requirements (again) and some general motherhood on good design practices, then briefly touches on authentication, authorization and permissioning, non-repudiation and auditing, and data confidentiality, but doesn’t explore any of these important areas further in the context of design. This is one area where the Stanford course is much stronger.

The discussion of implementation is good, starting with an explanation of Gary McGraw’s 7 Kingdoms and then a long list of implementation-specific issues, which may or may not apply to your system, but anyways offers some concrete guidance which may catch the attention of programmers.

Then an excellent, brief exploration of input handling problems, recognizing the importance of not trusting data. And a (too) brief discussion of cryptography in the context of storing confidential data.

Finally, one of the strongest parts of the course, on security code review practices and security testing. Starting with an explanation of why and how to do code reviews (including formal inspections and tool-assisted code reviews), and references to language-specific checklists. Then a discussion of static analysis and a survey of different tools: a bit out of date, missing Microsoft’s SDL tools for example, but a decent overview of available static analysis tools. Then a good survey of testing techniques from a security perspective, including fuzzing and pen testing and risk-based testing, and finally a discussion of attack surface analysis (which more properly belongs in the architecture section).

You have 4 months to complete the 3 hour course, which is more than adequate. Because this was a beta of a new course and a new online delivery model, there was no assessment included.

Comparison of the courses

It was interesting to see that, except for a common focus on basic secure software principles, the courses took different, almost complementary approaches to introducing secure software development.

The SANS course was much more broad and current, with more references to resources like OWASP and frameworks and tools, and more on secure SDLC practices. There were some notable gaps in the SANS course, in architecture and especially design, and it suffers from some structural weaknesses, but it covers many of the bases, and important concepts such as secure error handling, failing safe, input handling, and risk management and threat modeling techniques. With a few improvements in structure and content around architecture and design, this would be an excellent course. As it stands, if I can get the developers past the introduction on the vulnerability cycle, they should learn some useful things.

The Stanford course is more than twice as long and more focused, but doesn’t touch on secure SDLC practices such as reviews and security testing, or risk assessment and threat modeling. It digs deeper into specific security implementation problems such as XSRF, XSS, password management, and SQL injection; and it is also stronger in its coverage of basic secure design, and especially cryptography. This course would be of special interest to developers who want to go on with the full Advanced Security Program at Stanford.

Neither of these courses is perfect, but they are both professionally delivered, and offer good value for the money.

Continuously Putting Your Customers at Risk

Over the past year or so there has been some buzz around Continuous Deployment: immediately and constantly pushing out new features directly to customers. This practice is followed by companies such as Facebook, Flickr and IMVU, where apparently programmers push changes out to production up to 50 times per day.

Continuous integration and rapid deployment have a number of advantages, and continuous deployment appears to maximize value from these ideas: immediate and continuous feedback from customers, and cutting deployment and release costs way down, eliminating overheads and manual checks.

A profile on Extreme Agility at Facebook describes how the company’s small development team deploys rapid updates to production:

“…Facebook developers are encouraged to push code often and quickly. Pushes are never delayed and applied directly to parts of the infrastructure. The idea is to quickly find issues and their impacts on the rest of the system and surely fixing any bugs that would result from these frequent small changes.”

But there are some fundamental and serious challenges with continuous deployment. Success depends on a few key factors:

1. a comprehensive and fast automated test suite to catch mistakes, especially regressions;
2. customers that are willing to let you test in production;
3. an architecture that catches and isolates failures, preventing problems from chaining or cascading across the cluster;
4. a disciplined, proven, robust deployment model.

Automated Testing

At IMVU, all changes are run through an automated test suite, which executes on a cluster of test servers, before deploying to production.

"So what magic happens in our test suite that allows us to skip having a manual Quality Assurance step in our deploy process? The magic is in the scope, scale and thoroughness.”

The author goes on to say that

“We have around 15k test cases, and they’re run around 70 times a day. That’s a million test cases a day.”

Ummm, actually, no it isn't. That’s 15k test cases a day. Run 70 times [and no explanation why the tests are 70 times per day, since we’re talking about pushing out 50 changes per day, but anyways…]. You could run 15k test cases a million times and it would still be 15k test cases, in the same way that running 1 test case a million times is still only 1 test case.

A regression suite of 15,000 automated unit and functional tests sounds impressive – of course what is much more important than the number of tests is the quality of tests. We’re a small shop, and we run more than 15,000 automated tests as part of our continuous integration environment, and we also run static analysis checks on all code, and we do peer code reviews, and manual testing including operations tests, exploratory and destructive tests, system trials, stress and performance testing, and end-to-end integration tests before we push to production.

From subsequent statements, it seems clear that most of the changes deployed this way at IMVU at least are trivial, bug fixes and minor modifications: schema changes, for example, are made out of band (they take 2 days to rollout to production at IMVU). So we can assume that the scope, and therefore, risk of any one change can be contained. This is backed up by a comment made by the author at 50 Deployments A Day and the Perpetual Beta:

“when working on new features (not fixing bugs, not refactoring, not making performance enhancements, not solving scalability bottlenecks, etc), we’ll have a controlled deliberate roll out plan that involves manual QE checks along the way, as well as a gradual roll0out and A/B testing.”

I’m not sure why you wouldn’t have a controlled roll-out plan for solving scalability bottlenecks, but let’s assume that he was referring to minor tweaks to the code or configuration, say increasing the size of a resource pool or something.

Testing in Production

After hearing about the approach followed by IMVU last year, a couple of exploratory testing experts, Michael Bolton and James Bach, spent a few minutes trying out IMVU’s system. They, not surprisingly, found a lot of problems without making much of an effort:

“Yes folks, you can deploy 50 times a day. If you don’t care bout the quality of what you’re deploying…”

The writer from IMVU admits that they have a lot of bugs:

“continuous deployment lets you write software *regression free*, it sure doesn’t gift you high quality software.”

Ignore the “*regression free*” claim which assumes that the test suite will always catch *all* regressions. Continuous Deployment essentially concedes the job of testing to your customers: you do some superficial reviews and regression tests, and leave the real work of finding problems to your customers. I can appreciate that this might be acceptable to some customers, for example people participating in online communities or online games, as they trade off the inconvenience of occasional glitches against the chance to try out cool new features quickly.

There’s nothing wrong with running experiments, trying out new ideas with part of your customer base through A/B split testing, seeing what sticks and what doesn’t. But this doesn’t mean you need to, or should, deploy every change directly to production. Again, if your customers are trusting you with financial transactions or sensitive personal information, you would be irresponsible if you took this approach, even if, like Facebook, you only push out changes incrementally to a small number of clients at a time.

Failure Isolation and Fail Safe

If you are going to continually roll changes, and anticipate that some of these changes will fail, the architecture of the system needs to isolate and contain failures, using internal firewalling, fast-fail techniques, timeouts and retries and so on to reduce the likelihood of a failure chaining through layers or cascading across servers and taking the cluster down.

Unfortunately, this doesn’t seem to be the case, at least in the example described here, where Alex, a programmer, is preparing to deploy code containing a 1-character typo which can cause a failure cascade and take out the site:

“Alex commits. Minutes later warnings go off that the cluster is no longer healthy. The failure is easily correlated to Alex’s change and her change is reverted. Alex spends minimal time debugging, finding the now obvious typo with ease. Her changes still caused a failure cascade, but the downtime was minimal."

So the development team is not only conceding that they cannot write good code, and that they are incapable of doing a decent job testing their work, but also that they cannot take the steps to fail safe at an architectural level and minimize the scope of any failures that they cause. This is not the same as conceding, as Google does, that at massive scale, failures will inevitably happen, and you will have to learn how to deal with them. This is simply giving up, and pushing risk out to customers, again.

A Deployment Model that Works

The deployment model at IMVU is cool: of course, if you are going to do something 50 times per day, you should be pretty good at it.

“The code is rsync’d out to the hundreds of machines in our cluster. Load average, cpu usage, php errors and dies and more are sampled by the push script, as a basis line. A symlink is switched on a small subset of the machines throwing the code live to its first few customers. A minute later the push script again samples data across the cluster and if there has been a statistically significant regression then the revision is automatically rolled back. If not, then it gets pushed to 100% of the cluster and monitored in the same way for another five minutes. The code is now live and fully pushed.”

This rollback model assumes that problems will be found in the first minute, or few minutes of operation. It does not account for race conditions, deadlocks and other synchronization faults, or time-dependent problems or statistical bugs or downstream integration conflicts or intermittent problems which might not show up for hours or even days, by which time dozens or hundreds of other changes have been applied. Good luck finding out where the problem came from by then.

The rollback approach also assumes that all that needs to be done to fix the problem is to rollback the code. It does not account for what needs to be done to track down and repair broken transactions and corrupt data, which might be ok in an online gaming environment, but would be CTO-suicide in a real system.

And you wonder why some web sites have serious software security issues?

Let’s put aside the arguments above about reliability and quality and responsibility, and just look at the problem of security: building secure software in an environment where developers push each change immediately to production.

First, try to imagine someone detecting unauthorized changes with all of this going on. Was every one of those changes intended and authorized? Would you be able to detect an attack in the middle of all of this noise?

Then there’s the problem of building secure software, which is hard enough even if you follow good practice, especially if you are moving fast. There has been a lot of work over the past couple of years defining how developers can build secure software in an agile way. Microsoft’s SDL for Agile maps secure software design and development practices to agile development models like Scrum, and cuts secure software development controls and practices down to fit rapid incremental development.

But with Continuous Deployment, at least as described so far, there is no time or opportunity to do even a minimal set of security checks and reviews before software changes are pushed out by developers.

It’s bad enough to build insecure software out of ignorance. But by following continuous deployment, you are consciously choosing to push out software before it is ready, before you have done even the minimum to make sure it is safe. You are putting business agility and cost savings ahead of protecting the integrity or privacy of customer data.

Continuous deployment sounds cool. In a world where safety and reliability and privacy and security aren’t important, it would be fun to try. But like a lot of other developers, I live in the real world. And I need to build real software.

And now we need to be "Rugged"

A new initiative for secure software development, for Rugged Software Development was announced this week at a SANS conference. Rugged Software is

a value system for writing secure software

defined by some smart people in the application security industry.

Presumably the Rugged Software initiative is attempting to duplicate the success of the agile software movement, coming with its own Rugged Software Manifesto:

I am rugged… and more importantly, my code is rugged.

and so on.

The agile development movement was successful because it was driven by and for the people who actually build software: by programmers, for programmers. By smart, experienced programmers, people like Kent Beck and Ward Cunningham who built software for a living and were really good at it, and who were searching together for ways to solve the problems that programmers face in software development, problems that mattered to programmers. It came from inside the software development community, and set out to put programmers effectively back in charge of building software, to make better software, to make the making of software better.

And agile development, at least at the beginning, was cool, counter-culture: agile developers were sticking it to the man, doing what was right, subverting big upfront design and top-down planning and by-the-book project management and so on. It was certain to create a following…. and unfortunately, eventually to become an institutionalized Methodology subsidized by tool vendors and consultants, but that’s another story for another day.

According to one of the founders of the Rugged Software initiative

Getting the secure software development message to the masses won't be easy, and the plan is to get some initial support and momentum from the application security industry.

However well-intentioned and necessary, it looks like another set of ideas and values being imposed from outside on people who are busy building software. We already have other application security initiatives: Cigital's Build Security In and its maturity model for the enterprise, Microsoft’s SDL for the Microsoft community at least, OpenSAMM and other initiatives from OWASP, and half-baked ideas from the InfoSec community like SALSA.

And now we have Rugged Software Development.

To succeed, the initiative needs support and momentum not just from the application security community, but more importantly from the software development community – from the people who actually build software.

Fair enough, these smart and well-intentioned and hard working InfoSec guys are asking for input and participation from the development community. So after being challenged to “walk the walk" I signed up for the Rugged Software forums, blogs, lists and…. Well, there’s the announcement and some trade press coverage. And that Manifesto about ruggedness, and an empty blog and an empty forum. That’s it, that's all I have been able to find so far.

So, I guess I was walking too fast. I will wait and see if there is a real opportunity here, a chance for an initiative that speaks to, and for, the software development community, something that has a real chance to succeed.

Real Resources for Software Development Managers

With all of the blogs and books and training programs and even a few magazines still available on software development, agile methods and software project management, there is surprisingly little material that is of real value to a software development manager: information that is thoughtful, current and grounded in real experience.

A lot of what you can find is noise. Blog postings by enthusiastic kids who have finished a project or two using Scrum and are now self-made experts on team dynamics and Agile development. Someone pushing a Project 2.0 collaboration tool or yet another “Agile Software Development using…” or “Agile Project Management…” book, or blogs going over (and over) the same ground, or tiresome evangelism for “Big A” Agile training and consulting.

It’s difficult to find useful information in all of this for software development managers who want to dig deeper. I’ve put together a list here of the resources that I have found useful, and find myself going back to.

Construx Software Builders

Construx is a consultancy specializing in software development management, founded by Steve McConnell, a leading thinker on software engineering and best practices in software development, and author of some of the definitive books on software development: Rapid Development and Code Complete.I learned a lot about good software engineering and software development management from these books. While Code Complete, a guide to writing good, clean code, was significantly revised in 2004, Rapid Development is showing its age, and needs to be updated to take into account XP and Scrum and other new ideas in software development. But it is still the best overview available of SDLCs and the risks and success factors in software projects.

Construx offers a wide range of consulting services including organizational reviews and project reviews and software due diligence reviews for acquisitions. They also offer an excellent set of training programs on software project management and software engineering. Some of the courses that my team and I have attended include:

Code Complete Essentials on the basics of good software development

Master Class on Estimating Software Projects

Developer Testing Bootcamp

Professional Tester Bootcamp

and 10x Software Engineering an excellent course on improving software development results for experienced managers.

You can also get access to white papers and posters including their famous list of Classic Mistakes in software development; and CXOne, a lightweight framework for managing software projects, with templates, samples and checklists.

Once each year or so, Construx holds an Executive Software Summit for experienced software development managers, CTOs and other senior people interested in improving how software is built. This is head and shoulders above the other software management conferences that I have attended.

Agile Development Resources

Scrum seems to have won the Agile development methodology wars over XP and DSDM and Crystal and whatnot, fundamentally because it is much easier to understand and follow (and also, unfortunately, easier for teams to build sloppy software faster). So, until Lean Software Development or some other new idea establishes the next wave you should make sure to understand Scrum, even if you don’t swallow it whole.

Certified Scrum Master training is not expensive, it doesn’t take long, and you’ll leave with a decent understanding of the method and its values and driving principles. Make sure to get a good teacher: I went straight to the source, Ken Schwaber. His book Agile Project Management with Scrum summarizes what you’ll learn from the course and is a good resource for follow-up.

Of the Agile / Scrum community blogs, the most valuable that I have found is Mike Cohn’s Succeeding with Agile and I would recommend his book of the same name if you are serious about understanding and implementing Scrum.

Pragmatic Programmer’s Bookshelf

There are a lot of good books on software development in the Pragmatic Programmer’s Bookshelf including a set of special interest for software development managers:

Manage It! by Johanna Rothman, is a simple but excellent book on incremental and iterative development practices, scheduling and estimating and risk management for small and mid-size projects and teams. This is the best, most practical book I have found on applying lightweight, agile development methods, and while it leans towards Scrum it is not dependent on any single methodology. This book also introduces program management and project portfolio management, which is explored in more detail in Manage Your Project Portfolio: useful if you’re new to the problems of managing small programs using lightweight techniques.

If you are serious about program and portfolio management, and you have the time and money, I strongly recommend the professional program on Advanced Project Management at Stanford University which you can attend online or on campus. This is a world-class program on aligning strategy with execution, managing customers, understanding and exercising power and influence, and coordinating and planning integrated programs and project portfolios.

Johanna Rothman is also the co-author (together with Esther Derby) of Behind Closed Doors, an introductory but good book on coaching and mentoring developers and managing teams. It offers practical advice and good reminders on issues like managing priorities, the value of 1-on-1 meetings, how to deal with technical people, how to give feedback – a leadership resource, but written specifically for software development managers and team leads.

Ship It! by Jared Richardson, Will Gwaltney, Jr, outlines the basics of building and deploying software in an agile context: the use of tools for source code control and continuous integration, useful strategies for adopting automated unit testing, basic engineering practices, small team leadership techniques, fundamentals of incremental development, common problems faced.
Of the books in this series, this is the most basic / introductory, but it is worth a quick read for a framework for iterative, incremental development.

Release It! by Michael T. Nygard, is an excellent resource on technical architecture for distributed, web-based (especially Java) systems – a high-level view of the challenges that your team will face building real systems, patterns and anti-patterns for stability and scalability, and how to engineer a system for real world operations. It is the “hardest” of these books, and will be useful to your architects and Dev/Ops team. You can follow Michael’s current work at his blog, Wide Awake Developers.

Although not part of the Pragmatic Programmer’s suite, you also need to read The Visible Ops Handbook, a simple, practical introduction to IT systems change management and release management.

Software Project Management Resources

I wrote earlier about Scott Berkun’s Making Things Happen, a reissue of The Art of Project Management. This is a practical, focused and well-written book on basic issues in software project management, on leadership and communications and especially on execution, based on his experience managing programs at Microsoft.

The Mythical Man Month, by Fred Brooks – yes, it really is worth reading and re-reading if you haven’t read it in a while.

As a counterbalance to all of the agile and small team collaboration stuff, I enjoy following Herding Cats. This blog is all about large-scale, high-risk, high-complexity, safety-critical projects and large-scale programs like manned space flight and nuclear power and weapons systems. It offers a completely different and provocative perspective on project management problems, and is a good resource on complexity and risk management.

Leadership Resources for Software Development Managers

Cornell University offers an excellent online program on High Performance Leadership covering change management, leadership, negotiation and coaching.

Another excellent resource on leadership is the Center for Creative Leadership, which offers inexpensive live or pre-recorded webinars.

The American Management Association and its Canadian equivalent, the Canadian Management Center, provide excellent training programs on management and leadership.

The only leadership blog that I follow consistently is Great Leadership. It’s a bit cheerleadie at times, but it it is thoughtful and has good links to other leadership resources.

The general leadership books that I have found useful and worth going back to are Difficult Conversations and Getting to Yes both developed out of the Program on Negotiation at Harvard Law School. Its Program on Negotiation for Senior Executives is an excellent course, and definitely worth taking for development managers (don’t be put off by the “for Senior Executives” label).

The Best of the Software Development Manager Blogs

Of the many other software development management blogs and forums, there are only a couple that I follow regularly:

Joel on Software of course: it's well-written and provocative, and focuses on small-scale software development and on how to run a software business. Unfortunately, for the last few months the focus has been on one of Joel’s latest ventures, Stackoverflow: a useful and free resource for problem solving for developers. While this has been good for Stackoverflow, it has not made for especially interesting reading. Here’s hoping that Joel gets back to subjects of wider interest soon.

Hard Code by Erich Brechner, the head of the best practices group at Microsoft. While this is written for Microsoft’s internal developers, most of the issues and problems that he explores apply generally to the software development community, and it is entertaining and real.

I’ll add to this resource list as I find other useful information.

Software Project Manager's Bridge to Agility

The Software Project Manager’s Bridge to Agility by Michele Sliger and Stacia Broderick sets out to map the PMBOK project management practice framework to an Agile practice framework (essentially Scrum) for managing software development projects.

The book introduces Agile project management concepts, then explains how the PMBOK practices for Integration Management, Scope Management, Time Management, Cost Management, Quality Management, Human Resources Management, Communications Management, Risk Management and Procurement Management are realized in an Agile project management approach. This analysis is necessarily repetitive as Agile project management is essentially simple, solving a broad set of different problems using the same set of basic principles and techniques: a flat, collaborative team structure where the project manager primarily acts as a coach and facilitator; controlling scope and quality and managing risk through time-boxing and frequent reviews; maintaining a close working relationship with the customer through its representative product owner; and lightweight (just enough and no more) documentation.

The authors are certified PMPs and Agile consultants so they have a foot in both worlds, although it is clear where their bias (and their source of incomes) lays. The book that they have written is not a bad book – it offers an interesting analysis of two essentially different approaches to managing software projects, and reinforces the value of fundamental Agile development principles. But it falls short in a number of key areas:

Like too much writing on Agile management, the book offers a simplistic and naïve view of the scope of a project manager's responsibilities, and minimizes especially the need for preparation and planning and design. I suppose my concern here is not so much with the book, as with the weakness of the approach advocated: but to act as a bridge between the PMBOK and Agile management, the book should address the needs and concerns of project managers, especially for projects that are not trivial. In the Agile framework described in the book, there is an unfortunate lack of upfront planning and preparation: it essentially consists of some vague work on the part of the project manager to secure sponsorship for the project, and an upfront vision meeting where the team (who, naturally at this point, likely knows nothing at all about the domain or the problems to be solved) meets with the customer and together lays out the high-level boundaries of the project and the priorities and features (the Backlog), and agrees on how to describe the project’s vision in a few sentences (an Elevator Statement).

This raises a number of concerns of course, but one of the most fundamental is a bootstrapping problem: if you haven’t prepared, done some research and upfront analysis, if you don’t know anything about what you are being asked to try to build, then how do you decide who should be on the team? The fundamental assumption here is that the team is an initial constraint: you get a small group of cross-trained generalized specialists who can handle any kind of problem with minimal upfront work, and make do with that team. For simple business applications, I am sure this could work. For complex science or engineering problems that demand specific domain or technical (or both) experience to even understand what is going on and what the customer is talking about, never mind delivering what the customer needs…. umm, no.

Like others in the Agile community, the authors allow for an “Iteration Zero”: some brief upfront work to setup the environment, for ramp-up and discovery, approvals, design discussions, maybe some training for the team in the domain and tools. But the authors encourage the team to deliver working software in this first (zeroeth actually) iteration, again short changing time for upfront analysis and discovery.

I am a strong advocate of incremental, iterative design and development, and I fully appreciate the concerns of wasting time and resources in what Steve McConnell calls the “fuzzy front end” of a project. But rushing through the initial analysis and discovery and technical planning and integration planning steps before you get a chance to understand something about the domain and the technical requirements and the preparedness of the team would be suicide if you were managing a project that was part of a large program, or an embedded software development project, or building a safety-critical system, or really anything more than a trivial business system or maybe a web portal: and even here you are dependent on the team’s experience and preparedness, their technical competence to fill in the necessary gaps and catch up.

The Agile approach described in the book for project estimating and scheduling is fundamentally weak as well: there is no attempt (at least that I could find) to create a high-level schedule for the project. At the start of the project, in the project vision meeting

the scope is defined a very high level. It is not uncommon to leave the vision meeting with only a dozen or so features identified…

In the section on Time Management, the book explains how estimates are made “at the strategic level” – the level of release planning – through abstract story point estimates, although the method by which the team translates story points into useful measures of time and cost is not made clear (yes I know how it works, but you won’t understand it from this book). A further weakness here is the approach taken to create the estimates: the team picks a feature, estimates it in an abstract way based on their understanding of its complexity, then goes on to estimate the complexity of other features relative to the first one. Of course, the first feature is the least likely to be estimated correctly, as the team has the least amount of information available when preparing that estimate. All of the other estimates are calibrated to this first estimate, amplifying its inaccuracy.

The only time that the team does any substantial estimating is when they are working through the details at the start of each iteration inside a release. Although there is a reference to the team using a “prioritized, estimated product backlog” in release planning, it is not clear when these estimates are created, or how the project schedule itself is defined – i.e., how the team arrives at a date by which the project is scheduled to be completed.

Now I understand and agree that estimates done in initial planning of a project are notional only, that there is a wide variance in accuracy of these estimates, and that the accuracy of estimates will improve over time. This is commonly understood in software development, inside and outside of the Agile community. But there needs to be some attempt at estimating the scope of delivery as understood at the beginning so that the project can be chartered, so that money and people and supporting resources can be committed; and, if this is more than a trivial software development project, so that requirements of other teams working on related projects or programs can be understood, integration points and dependencies defined, and commitments made. This is a fault of the book and not of the method: there is no clear tie-in from project initiation to the planning of each release in the book, although there must be one.

The book also assumes a narrow scope for the project manager’s, and the team’s, responsibilities and concerns: that they are focused only on developing the software, and don’t need to be concerned with integration and coordination with other projects or programs. The authors do allow for an “Iteration H”, a bracketing iteration at the end of the project for “hardening” the system: preparing documentation, packaging, training operations and so on – oh and of course leave time for celebration. Ah, if project management was only that simple. While the book makes a brief reference to the need for the PM to take care of external dependencies, it does not explain how these are managed within the Agile context – as far as I can see, the answer is: they are not.

I was hoping that the book would offer some ideas on how to manage Agile projects with strict contracting models and fixed constraints. Unfortunately again, this concern is disregarded:

“Agile projects have the ability to always be on time, and within budget, if the scope is flexible”.

Umm, stating the obvious, yes this is true. But if you are not delivering what the customer needs, or asked for in the contract, when they need it, what is the point? The book doesn’t offer guidance to the PM on how to reconcile iterative delivery and elastic scope with the cost and time and other business constraints of the contract, and other constraints in a larger program: in any real project, the work of other teams is dependent on your team delivering an expected scope of work by an expected date. Instead, the book assumes that the team working with the customer’s product owner will be able to resolve changes in scope collaboratively, and presumably that there will be no need to worry about contractual terms and commitments and other business issues. In this world nobody appears to be accountable for managing the tie-in back to the contract: as long as working software is delivered, everyone will be happy and everyone will be paid.

It is disappointing, but not a surprise, that the authors don’t offer ideas on how to effectively manage fixed-price, fixed-schedule projects using Agile techniques. Ken Schwaber, one of the authors of the Scrum method, admits in Agile Project Management with Scrum that he does not know of a way to deal with this problem either, and suggests trying a different contracting model based on essentially conceding upfront that you won’t be able to meet the customer’s requirements, and selling them instead on an iterative, collaborative business model. If you are able to stay in business trying this, good on ya mate.

The book does address Risk Management, primarily relying on the intrinsic risk management practices in iterative, incremental development. I agree that iterative, incremental development in short time-boxes, coupled with solid engineering practices, significantly reduces many kinds of project risks – this is why we follow this approach. The book does go further to describe a basic model for those teams who decide that they need to do explicit risk management – reviewing risks and maintaining a Risk Board for each iteration. This is a good approach, but does not take into account larger or longer-term risks: the team is focused on dealing with risks and technical issues immediately in front of them.

My biggest concern with this book is that it takes such a fundamentally simplistic and unnecessarily critical view of “traditional plan-driven projects”. In these projects quality is short changed; project managers only go through the motions on risk management; and of course the planning process does not allow for iteration – all projects except for Agile projects follow a pure waterfall model and attempt to plan everything upfront in detail and fail manage to this plan. Where does this come from? Almost all large projects that I am aware of are managed through spiral or phased development SDLCs, and include prototyping and other iterative techniques. Rolling wave planning has been a recognized best practice in project management for a long time. It is absurd to take the position that PMs managing large, complex projects and programs are not aware of, and not following, these practices and that the only alternative is to follow Scrum.

It’s not clear to me who the audience for this book is. Project Managers (specifically PMPs) who don’t understand Agile methods? But how is this possible with all of the noise about Agile methods, and taking into account that Agile methods are essentially and purposefully simple – you can learn the Scrum approach in an afternoon, and a Certified Scrum Master course involves only a couple days of training.

The naïve and critical view that the authors take of planning and project management will not convince PMPs who manage large or complex projects that they will be more successful following Agile techniques. Or is the audience inexperienced managers who are trying to chose between the PMBOK and Agile methods, not understanding that they can (and should, and in fact if they are managing anything of any complexity, must) do both?

Microsoft's SDL: Software Security for the Rest of Us

Microsoft’s Secure Development Lifecycle (SDL) is the most comprehensive, practical resource for secure software development that I have found. No other organization comes close to the level of transparency that Microsoft offers into its secure SDLC, even to the point of publicly analyzing problems which made it through the SDL security reviews and gates, and what the team learned from these failures.

Of course, you can argue that Microsoft has a lot of making up to do for the security mess that they helped to create by ignoring quality and security issues for so long, but the SDL, with its detailed process framework and tools and other supporting resources, all freely accessible, is a valuable contribution to the general software development community.

The SDL includes:

1. the SDL process framework, a prescriptive, step-by-step guideline for applying software security practices throughout development and implementation;
2. a Starter Kit which provides guidance for adopting the SDL and training resources;
3. business case resources if you or your boss needs a justification for investment in secure software development;
4. a repository of free tools, including fuzzers, static analysis tools for .NET code and other tools that are Microsoft-specific. However, this repository also includes a general-purpose Threat Modeling tool which helps you walk through and document threat models;
5. a set of white papers and presentations supporting the SDL;
6. and a blog on software security and SDL related topics.
   

Earlier versions of the Microsoft SDL as described in the 2006 book (which, while sometimes preachy and shrill in defense of Microsoft, is still worth reading) are more approachable, especially to the wider software development community. As the SDL continues to develop, becomes more institutionalized within Microsoft, it is becoming more inward-looking, bound up with specific practices and technologies – it’s become, well, more Microsofty. This is to be expected, and reflects Microsoft’s priorities for the SDL:

1. first to address Microsoft’s own weaknesses in software security and reliability, after years of focusing on shipping features;
2. to help make Microsoft’s .Windows a more secure platform, and .NET a more secure development environment, to further the sale of Microsoft technology;
3. and finally to share Microsoft’s insights, experience and frameworks on building secure software with the software community, and polish up the Microsoft brand by establishing the "new Microsoft" as a thought leader in the field.

So what lessons, what value can the wider software community take from the SDL, especially teams that do not develop on or for MS Windows, don’t use .NET or Microsoft development tools? Here are the main points that I have taken from the SDL, the ideas and practices that stood out for me:

Information Privacy

First, the SDL makes a special point of separating security and privacy issues, and lays out design guidelines and considerations, best practices and review gates for ensuring both security and privacy in software design and construction, implementation and support. This is an important and useful perspective for software teams working in regulated industries such as finance or health with special obligations for ensuring information privacy.

Training

The SDL emphasizes the value of training for the entire team – not only awareness training, but a training ladder which addresses the needs of different specialists (architects, developers, testers, project managers). It also emphasizes the need for constant upgrading of training – every member of the team must complete at least 1 training course every year – to ensure that the team is updated with the changing threat landscape and advances in the state of the art, as well as to reinforce basic practices.

Issue Tracking

Security issues need to be tracked in the bug tracking system, and made visible to the team. For each security bug, information about the cause and impact/effect should be recorded for metrics and root cause analysis purposes.

Design

Secure design is centered on threat modeling which the authors describe as the corner stone of their secure development process, using the STRIDE approach. There is a growing emphasis in more recent versions of the SDL on the importance of creating and maintaining artifacts from threat modeling exercises to be used in other steps of the SDL (such as attack surface reviews, testing, fuzzing and other review gates), and using software tools to assist in threat modeling (not surprising, coming from a company that builds tools for other developers).

The other important consideration in secure design is reviewing and reducing the attack surface of the application – fundamentally a good idea of course, while not necessarily easy to do (the reducing part).

Construction

Security practices and controls here emphasize the use of secure coding guidelines and checklists, and the use of static binary analysis tools – some of which are provided by Microsoft in the free SDL tool repository.

Verification

I was interested in the emphasis on fuzzing by Microsoft. It makes sense, given that they started with a large legacy code base, with well known reliability problems: fuzzing protocols and files and other interfaces offers a high return in finding basic problems in validation and other fundamental mistakes which contribute to a lot of reliability and security problems. This encouraged me to introduce fuzzing into our testing regimen last year, something that we are continuing to work on.

The other key practices in verification are vulnerability regression testing (if a vulnerability is found, create tests to ensure that the problem is not reintroduced); and code reviews, following a risk-based approach to focus on high-risk areas of code:

- internet-facing and network-facing code
- code that runs with elevated privileges
- code that handles secrets
- code that has a history of vulnerabilities
- older code

Release Management

Security incident response planning: face that you may encounter problems in production, prepare to handle them.

And final reviews for security and privacy: review the threat model (again); review the team's compliance with SDL practices; and review security issues that were deferred or not addressed by the team.

Scaling the SDL for agile development

Microsoft recognizes that the SDL is heavyweight (it was created to support long, spiral development cycles for products like Microsoft Windows and MS Office) and that teams following agile, rapid delivery methods will not have enough time to apply the full SDL. In its most recent release, Microsoft provides guidelines for smaller, agile teams to scale the SDL, breaking out its activities as follows:

One-time – generally done at the start of the project (iteration 0)
- choose a security advisor
- baseline threat models
- setup bug tracking

Every iteration – must be done
- static analysis
- threat model for all new features
- strong crypto
- review of error messages

Buckets: 1 of each activity type needs to be done in each iteration
- verification: fuzzing, attack surface analysis, or code review of high-risk code
- design: crypto review, privacy review, threat model training, or attack surface reduction
- response planning: contacts, planning and updating for incident response

At the end of every sprint, there is a lightweight review gate to ensure that the team is following the SDL guidelines.

The SDL for agile teams is a compromise: but it may be a compromise that not all small teams can handle, at least to start. An extremely lightweight approach is offered by Michael Howard, a principal security program manager at Microsoft and one of the author’s of the SDL, in his Everything Developer Security presentation, where he hilights the critical importance of 1 core practice each for designers, developers and testers, which together address a significant number of security problems:

1. Designers: threat modeling.
   
2. Coders: never trust data (“all input is evil until proven otherwise”) - half of all security bugs are input validation problems.
3. Testers: fuzzing – which is described as “brutally effective” for both security and reliability issues.

This list can be thought of as the lowest possible bar, a minimum set of practices that teams need to build on.

I like that the ideas, practices and tools in the SDL can be scaled up and down; that it can be applied at different levels of organizational maturity; and that it provides so many useful resources, even to teams who are not building software in the Microsoft world.