Saturday, January 23, 2010

Software Project Manager's Bridge to Agility

The Software Project Manager’s Bridge to Agility by Michele Sliger and Stacia Broderick sets out to map the PMBOK project management practice framework to an Agile practice framework (essentially Scrum) for managing software development projects.

The book introduces Agile project management concepts, then explains how the PMBOK practices for Integration Management, Scope Management, Time Management, Cost Management, Quality Management, Human Resources Management, Communications Management, Risk Management and Procurement Management are realized in an Agile project management approach. This analysis is necessarily repetitive as Agile project management is essentially simple, solving a broad set of different problems using the same set of basic principles and techniques: a flat, collaborative team structure where the project manager primarily acts as a coach and facilitator; controlling scope and quality and managing risk through time-boxing and frequent reviews; maintaining a close working relationship with the customer through its representative product owner; and lightweight (just enough and no more) documentation.

The authors are certified PMPs and Agile consultants so they have a foot in both worlds, although it is clear where their bias (and their source of incomes) lays. The book that they have written is not a bad book – it offers an interesting analysis of two essentially different approaches to managing software projects, and reinforces the value of fundamental Agile development principles. But it falls short in a number of key areas:

Like too much writing on Agile management, the book offers a simplistic and naïve view of the scope of a project manager's responsibilities, and minimizes especially the need for preparation and planning and design. I suppose my concern here is not so much with the book, as with the weakness of the approach advocated: but to act as a bridge between the PMBOK and Agile management, the book should address the needs and concerns of project managers, especially for projects that are not trivial. In the Agile framework described in the book, there is an unfortunate lack of upfront planning and preparation: it essentially consists of some vague work on the part of the project manager to secure sponsorship for the project, and an upfront vision meeting where the team (who, naturally at this point, likely knows nothing at all about the domain or the problems to be solved) meets with the customer and together lays out the high-level boundaries of the project and the priorities and features (the Backlog), and agrees on how to describe the project’s vision in a few sentences (an Elevator Statement).

This raises a number of concerns of course, but one of the most fundamental is a bootstrapping problem: if you haven’t prepared, done some research and upfront analysis, if you don’t know anything about what you are being asked to try to build, then how do you decide who should be on the team? The fundamental assumption here is that the team is an initial constraint: you get a small group of cross-trained generalized specialists who can handle any kind of problem with minimal upfront work, and make do with that team. For simple business applications, I am sure this could work. For complex science or engineering problems that demand specific domain or technical (or both) experience to even understand what is going on and what the customer is talking about, never mind delivering what the customer needs…. umm, no.

Like others in the Agile community, the authors allow for an “Iteration Zero”: some brief upfront work to setup the environment, for ramp-up and discovery, approvals, design discussions, maybe some training for the team in the domain and tools. But the authors encourage the team to deliver working software in this first (zeroeth actually) iteration, again short changing time for upfront analysis and discovery.

I am a strong advocate of incremental, iterative design and development, and I fully appreciate the concerns of wasting time and resources in what Steve McConnell calls the “fuzzy front end” of a project. But rushing through the initial analysis and discovery and technical planning and integration planning steps before you get a chance to understand something about the domain and the technical requirements and the preparedness of the team would be suicide if you were managing a project that was part of a large program, or an embedded software development project, or building a safety-critical system, or really anything more than a trivial business system or maybe a web portal: and even here you are dependent on the team’s experience and preparedness, their technical competence to fill in the necessary gaps and catch up.

The Agile approach described in the book for project estimating and scheduling is fundamentally weak as well: there is no attempt (at least that I could find) to create a high-level schedule for the project. At the start of the project, in the project vision meeting
the scope is defined a very high level. It is not uncommon to leave the vision meeting with only a dozen or so features identified…
In the section on Time Management, the book explains how estimates are made “at the strategic level” – the level of release planning – through abstract story point estimates, although the method by which the team translates story points into useful measures of time and cost is not made clear (yes I know how it works, but you won’t understand it from this book). A further weakness here is the approach taken to create the estimates: the team picks a feature, estimates it in an abstract way based on their understanding of its complexity, then goes on to estimate the complexity of other features relative to the first one. Of course, the first feature is the least likely to be estimated correctly, as the team has the least amount of information available when preparing that estimate. All of the other estimates are calibrated to this first estimate, amplifying its inaccuracy.

The only time that the team does any substantial estimating is when they are working through the details at the start of each iteration inside a release. Although there is a reference to the team using a “prioritized, estimated product backlog” in release planning, it is not clear when these estimates are created, or how the project schedule itself is defined – i.e., how the team arrives at a date by which the project is scheduled to be completed.

Now I understand and agree that estimates done in initial planning of a project are notional only, that there is a wide variance in accuracy of these estimates, and that the accuracy of estimates will improve over time. This is commonly understood in software development, inside and outside of the Agile community. But there needs to be some attempt at estimating the scope of delivery as understood at the beginning so that the project can be chartered, so that money and people and supporting resources can be committed; and, if this is more than a trivial software development project, so that requirements of other teams working on related projects or programs can be understood, integration points and dependencies defined, and commitments made. This is a fault of the book and not of the method: there is no clear tie-in from project initiation to the planning of each release in the book, although there must be one.

The book also assumes a narrow scope for the project manager’s, and the team’s, responsibilities and concerns: that they are focused only on developing the software, and don’t need to be concerned with integration and coordination with other projects or programs. The authors do allow for an “Iteration H”, a bracketing iteration at the end of the project for “hardening” the system: preparing documentation, packaging, training operations and so on – oh and of course leave time for celebration. Ah, if project management was only that simple. While the book makes a brief reference to the need for the PM to take care of external dependencies, it does not explain how these are managed within the Agile context – as far as I can see, the answer is: they are not.

I was hoping that the book would offer some ideas on how to manage Agile projects with strict contracting models and fixed constraints. Unfortunately again, this concern is disregarded:
“Agile projects have the ability to always be on time, and within budget, if the scope is flexible”.
Umm, stating the obvious, yes this is true. But if you are not delivering what the customer needs, or asked for in the contract, when they need it, what is the point? The book doesn’t offer guidance to the PM on how to reconcile iterative delivery and elastic scope with the cost and time and other business constraints of the contract, and other constraints in a larger program: in any real project, the work of other teams is dependent on your team delivering an expected scope of work by an expected date. Instead, the book assumes that the team working with the customer’s product owner will be able to resolve changes in scope collaboratively, and presumably that there will be no need to worry about contractual terms and commitments and other business issues. In this world nobody appears to be accountable for managing the tie-in back to the contract: as long as working software is delivered, everyone will be happy and everyone will be paid.

It is disappointing, but not a surprise, that the authors don’t offer ideas on how to effectively manage fixed-price, fixed-schedule projects using Agile techniques. Ken Schwaber, one of the authors of the Scrum method, admits in Agile Project Management with Scrum that he does not know of a way to deal with this problem either, and suggests trying a different contracting model based on essentially conceding upfront that you won’t be able to meet the customer’s requirements, and selling them instead on an iterative, collaborative business model. If you are able to stay in business trying this, good on ya mate.

The book does address Risk Management, primarily relying on the intrinsic risk management practices in iterative, incremental development. I agree that iterative, incremental development in short time-boxes, coupled with solid engineering practices, significantly reduces many kinds of project risks – this is why we follow this approach. The book does go further to describe a basic model for those teams who decide that they need to do explicit risk management – reviewing risks and maintaining a Risk Board for each iteration. This is a good approach, but does not take into account larger or longer-term risks: the team is focused on dealing with risks and technical issues immediately in front of them.

My biggest concern with this book is that it takes such a fundamentally simplistic and unnecessarily critical view of “traditional plan-driven projects”. In these projects quality is short changed; project managers only go through the motions on risk management; and of course the planning process does not allow for iteration – all projects except for Agile projects follow a pure waterfall model and attempt to plan everything upfront in detail and fail manage to this plan. Where does this come from? Almost all large projects that I am aware of are managed through spiral or phased development SDLCs, and include prototyping and other iterative techniques. Rolling wave planning has been a recognized best practice in project management for a long time. It is absurd to take the position that PMs managing large, complex projects and programs are not aware of, and not following, these practices and that the only alternative is to follow Scrum.

It’s not clear to me who the audience for this book is. Project Managers (specifically PMPs) who don’t understand Agile methods? But how is this possible with all of the noise about Agile methods, and taking into account that Agile methods are essentially and purposefully simple – you can learn the Scrum approach in an afternoon, and a Certified Scrum Master course involves only a couple days of training.

The naïve and critical view that the authors take of planning and project management will not convince PMPs who manage large or complex projects that they will be more successful following Agile techniques. Or is the audience inexperienced managers who are trying to chose between the PMBOK and Agile methods, not understanding that they can (and should, and in fact if they are managing anything of any complexity, must) do both?

Tuesday, January 5, 2010

Microsoft's SDL: Software Security for the Rest of Us

Microsoft’s Secure Development Lifecycle (SDL) is the most comprehensive, practical resource for secure software development that I have found. No other organization comes close to the level of transparency that Microsoft offers into its secure SDLC, even to the point of publicly analyzing problems which made it through the SDL security reviews and gates, and what the team learned from these failures.

Of course, you can argue that Microsoft has a lot of making up to do for the security mess that they helped to create by ignoring quality and security issues for so long, but the SDL, with its detailed process framework and tools and other supporting resources, all freely accessible, is a valuable contribution to the general software development community.

The SDL includes:
  1. the SDL process framework, a prescriptive, step-by-step guideline for applying software security practices throughout development and implementation;
  2. a Starter Kit which provides guidance for adopting the SDL and training resources;
  3. business case resources if you or your boss needs a justification for investment in secure software development;
  4. a repository of free tools, including fuzzers, static analysis tools for .NET code and other tools that are Microsoft-specific. However, this repository also includes a general-purpose Threat Modeling tool which helps you walk through and document threat models;
  5. a set of white papers and presentations supporting the SDL;
  6. and a blog on software security and SDL related topics.
Earlier versions of the Microsoft SDL as described in the 2006 book (which, while sometimes preachy and shrill in defense of Microsoft, is still worth reading) are more approachable, especially to the wider software development community. As the SDL continues to develop, becomes more institutionalized within Microsoft, it is becoming more inward-looking, bound up with specific practices and technologies – it’s become, well, more Microsofty. This is to be expected, and reflects Microsoft’s priorities for the SDL:
  1. first to address Microsoft’s own weaknesses in software security and reliability, after years of focusing on shipping features;
  2. to help make Microsoft’s .Windows a more secure platform, and .NET a more secure development environment, to further the sale of Microsoft technology;
  3. and finally to share Microsoft’s insights, experience and frameworks on building secure software with the software community, and polish up the Microsoft brand by establishing the "new Microsoft" as a thought leader in the field.
So what lessons, what value can the wider software community take from the SDL, especially teams that do not develop on or for MS Windows, don’t use .NET or Microsoft development tools? Here are the main points that I have taken from the SDL, the ideas and practices that stood out for me:

Information Privacy

First, the SDL makes a special point of separating security and privacy issues, and lays out design guidelines and considerations, best practices and review gates for ensuring both security and privacy in software design and construction, implementation and support. This is an important and useful perspective for software teams working in regulated industries such as finance or health with special obligations for ensuring information privacy.

Training

The SDL emphasizes the value of training for the entire team – not only awareness training, but a training ladder which addresses the needs of different specialists (architects, developers, testers, project managers). It also emphasizes the need for constant upgrading of training – every member of the team must complete at least 1 training course every year – to ensure that the team is updated with the changing threat landscape and advances in the state of the art, as well as to reinforce basic practices.

Issue Tracking

Security issues need to be tracked in the bug tracking system, and made visible to the team. For each security bug, information about the cause and impact/effect should be recorded for metrics and root cause analysis purposes.

Design

Secure design is centered on threat modeling which the authors describe as the corner stone of their secure development process, using the STRIDE approach. There is a growing emphasis in more recent versions of the SDL on the importance of creating and maintaining artifacts from threat modeling exercises to be used in other steps of the SDL (such as attack surface reviews, testing, fuzzing and other review gates), and using software tools to assist in threat modeling (not surprising, coming from a company that builds tools for other developers).

The other important consideration in secure design is reviewing and reducing the attack surface of the application – fundamentally a good idea of course, while not necessarily easy to do (the reducing part).

Construction

Security practices and controls here emphasize the use of secure coding guidelines and checklists, and the use of static binary analysis tools – some of which are provided by Microsoft in the free SDL tool repository.

Verification

I was interested in the emphasis on fuzzing by Microsoft. It makes sense, given that they started with a large legacy code base, with well known reliability problems: fuzzing protocols and files and other interfaces offers a high return in finding basic problems in validation and other fundamental mistakes which contribute to a lot of reliability and security problems. This encouraged me to introduce fuzzing into our testing regimen last year, something that we are continuing to work on.

The other key practices in verification are vulnerability regression testing (if a vulnerability is found, create tests to ensure that the problem is not reintroduced); and code reviews, following a risk-based approach to focus on high-risk areas of code:

- internet-facing and network-facing code
- code that runs with elevated privileges
- code that handles secrets
- code that has a history of vulnerabilities
- older code

Release Management

Security incident response planning: face that you may encounter problems in production, prepare to handle them.

And final reviews for security and privacy: review the threat model (again); review the team's compliance with SDL practices; and review security issues that were deferred or not addressed by the team.

Scaling the SDL for agile development

Microsoft recognizes that the SDL is heavyweight (it was created to support long, spiral development cycles for products like Microsoft Windows and MS Office) and that teams following agile, rapid delivery methods will not have enough time to apply the full SDL. In its most recent release, Microsoft provides guidelines for smaller, agile teams to scale the SDL, breaking out its activities as follows:

One-time – generally done at the start of the project (iteration 0)
- choose a security advisor
- baseline threat models
- setup bug tracking

Every iteration – must be done
- static analysis
- threat model for all new features
- strong crypto
- review of error messages

Buckets: 1 of each activity type needs to be done in each iteration
- verification: fuzzing, attack surface analysis, or code review of high-risk code
- design: crypto review, privacy review, threat model training, or attack surface reduction
- response planning: contacts, planning and updating for incident response

At the end of every sprint, there is a lightweight review gate to ensure that the team is following the SDL guidelines.

The SDL for agile teams is a compromise: but it may be a compromise that not all small teams can handle, at least to start. An extremely lightweight approach is offered by Michael Howard, a principal security program manager at Microsoft and one of the author’s of the SDL, in his Everything Developer Security presentation, where he hilights the critical importance of 1 core practice each for designers, developers and testers, which together address a significant number of security problems:
  1. Designers: threat modeling.
  2. Coders: never trust data (“all input is evil until proven otherwise”) - half of all security bugs are input validation problems.
  3. Testers: fuzzing – which is described as “brutally effective” for both security and reliability issues.
This list can be thought of as the lowest possible bar, a minimum set of practices that teams need to build on.

I like that the ideas, practices and tools in the SDL can be scaled up and down; that it can be applied at different levels of organizational maturity; and that it provides so many useful resources, even to teams who are not building software in the Microsoft world.