Agile - What’s a Manager to Do?

As a manager, when I first started learning about Agile development, I was confused by the fuzzy way that Agile teams and projects are managed (or manage themselves), and frustrated and disappointed by the negative attitude towards managers and management in general.

Attempts to reconcile project management and Agile haven't answered these concerns. The PMI-ACP does a good job of making sure that you understand Agile principles and methods (mostly Scrum and XP with some Kanban and Lean), but is surprisingly vague about what an Agile project manager is or does. Even a book like the Software Project Manager’s Bridge to Agility, intended to help bridge PMI's project management practices and Agile, fails to come up with a meaningful job for managers or project managers in an Agile world.

In Scrum (which is what most people mean when they say Agile today), there is no place for project managers at all: responsibilities for management are spread across the Product Owner, the Scrum Master and the development team.

We have found that the role of the project manager is counterproductive in complex, creative work. The project manager’s thinking, as represented by the project plan, constrains the creativity and intelligence of everyone else on the project to that of the plan, rather than engaging everyone’s intelligence to best solve the problems.
In Scrum, we have removed the project manager. The Product Owner, or customer, provides just-in-time planning by telling the development team what is needed, as often as every month. The development team manages itself, turning as much of what the product owner wants into usable product as possible. The result is high productivity, creativity, and engaged customers.

We have replaced the project manager with the Scrum Master, who manages the process and helps the project and organization transition to agile practices.

Ken Schwaber, Agility and PMI, 2011

Project Managers have the choice of becoming a Scrum Master (if they can accept a servant leader role and learn to be an effective Agile coach – and if the team will accept them) or a Product Owner (if they have deep enough domain knowledge and other skills), or find another job somewhere else.

Project Manager as Product Owner

The Product Owner is command-and-control position responsible for the “what” part of a development project. It's a big job. The Product Owner owns the definition of what needs to be built, decides what gets done and in what order, approves changes to scope and makes scope / schedule / cost trade-offs, and decides when work is done. The Product Owner manages and represents the business stakeholders, and makes sure that business needs are met. The Product Owner replaces the project manager as the person most responsible for the success of the project (“the one throat to choke”).

But they don’t control the team’s work, the technical details of who does the work or how. That’s decided by the team.

Some project managers may have the domain knowledge and business experience, the analytical skills and the connections in the customer organization to meet the requirements of this role. But it’s also likely to be played by an absentee business manager or sponsor, backed up by a customer proxy, a business analyst or someone else on the team without real responsibility or authority in the organization, creating potentially serious project risks and management problems. Some organizations have tried to solve this by sharing the role across two people: a project manager and a business analyst, working together to handle all of the Product Owner’s responsibilities.

Project Manager as Scrum Master

It seems like the most natural path for a project manager is to become the team’s Scrum Master, although there is a lot of disagreement over whether a project manager can be effective – and accepted – as a Scrum Master, whether they will accept the changes in responsibilities and authority, and be willing to change how they work with the team and the rest of the organization.

The Scrum Master is a “process owner” and coach, not a project manager. They help the team – and the Product Owner – understand how to work in an Agile process framework, what their roles and responsibilities are, set up and guide the meetings and reviews, and coach team members through change and conflict.

The Scrum Master works a servant leader, a (nice) process cop, a secretary and a gofer. Somebody who supports the team and the Product Owner, “carries food and water” for them, tries to protect them from the world outside of the project and helps them solve problems. But the Scrum Master has no direct authority over the project or the team and does not make decisions for them, because Agile teams are supposed to be self-directing, self-organizing and self-managing.

Of course that’s not how things start off. Any group of people must work their way through Tuckman’s 4 stages of team development: Forming-Storming-Norming-Performing. It’s only when they reach the last stage that a group can effectively manage themselves. In the mean time, somebody (the Scrum Master / Coach) has to help the team make decisions that they aren’t ready to make on their own. It can take a long time for a team to reach this point, for people to learn to trust each other – and the organization – enough. And it may not last long, before something outside of the team’s control sets them back: a key person leaving or joining the team, a change in leadership, a shock to the project like a major change in direction or cuts to the budget. Then they need to be led back to a high performing state again.

Coaching the team and helping them out can be a full-time job in the beginning. After the team has got together and learned the process? Not so much. Which is why the Scrum Master is sometimes played part-time by a developer or sometimes even rotated between people on the development team.

But even when the team is performing at a high level, there’s more to managing an Agile project than setting up meetings, buying pizza and trying to stay out of the way. I've come to understand that Agile doesn't make a manager’s job go away. If anything, it expands it.

Managing Upfront

First, there’s all of the work that has to be done upfront at the start of a project – before Iteration Zero. Identifying stakeholders. Securing the charter. Negotiating the project budget and contract terms. Understanding and navigating the organization’s bureaucracy. Figuring out governance and compliance requirements and constraints, what the PMO needs. Working with HR, line managers and functional managers to put the team together, finding and hiring good people, getting space for them to work in and the tools that they need to work with. Lining up partners and suppliers and contractors. Contracting and licensing and other legal stuff. >/p>

The Product Owner might do some of this work - but they can't do it all.

Managing Up and Out

Then there’s the work that needs to be managed outside of the team.

Agile development is insular, insulated and inward-looking. The team is protected from the world outside so they can focus on building features together. But the world outside is too important to ignore. Every development project involves more than designing and building software – often much more than the work of development itself. Every project, even a small project, has dependencies and hand-offs that need to be coordinated with other teams in other places, with other projects, with specialists outside of the team, with customers and partners and suppliers. There is forward planning that needs to be done, setting and tracking drop-dead dates, defining and maintaining interfaces and integration points and landing zones.

Agile teams move and respond to change quickly. These changes can have impacts outside of the team, on the customer, other teams and other projects, other parts of the organization, suppliers and partners. You can try using a Scrum of Scrums to coordinate with other Agile teams up to a point, but somebody still has to keep track of dependencies and changes and delays and orchestrate the hand-offs.

Depending on the contracting model and your compliance or governance environment, formal change control may not go away either, at least not for material changes. Even if the Product Owner and the team are happy, somebody still has to take care of the paperwork to stay onside of regulatory traceability requirements and to stay within contract terms.

There are a lot of people who need to know what’s going on in a project outside of the development team – especially in big projects in big organizations. Communicating outwards, to people outside of the team and outside of the company. Communicating upwards to management and sponsors, keeping them informed and keeping them onside. Task boards and burn downs and big visible charts on the wall might work fine for the team, but upper management and the PMO and other stakeholders need a lot more, they need to understand development status in the overall context of the project or program or business change initiative.

And there’s cost management and procurement. Forecasting and tracking and managing costs, especially costs outside of development labor costs. Contracts and licensing need to be taken care of. Stuff needs to be bought. Bills need to be paid.

Managing Risks

Scrum done right (with XP engineering practices carefully sewed in) can be effective in containing many common software development risks: scope, schedule, requirements specification, technical risks. But there are other risks that still need to be managed, risks that come from outside of the team: program risks, political risks, partner risks and other logistical risks, integration risks, data quality risks, operational risks, security risks, financial risks, legal risks, strategic risks.

Scrum purposefully has many gaps, holes, and bare spots where you are required to use best practices – such as risk management.
Ken Schwaber

While the team and the Product Owner and Scrum Master are focused on prioritizing and delivering features and resolving technical issues, somebody has to look further out for risks, bring them up to the team, and manage the risks that aren't under the team’s control.

Managing the End Game

And just like at the start of a project, when the project nears the end game, somebody needs to take care of final approvals and contractual acceptance, coordinate integration with other systems and with customers and partners, data setup and cleansing and conversion, documentation and training. Setting up the operations infrastructure, the facilities and hardware and connectivity, the people and processes and tools needed to run the system. Setting up a support capability. Packaging and deployment, roll out planning and roll back planning, the hand-off to the customer or to ops, community building and marketing and whatever else is required for a successful launch. Never mind helping make whatever changes are required to business workflows and business processes that may be required with the new system.

Project Management doesn't go away in Agile

There are lots of management problems that need to be taken care of in any project. Agile spreads some management responsibilities around and down to the team, but doesn’t make management problems go away. Projects can’t scale, teams can’t succeed, unless somebody – a project manager or the PMO or someone else with the authority and skills required – takes care of them.

Classic Mistakes in Software Development and Maintenance

…the only difference between experienced and inexperienced developers is that the experienced ones realize when they’re making mistakes.
Jeff Atwood, Escaping from Gilligan’s Island

An important part of risk management, and responsible management at all, is making sure that you aren't doing anything obviously stupid. Steve McConnell’s list of Classic Mistakesis a place to start: a list of common basic mistakes in developing software and in managing development work, mistakes that are made so often, by so many people, that we all need to be aware of them.

McConnell originally created this list in 1996 for his book Rapid Development (still one of the best books on managing software development). The original list of 36 mistakes was updated in 2008 to a total of 42 common mistakes based on a survey of more than 500 developers and managers. The mistakes that have the highest impact, the mistakes that will most likely led to failure, are:

1. Unrealistic expectations
2. Weak personnel
3. Overly optimistic schedules
4. Wishful thinking
5. Shortchanged QA
6. Inadequate design
7. Lack of project sponsorship
8. Confusing estimates with targets
9. Excessive multi-tasking
10. Lack of user involvement

Most of the mistakes listed have not changed since 1996 (and were probably well known long before that). Either they’re fundamental, or as an industry we just aren't learning, or we don’t care. Or we can't find the time or secure a mandate to do things right, because of the relentless focus on short-term results:

Stakeholders won’t naturally take a long-term view: they tend to minimize the often extreme down-the-road headaches that result from the cutting of corners necessitated by the rush, rush, rush mentality. They’ll drive the car without ever changing the oil.
Peter Kretzman, Software development’s classic mistakes and the role of the CTO/CIO

The second most severe mistake that a development organization can make is to staff the team with weak personnel: hiring fast or cheap rather than holding out for people who have more experience and better skills, but who cost more. Although the impact of making this mistake is usually severe, it happens in only around half of projects – most companies aren't stupid enough to staff a development team with weak developers, at least not a big, high-profile project.

Classic Mistakes in Software Maintenance

But a lot of companies staff maintenance teams this way, with noobs and maybe a couple of burned out old-timers who are putting in their time and willing to deal with the demands of maintenance until they retire.

You get stuck in maintenance only if you are not good enough to work on new projects. After spending millions of dollars and many developer-years of effort on creating an application, the project is entrusted to the care of the lowest of the low. Crazy!
Pete McBreen, Software Craftsmanship

Capers Jones (Geriatric Issues of Ageing Software 2007, Estimating Software Costs 2008) has found that staffing a maintenance team with inexperienced people destroys productivity and is one of the worst practices that any organization can follow:

Worst Practice	Effect on Productivity
Not identifying and cleaning up error-prone code – the 20% of code that contains 80% of bugs	-50%
Code with embedded data and hard-coded variables – which contributes to “mass update” problems when this data changes	-45%
Staffing maintenance teams with inexperienced people	-40%
High complexity code that is hard to understand and change (often the same code that is error-prone)	-30%
Lack of good tools for source code navigation and test coverage	-28%
Inefficient or nonexistent change control methods	-27%

Many of these mistakes are due to not recognizing and not dealing with basic code quality and technical debt issues, figuring out what code is causing you the most trouble and cleaning it up.

The rest are basic, obvious management issues. Keep the people who built the system and who understand it and know how and why it works working on it as long as you can. Make it worth their while to stay, give them meaningful things to work on, and make sure that they have good tools to work with. Find ways for them to work together efficiently, with each other, with other developers, with operations and with the customer.

These simple, stupid mistakes add up over time to huge costs, when you consider that maintenance makes up between 40% and 80% of total software costs. Like the classic mistakes in new development, mistakes in maintenance are obvious and fixable. We know we shouldn't do these things, we know what’s going to happen, and yet we keep doing them, over and again, and we're surprised when things fail. Why?

Does the PMI-ACP set the bar high enough on Risk Management?

I’m trying to understand the PMI’s new certification for Agile Certified Practitioners, and what value the PMI brings to managing software development projects using Agile methods. So I bought RMC’s PMI-ACP Exam Prep Guide which is written by Mike Griffiths, a guy who understands a lot about project management and Agile methods, and who has been heavily involved in the PMI-ACP program.

How PMI-ACP looks at Risk

I started with how the PMI says risk management should be done in Agile projects. Unlike the PMBOK, the PMI-ACP does not treat risk management as a knowledge area. Instead, it integrates risk into the different practice domains and activities in Agile projects, from prioritization to delivery and problem management.

The first mention of risk management is in “Value-Driven Delivery”, treating risks as “anti-value” when considering what is important to the customer and the business. Fair enough.

Later in the same section there is a discussion of how risks need to be considered when managing the backlog – that you should schedule risk avoidance and risk mitigation activities early in the project, and explaining how to rank work by business value and risk. They suggest leveling the playing field by ranking all work (new features and changes and risks) by financial value, expressing everything in monetary terms. Risks have a negative financial return: risk impact in $ x risk probability in %. This only applies to risks that have avoidance / mitigation activities that can be scheduled and costed in the project – not for risks that are accepted or transferred.

I like the approach of managing risks the same as any other work, using the same costing and prioritization approach. It’s more consistent and more actionable than managing risks from separate lists.

Risk Management comes up one more time in Value-Driven Delivery, under a discussion of reporting tools and techniques, in this case how to create and use Risk Burn Down reports.

Then risk comes up again in Adaptive Planning – which makes sense. Risk assessment, like everything else in planning an Agile project, needs to be done incrementally and iteratively. But unfortunately there’s not a lot on how teams are supposed to identify risks in planning.

Later Griffiths suggests a collaborative game called Speedboat or Sailboat, to help the team come up with a list of risks and opportunities. This is Agile, so everything including risk management needs to be fun, and we don’t want to get people bummed out, so it’s important to spend time identifying opportunities too upfront. Team members post anchor (risk) and wind (opportunity) sticky notes around the picture of a boat on the water. Isn’t that nice…

Griffith does say that

“For any project, we should engage the development team, sponsors, customers, and other relevant stakeholders in the process of risk identification. Their ideas, along with reviews of previous projects’ lessons learned lists, risk logs and industry risk profiles, should be used to identify the known and likely risks for the project.”

But you can only use “lessons learned lists” and “risk logs” from previous projects if somebody on the previous project created them – and there are no actions in the PMI-ACP description of risk management to make sure that this gets done. As part of Continuous Improvement, Agile teams do conduct lessons learned reviews in each iteration, rather than waiting until the end of the project (a step that is often skipped because time and money have run out). The point is to act on lessons learned information immediately – not maybe on some other project in the future. This is good, but if people don’t save information for future use, then you can’t talk about using it in the future.

The last reference to risk management is under Problem Detection and Resolution – recommending running risk-based spikes early in the project to assess technical risks. Emphasizing that it is better to find out and fail early if you run into technical problems or limitations.

Is integrated and implicit risk management enough?

The PMI-ACP emphasizes integrated and active risk management as part of incremental planning and delivery.

“Risk management should serve as a driver for work scheduling, moving high-risk activities into earlier iterations of the project and incorporating risk mitigation activities into the backlog.”

Because risk-management activities are treated the same as other backlog items, work is always being done on reducing or containing risk based on negative value. But because risk management is built-in to different practice domains and into different tools and techniques, there’s no one place to understand how risk management should be done in an Agile project, and to assess whether it is being done well or not. You need to look at each practice area and how risk applies in each context. The way that it is organized makes it difficult to get your head around how risk management should be done in an Agile way – which is a source of risk in itself.

My criticisms aren’t of the study guide, which is well-written. They are of the PMI and the PMI-ACP framework. The PMI-ACP does put more emphasis on risk management than other descriptions of Agile development that I have seen so far. But it’s disappointing that the PMI did not take the opportunity to shore up a fundamental weakness in the Agile approach to development and recommend making risk management explicit, adding risk management activities to planning and reviews as a standard practice.

Some of these ideas are described in the Software Project Manager’s Bridge to Agility, a book that maps Agile development to PMI’s project management framework, and one of the books referenced in the PMI-ACP. But in the PMI-ACP as it is described, like most Agile development today, there’s too much reliance on the kind of risk management that comes for free in iterative and incremental development. This is probably enough for small teams working on simple application development projects, but that’s not the audience for PMI certification. Anyone using Agile methods on a larger scale or in high-risk development will need to look someplace else for help.

In Agile development planning, a security framework loses out

In Agile Estimating and Planning, Mike Cohn explains the different factors that go into prioritizing work on a software development project: financial value, cost, knowledge and risk. He then works through a couple of examples to show how these decisions are made. One of these examples is whether or not to build a security framework for an application – an example that I want to explore in more detail here.

Financial value / return

The first factor to consider when prioritizing work is the financial return that the customer or business will get from a feature: how much money will the customer/business earn or save from this feature over a period of time. For some features the business case is clear. In other cases, you have to do some creative accounting work to invent a business case, or decide based on other subjective business factors – often, how badly somebody important wants something.

Based on financial return, there is no clear business case for having a better/simpler security capability in the application by implementing a security framework. Security, and how it is implemented, is usually hidden from the business – it’s not something that they can see or use or make money from. You could try to argue that by improving the security of an application you could save on direct and indirect future costs of security breaches, and try to build a financial model around this. But unless the company or a close competitor has recently suffered from an expensive and embarrassing security problem, it is difficult to make a convincing case based on some future possibility.

Cost of development

How long will it take, how much will it cost to build it – and does this cost change over time? Is it cheaper to build it in early, or is it better to wait until later when you understand more about the problem and how to solve it properly?

Will it cost more to build security in later? Maybe. But probably not – because unless there’s a management or compliance requirement to make sure that everything is done correctly, there’s a chance that some of the security work won’t get done at all.

If you do decide to do the work early, there’s a greater chance that you will have to come back and change this code again later because of new requirements. So this adds to the future cost of development, rather than saving money and time.

Knowledge – how much do we learn by working on this

Another important factor to the development team is how much they might learn about the problem space or design, or how much they might learn about their ability to deliver the project, by working on something sooner rather than later. This is the point of technical spikes and prototyping – to learn and to reduce uncertainty.

As Mike Cohn points out, implementing a security framework doesn’t add much if anything to the team’s knowledge about the problem space or the design of the product. And it doesn’t answer any important questions for the team about their ability to deliver the project, about the technology platform or their tools and practices. It’s unlikely that whatever the team might learn from implementing a security framework will be material to the project’s success.

Risk

Of course there are risks to not implementing security correctly from the beginning. But the risks that we’re most concerned about when making prioritization decisions like this are schedule risks (how long is it going to take to deliver the key features, can we hit deadlines) and fundamental technical risks (is the architecture sound, will the system perform and stay up under load).

“Is there a risk to the project’s success that can be reduced or eliminated by implementing security earlier rather than later?”

The answer in this case is: probably not.

The project will still succeed – you’ll be able to deliver the project without a security framework. Bad things may happen later when the system is live, assuming that without a framework the team won’t do as good a job securing the app, but that won’t stop the project from getting delivered.

Security isn’t a feature

Using these decision factors (financial value, cost, knowledge and risk), there’s no compelling reason to build a security framework into the application. It doesn’t make the business any money, it doesn’t save time, it doesn’t tell us anything useful about the project or the problem domain, it doesn’t reduce project risk or technical risk in a significant way.

There’s nothing wrong with prioritizing features this way. What’s wrong is treating a security framework as a feature.

The decision on whether to implement a security framework shouldn’t be made based on any of these factors. Security isn’t a feature that can be traded off with other features, or cut because of cost. In the same way that data integrity isn’t a feature. Making sure that the system writes data correctly and consistently to the database or files, that data isn’t lost or partially updated or duplicated if something goes wrong, is a necessary part of writing a system. You don’t decide to do this now or never based on “customer value” or cost or time or whether you will learn something by doing this.

Whatever security is required by the type of system and the business needs to be architected in. It's not a product management decision. It’s part of the development team’s job to do this right. They can’t depend on other people, on the customer or a product manager to decide whether they should implement security sooner or later or not at all.

Rolling Forward and other Deployment Myths

There is more and more writing on Devops lately, which is good and bad. There still remains a small core of thoughtful people that are worth listening to and learning from. There’s more and more marketing from vendors and consultants jumping on the Devops bandwagon. There’s some naïve silliness (“Hire wicked smart people and give them all access to root.”) which can probably be safely ignored. And then there’s stuff that is half-right and half-wrong, too dangerous to be ignored. Like this recent post on Rollbacks and Other Deployment Myths, in which John Vincent lists 5 “myths” about system deployment, which I want to take some time to respond to here.

Change is change?

The author tries to make the point that “Change is neither good or bad. It’s just change.” Therefore we do not need to be afraid of making changes.

I don’t agree. This attitude to change ignores the fact that once a system is up and running and customers are relying on it to conduct their business, whatever change you are making to the system is almost never as important as making sure that the system keeps running properly. Unfortunately, changes often lead to problems. We know from Visible Ops that based on studies of hundreds of companies, 80% of operational failures are caused by mistakes made during changes. This is where heavyweight process control frameworks like ITIL and COBIT, and detective change control tools like Tripwire came from. To help companies get control over IT change, because people had to find some way to stop shit from breaking.

Yes, I get the point that in IT we tend to over-compensate, and I agree that calling in the Release Police and trying to put up a wall around all changes isn’t sustainable. People don’t have to work this way. But trivializing change, pretending that changes don't lead to problems, is dangerous.

Deploys are not risky?

You can be smart and careful and break changes down into small steps and try to automate code pushes and configuration changes, and plan ahead and stage and review and test all your changes, and after all of this you can still mess up the deploy. Even if you make frequent small changes and simplify the work and practice it a lot.

For systems like Facebook and online games and a small number of other cases, maybe deployment really is a non-issue. I don’t care if Facebook deploys in the middle of the day – I can usually tell when they are doing a “zero downtime” deploy (or maybe they are “transparently” recovering from a failure) because data disappears temporarily or shows up in the wrong order, functions aren’t accessible for a while, forms don’t resolve properly, and other weird shit happens, and then things come back later or they don’t. As a customer, do I care? No. It’s an inconvenience, and it’s occasionally unsettling ("WTF just happened?"), but I get used to it and so do millions of others. That’s because most of us don’t use Facebook or systems like this for anything important.

For business-critical systems handling thousands of transactions a second that are tied into hundreds of other company’s systems (the world that I work in) this doesn’t cut it. Maybe I spend too much time at this extreme, where even small problems with compatibility that only affect a small number of customers, or slight and temporary performance slow downs, are a big deal. But most people I work with and talk to in software development and maintenance and system operations agree that deployment is a big deal and needs to be done with care and attention, no matter how simple and small the changes are and no matter how clean and simple and automated the deployment process is.

Rollbacks are a myth?

Vincent wants us to “understand that it’s typically more risky to rollback than rolling forward. Always be rolling forward.”

Not even the Continuous Deployment advocates (who are often some of the most radical – and I think some of the most irresponsible – voices in the Devops community) agree with this – they still roll back if they find problems with changes.

“Rollbacks are a myth” is an echo of the “real men fail forward” crap I heard at Velocity last year and it is where I draw the line. It's one thing to state an extreme position for argument's sake or put up a straw man – but this is just plain wrong.

If you're going to deploy, you have to anticipate roll back and think about it when you make changes and you have to test rolling back to make sure that it works. All of this is hard. But without a working roll back you have no choice other than to fail forward (whatever this means, because nobody who talks about it actually explains how to do it), and this is putting your customers and their business at unnecessary risk. It’s not another valid way of thinking. It’s irresponsible.

James Hamilton wrote an excellent paper on Designing and Delivering Internet-Scale Services when he was at Microsoft (now he’s a an executive and Distinguished Engineer at Amazon). Hamilton’s paper remains one of the smartest things that anyone has written about how to deal with deployment and operational problems at scale. Everyone who designs, builds, maintains or operates an online system should read it. His position on roll back is simple and obvious and right:

Reverting to the previous version is a rip cord that should always be available on any deployment.

Everything fails. Embrace failure.

I agree that everything can and will fail some day, that we can’t pretend that we can prevent failures in any system. But I don’t agree with embracing failure, at least in business-critical enterprise systems, where recovering from a failure means lost business and requires unraveling chains of transactions between different upstream and downstream systems and different companies, messing up other companies' businesses as well as your own and dealing the follow-on compliance problems. Failures in these kinds of systems, and a lot of other systems, are ugly and serious, and they should be treated seriously.

We do whatever we can to make sure that failures are controlled and isolated, and we make sure that we can recover quickly if something goes wrong (which includes being able to roll back!). But we also do everything that we can to prevent failures. Embracing failure is fine for online consumer web site startups – let’s leave it to them.

SLAs

I wanted to respond to the points about SLAs, but it’s not clear to me what the author was trying to say. SLAs are not about servers. Umm, yes that’s right of course...

SLAs are important to set business expectations with your customers (the people who are using the system) and with your partners and suppliers. So that your partners and suppliers know what you need from them and what you are paying them for, and so that you know if you can depend on them when you have to. So that your customers know what they are paying you for. And SLAs (not just high-level uptime SLAs, but SLAs for Recovery Time and Recovery Point goals and incident response and escalation) are important so that your team understands the constraints that they need to work under, what trade-offs to make in design and implementation and in operations.

Under-compensating is worse than Over-compensating

I spent more time than I thought I would responding to this post, because some of what the author says is right – especially in the second part of his post, Deploy All the Things where he provides some good practical advice on how to reduce risk in deployment. He’s right that Operations main purpose isn’t to stop change – it can’t be. We have to be able to keep changing, and developers and operations have to work together to do this in safe and efficient ways. But trivializing the problems and risks of change and over-simplifying how to deal with these risks and how to deal with failures, isn’t the way to do this. There has to be a middle way between the ITIL and COBIT world of controls and paper and process, and cool Web startups failing forward, a way that can really work for the rest of us.

Risk Management in Software Security

Software security is a problem of managing risks. But most of the information on software risk management is about how to manage generic business, project and technical risks - not software security risks. Even the section on risk analysis in Gary McGraw’s book Software Security: Building Security In describes a generic risk management framework that can be used in a software security program. This is good, but most people involved in software development, and hopefully anyone who is responsible for software security, already understands the fundamentals of risk management. What people need to understand better is how to manage risks from a software security perspective, and how this is different from managing project risks or business risks.

This post is based on feedback that I shared with some of the team working on the next OWASP Development Guide, in my review of an early draft of the introductory section on risk management. I’m not sure if or how my feedback will be reflected in the new guide, since the project has recently undergone a change in leadership and there has been a lot of rethinking and resetting of expectations. So I have taken some time to put my thoughts together here and think through the issues some more.

Risk Management in Software Development

As a software development manager or project manager, our focus (and our training) is on how to identify and manage risks that could impact the delivery of a project. We learn to avoid project failure through project management structure and controls, by following good practices, and by actively and continuously managing risks inside and outside of the project team, including:

• Schedule and estimation risks
  
• Requirements and scope risks – the problem of managing change
  
• Cost and budget and financial/funding risks
  
• Staffing and personnel risks: team skills and availability, turnover, subcontractors, dependencies on partners
  
• Business strategy risks, portfolio risks, ROI/business case
  
• Stakeholder and sponsorship risks – the basic politics of business
  
• Program risks – interfaces and dependencies with other projects
  
• Legal and contracting risks
  
• Technical risks: architecture, platforms, tooling: how well do we understand them, are we on the bleeding edge (will it work).
  

Risk Management in Software Security

Discovering and managing risks from a security perspective is different: the perspective that you need to take is different, and the issues that you need to manage are different.

To find software security risks, you need to think beyond the risks of delivery and consider the operational and business context of the system. You need to look at the design of the system and its infrastructure, IT and business operations, your organization and its security posture, and your organization’s business assets.

Assets: What does the business have that is worth stealing or damaging?

Think like an attacker. Put yourself in the position of a motivated and skilled attacker: most of us can ignore the possibility of being attacked by script kiddies or amateurs who want to show off how smart they are. The bigger, more serious threat now comes from either disgruntled insiders or former employees, or from motivated professional criminals or even nation states.

Every business has something worth stealing or damaging. You need to understand what somebody would want, and how badly they might want it:

Confidential and personal data – are you holding private, personally identifiable or confidential data on customers, on your partners, your personnel, or anyone else?

Are you handling financial payments, or other high-value transactions?

Information about buying patterns or other business activities that would be valuable to competitors or other outside parties.

Information about your financials, investments and spending, or your business plans and strategy.

Intellectual Property: research data, design work, or information not about what you are doing, but how you are doing it – your operational systems, supply chain management. Or the designs and algorithms that drive the technology platform – it may not be the data behind the system that is valuable, the target could be the system itself, your technical knowledge.

Start with data and other assets, then look at the systems that you are building, that you support.

Is the system a critical online service? In rare cases, the system could be part of critical infrastructure (electrical power transmission, or a core financial system such as the NYSE, or maybe an emergency notification system). Or you may be running a completely online business – if the system is down, your business stops. Such systems may be vulnerable to Denial of Service attacks or other attacks that could affect service over an extended period of time. Although DDOS attacks are so 2006, it’s worth remembering what happened to those Internet offshore betting systems held to ransom by distributed DOS attacks a few years ago…

Or attackers could use your system as a launch platform to attack other more valuable systems – by compromising and exploiting your connectivity and trust relationships with other, more strategic or valuable systems or organizations.

Starting early with the idea of identifying assets under threat naturally supports threat modeling later.

Attack Surface: Finding the open doors and windows

Once you know what’s valuable to bad guys, you need to consider how they could get it. Look at your systems, identify and understand the paths into and out of the system, and what opportunities are offered to an attacker. Do this through high-level attack surface analysis: walk around the house, check to see how many doors and windows there are, and how easy they are to force open. The idea behind attack surface analysis is straightforward: the fewer entry points, and the harder they are to access, the safer you are.

Most enterprise systems, especially e-commerce systems, have a remarkable number of interfaces to clients and to other systems, including shared services and shared data. Focus on public and remote interfaces. What is reachable or potentially reachable by outside parties, especially unauthenticated / anonymous access?

What other systems are we sharing information with and how? What do we know about these other systems, can we trust them, rely on them?

Is the application client-facing across a private network, or public-facing across the Internet? Do you offer an application API to partners or customers to build their own interfaces? A desktop client? A browser-based web client? How much dynamic content? Online ordering, queries? How rich an experience does your client provide, using what technologies: Ajax, Flash, Java, Silverlight, …? What about mobile clients?

How much personalization and customization do you offer to your customers: more options and more combinations to design, test and review, more chances to miss something or get something wrong.

Then you look behind the attack surface to the implementation details for each interface, at the technology stack, the trust boundaries, look at the authorization and authentication controls, trace the flow of control and flow of data. And do this each time you make changes or create a new type of interface. This is the basis of threat modeling.

Capabilities and Countermeasures: How hard / expensive is it for them to get in?

What is the state of your defenses, your protective controls?

How secure is the network? Is the server OS, database, and the rest of the technology stack hardened? Are your patches up to date – are you sure?

Was the application built in a secure way? Has it been maintained in a secure way? Is it deployed and operated in a secure way? How do you know? When was your last pen test? Your last security design review? How many outstanding security vulnerabilities or bugs do you have in your development backlog?

Are there any regulatory or legal requirements that need to be satisfied? PCI, HIPAA, GLBA, … Do you understand your obligations, are you satisfying them?

Do you know enough to know how much trouble you might be in? What is the security posture/capability of the organization, of your team? Do you have someone (or some group) in the company responsible for setting security policies and helping with security-based decisions? Has the development team been trained in security awareness, and in defensive coding and security testing? Is the team following a secure SDLC? Are you prepared to deal with security incidents – do you have an incident response team, do they know what to do?

It is important for the stakeholders to understand upfront what we know and what we don’t know. How confident they should be in the team’s ability to deliver a secure solution – and the team’s ability to understand and anticipate security risks in the first place. There are a couple of good, freely-available frameworks for assessing your organization’s software security capabilities. OWASP’s SAMM framework is one that I have used before with some success. Another comprehensive organizational assessment framework is Cigital’s BSIMM which has been built up using data from 30 different software security programs, generally at larger companies.

Back to Managing the Risks

Now you can assess your risk exposure: the likelihood of a successful attack, and the impact, the cost to your company if an attack was successful. With this information you can decide how much more to spend on defenses, and put into place a defensive action plan.

Risk management is done at the strategic level first: working with business stakeholders, managers, the people who make business decisions, who decide where and how to spend money. You need to describe risks in business terms, spend some time selling and educating. The point here is to secure a mandate for security: an agreement with the people who run the business on security priorities and goals and budgets.

Then you move to tactical implementation of your security mandate: figuring out where and how much to focus, what technical and project-based trade-off decisions to make within your mandate.

Start at a high level, with a rough H/M/L rule-of-thumb rating of the probability and impact of each risk or risk area. Use this initial assessment, and your understanding of the team’s security capabilities, to determine where to dig deeper and how deep to dig, where you will need to focus more analysis, reviews, testing.

As you find more, and more specific risks and weaknesses through threat modeling or architectural risk analysis, through code reviews, or pen testing or fuzzing or whatever, consider using Microsoft’s DREAD risk assessment model. I like this model because it asks you to evaluate each risk from different perspectives, forces you to answer more questions:

D: Damage potential, what would happen to the business?
R: Reproducibility – how often does the problem come up, does it happen every time?
E: Exploitability – how easy is to take advantage of, how expensive, how skilled does the bad guy need to be, what kind of tools does he need?
A: Affected users – who (how important) and how many?
D: Discoverability: how easy is it to find?

As you identify risks, you go back and apply generic risk management methods to determine the cost trade-offs and your response:

• accept (plan for the cost, or choose to ignore it because the likelihood is small and/or the cost to fix is high)
  
• avoid (do something else, if you can)
• prevent (plug the hole, fix the problem)
  
• reduce (take steps to reduce the likelihood or impact: put in early warning tools to detect, like an IDS, so that you can react quickly; or contain the risk through fire walls, isolation and partitioning, layered defenses, ...).

In the end, risk management comes down to the the same delicate balance of the same decision-making factors:

• informed judgment
  
• legal requirements and contractual commitments
  
• the business’ general tolerance for any kind of risk, and the company’s willingness to face risks – a startup for example has a much higher tolerance for risk than an established brand
  
• politics: some issues aren’t fundamentally important to the business, but they are important to somebody important – and some issues are important to the business (or should be), but unfortunately aren’t important to anybody important
  
• and economic trade-offs – the cost today vs the potential cost tomorrow.

And then do this over and over again as your circumstances change, as you get new information, as the company's risk profile changes, as the threat landscape changes. The decisions are the same: you just take a different path to get there.

Kanban, Scrum/XP and the Paradox of Constraints

Building and maintaining software is about confronting limits:

• Limits on the time that you have to get your work done.
  
• Limits on the money that you have to spend, on the talent that you can get.
  
• The limits of your technology platform, on its performance and reliability and cost of operations.
  
• Limits imposed by decisions that have already been made, decisions on architecture and design that limit how much you can change, how safely and how quickly.
  
• And you must maintain compatibility with legacy systems and interfaces to partners.
  
• You may be forced to work with certain suppliers and partners, even if they can’t meet all of your needs.
  
• You will have to meet hard, unmovable deadlines.
  
• You will have to deal with regulatory oversight and legal compliance dictates, and follow change control and change management controls that will slow you down and make your work more expensive.
  
• And in maintenance you have a running system, people rely on it to get their jobs done, and they expect it and need it to work a certain way.
  

All of these constraints box you in. They make your work more difficult and expensive. Project Managers are taught to identify and understand constraints and manage them as sources of risk and cost.

But there’s a wonderful paradox in constraints and limits that I enjoy thinking about:

Constraints take control away from you: by dictating, by forcing you to think and work a certain way, by limiting your options.

But…

Constraints help you to take control: by dictating, by forcing you to think and work a certain way, by limiting your options.

The story of Chandler in Dreaming in Code made a strong case for building software under constraints, as you follow the agonizing unraveling of a project that set out to solve too much, with too much money, too much time, and too much talent. By contrast, the book also profiled 37signals, the small team behind Ruby on Rails, which set out to build simple software with almost no resources. Their ethos is to “Embrace Constraints”:

“Constraints drive innovation and force focus. Instead of trying to remove them, use them to your advantage.”

This is the point of software development management frameworks like Scrum, Extreme Programming and Kanban: to place practical limits on the way that the team works, to force the team into a box, and so control the way that work is done.

Constraints in Scrum and XP

Scrum and XP force the team to work in fixed, short time boxes. Time boxes limit how much work that you can consider and take on at one time. You have to come up with a solution in a short time period, and while it may be a sub-optional solution (you probably would have thought of something smarter if you had more time), at least you get something done and can test the results. And there is a good chance that you will be able to build on what you have done so far, fill it in some more in later steps.

Time boxes serve as a hedge against perfectionism and gold plating and procrastination. Forcing people to work under time constraints helps fight off Parkinson’s Law: that work expands to fill the time available for its completion. It blows the whistle on people who don’t want to let go, who continue to redesign, rewrite, refactor, tune, polish and fiddle with code longer than they need to. And being pushed forward from one deadline to the next keeps you from over-thinking, from letting the problem space or solution space expand.

Doing work in small batches with fixed time limits helps to manage delivery risk. You can only get so far off track in a short time, and if it turns out that you got it wrong and you have to throw it all away, you aren’t wasting too much. And it shouldn’t take too long to understand why you failed, how to get back on track. Small decisions are easier and safer to make, and to unmake. And frequent planning gives you the flexibility to adapt to changing circumstances and take new information into account, to reset and re-prioritize.

With time boxes you are forced to work "in the small”, to think, really think, about how to get work done. It’s all about execution: who, what, when, what happens first, second. It creates a sense of urgency. And a sense of satisfaction, in seeing work done, in the feedback that you get.

In his book Succeeding with Agile Development Using Scrum, Mike Cohn makes the point that the pressure to deliver working software on a regular basis drives teams to get better at their work, to become more efficient, to adopt better technical practices and disciplines like automated testing and Continuous Integration, to get or build better tools.

And the time box becomes a commitment to management and to the customer, it forces your work to be more predictable and transparent. Your customers can plan around a regular delivery cycle, and make their own commitments.

Lean and Kanban: Constraining Work in Progress

I have spent the last few months looking into Kanban for software development, to see what we can learn from and use. Kanban, based on Lean Manufacturing principles, also constrains the amount of work that the team does, but in different ways. Rather than limiting the amount of time that the team has available to get their work done, you explicitly limit the number of things that the team can do at any point in time: the amount of Work in Progress. Each person works on one thing at a time (with maybe some small allowance for issues that could get blocked), following the idea that if small batches are good, then a batch size of one is optimal.

The team uses input queues and buffers between steps (for example: analysis, design, coding, testing, deployment) to smooth out work as it is being done:

• to minimize slack (over capacity) and stalls (under capacity);
  
• to account for variability in the work to be done (a fundamental problem with applying Kanban to software development: work that is easy to code, but hard to test; or hard to code but easy to test, and so on); and
  
• to adjust as the workload or the team changes, and as you learn more about how you are doing and how to do it better.
  

The goal is to keep the input queues and the buffers as small as possible, just big enough to keep the work constantly moving (a constant state of flow).

In Kanban, like in Scrum and XP, planning and prioritization is done just-in-time. The difference is that in a time boxed method, planning is done at the beginning of each time box; in Kanban, planning and prioritization are decoupled from the rest of the work. Prioritization can be scheduled at a regular interval, or done on demand, just before the work starts – again with a small amount of buffering to make sure that team members don’t stall waiting for work. You decide with the customer what to do next and qualify it, put it into the queue, and then work is pulled based on priority and the team member’s ability to do the job. This forces you and the customer to always be ready, to focus on what is important.

Planning further out is considered waste, since the situation may change and make your plans (like most plans) out dated.

You don’t accept more work than you can take on immediately. If you run into problems downstream, the queues will fill up, and the pipeline of work upstream will stall – there’s no point taking on more analysis and estimation work if the team is falling behind on building and testing and deploying code. This shines a bright light on inefficiencies and bottlenecks. As your work changes, and the bottleneck moves, you recognize this and adapt. And you constantly look for ways to eliminate unnecessary work, rework, delays and blocks – anything that stops you from getting work done as efficiently as possible.

Constraints as a Management Tool

Limiting the amount of work that the team can do at one time, either through time boxing or setting limits on work in progress queues, is a coarse management tool but an effective one.
I’ve worked with time boxing methods at different companies and come to rely heavily on this approach. Time boxing works, and today there are good tools and supporting practices that make time boxing easier to implement and sustain.

If you find it difficult to break your work into time boxes, if your work is highly interrupt-driven like in many maintenance and support shops, then Kanban offers an alternative control structure. I am not sure that I buy into its manufacturing control-line metaphor, and that all of these ideas map well from manufacturing to the highly variable work involved in building and supporting software. I think that there is more work that still needs to be done by the community, to make the fit cleaner and clearer. More people need to try it out, see what works, and whether it sticks. But I understand why Kanban could work. It makes explicit what many well-run teams already do: track and control and buffer work demands across different people on the team.

Whether you follow any of these methods is not important. What is important is that you understand constraints. Understand how constraints take control away, and how they can be used to take control back - how use them to your advantage.

Code quality, refactoring and the risk of change

When you are working on a live, business-critical production system, deciding what work needs to be done, and how to do it, you need to consider different factors:

1. business value: the demand and urgency for new business features and changes requested by customers or needed to get new customers onboard.


2. compliance: ensuring that you stay onside of regulatory and security requirements.


3. operational risk and safety: the risks of injecting bugs into the live system as a side-effect of your change, the likelihood and impact of errors on the stability or usability of the system.


4. cost: immediate development costs, longer-term maintenance and support costs, operational costs and other downstream costs, and opportunity costs of choosing one work item over another or one design approach over another.


5. technical: investments needed to upgrade the technology stack, managing the complexity and quality of the design and code.

These factors also come into play in refactoring: deciding how much code to cleanup when making a change or fix. The decision of what, and how much to refactor, isn’t simple. It isn’t about a developer’s idea of what is beautiful and their need for personal satisfaction. It isn’t about being forced to compromise between doing the job the right way vs. putting in a hack. It’s much more difficult than that. It’s about balancing technical and operational risks, technical debt, cost factors, and trading off short-term advantages for longer-term costs and risks.

Let’s look at the longer term considerations first. A live system is going to see a lot of changes. There will be regulatory changes, fixes and new features, upgrades to the technology platform. There will also be false starts and back-tracking as you iterate, and changes in direction, and short-sighted design and implementation decisions made with insufficient time or information. Sometimes you will need to put in a quick fix, or cut-and-paste a solution. You will need to code-in exceptions, and exceptions to the exceptions, especially if you are working on an enterprise system integrated with tens or hundreds of other systems. People will leave the team and new people will join, and everyone’s understanding of the domain and the design and the technology will change over time. People will learn newer and better ways of solving problems and how to use the power of the language and their other tools; they will learn more about how the business works; or they might forget or misunderstand the intentions of the design and wander off course.

These factors, the accumulation of decisions made over time will impact the quality, the complexity, the clarity of the system design and code. This is system entropy, as described by Fred Brooks back in the Mythical Man Month:

“All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent on fixing the original design flaws: more and more is spent on fixing flaws introduced by earlier fixes… Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one.”

So, the system will become more difficult and expensive to operate and maintain, and you will end up with more bugs and security vulnerabilities – and these bugs and security holes will be harder to find and fix. At the same time you will have a harder time keeping together a good team because nobody wants to wade knee deep in garbage if they don’t have to. At some point you will be forced to throw away everything that you have learned and all the money that you have spent, and build a new system. And start the cycle all over again.

The solution to this is of course is to be proactive: to maintain the integrity of the design by continuously refactoring and improving the code as you learn, fill-in short-cuts, eliminate duplication, clean up dead-ends, simplify as much as you can. In doing this, you need to balance the technical risks and costs of change in the short-term, with the longer-term costs and risks of letting the system slowly go to hell.

In the short term, we need to understand and overcome the risk of making changes. Michael Feathers, in his excellent book Working Effectively with Legacy Code talks about the fear and risk of change that some teams face:

“Most of the teams that I’ve worked with have tried to manage risk in a very conservative way. They minimize the number of changes they make to the code base. Sometimes this is a team policy: ‘if it’s not broke, don’t fix it’…. ‘What? Create another method for that? No, I’ll just put the lines of code right here in the method, where I can see them and the rest of the code. It involves less editing, and it’s safer.’

It’s tempting to think we can minimize software problems by avoiding them, but, unfortunately, it always catches up with us. When we avoid creating new classes and methods, the existing ones grow larger and harder to understand. When you make changes in any large system, you can expect to take a little time to get familiar with the area you are working with. The difference between good systems and bad ones is that, in the good ones, you feel pretty calm after you’ve done that learning, and you are confident in the change you are about to make. In poorly structured code, the move from figuring things out to making changes feels like jumping off a cliff to avoid a tiger. You hesitate and hesitate.

Avoiding change has other bad consequences. When people don’t make changes often they get rusty at it…The last consequence of avoiding change is fear. Unfortunately, many teams live with incredible fear of change and it gets worse every day. Often they aren’t aware of how much fear they have until they learn better techniques and the fear starts to fade away.”

It’s clear that avoiding changes won’t work. We need to get and keep control over the situation, we need to make careful and disciplined changes. And we need to protect ourselves from making mistakes.

Back to Mr. Feathers:

“Most of the fear involved in making changes to large code bases is fear of introducing subtle bugs; fear of changing things inadvertently”.

The answer is to ensure that you have a good testing safety net in place (from Michael Feathers one more time):

“With tests, you can make things better with impunity… With tests, you can make things better. Without them, you just don’t know whether things are getting better or worse.”

You need enough tests to ensure that you understand what the code does; and that you target tests that will will detect changes in behavior in the area that you want to change.

Put in a good set of tests. Refactor. Review and verify your refactoring work. Then make your changes and review and verify again. Don’t change implementation and behavior at the same time.

But there are still risks – you are making changes, and there are limits to how much protection you can get from developer testing, even if you have a high level of coverage in your automated tests and checks. Ryan Lowe reinforces this in “Be Mindful of Code Entropy”:

“The reason you don’t want to refactor established code: this ugly working code has stood the test of time. It’s been beaten down by your users, it’s been tested to hell in the UI by manual testers and QA people determined to break it. A lot of work has gone into stabilizing that mess of code you hate.

As much as it might pain you to admit it, if you refactor it you’ll throw away all of the manual testing effort you put into it except the unit tests. The unit tests… can be buggy and aren’t nearly as comprehensive as dozens/hundreds/thousands of real man hours bashing against the application.”

As with any change, code reviews will help find mistakes, and so will static analysis. We also ask our developers to make sure that the QA team understands the scope of their refactoring work so that they can include additional functional and system regression testing work.

Beyond the engineering discipline, there is the decision of how much to refactor. So as a developer, how do you make this decision, how do you decide how much is right, how much is necessary? There is a big difference between minor, in-phase restructuring of code and just plain good coding; and fundamental re-design work, what Martin Fowler and Kent Beck call “Big Refactorings” which clearly need to be broken out and done as separate pieces of work. The answer lies somewhere in between these points.

I recently returned from a trip to the Buddhist kingdom of Bhutan, where I was reminded of the value of finding balance in what we do, the value of following the Middle Way. It seems to me that the answer is to do “just enough”. To refactor only as much as you need to make the problem clear, to understand better how the code works, to simplify the change or fix… and no more.

By doing this, we still abide by Bob Martin’s Boy Scout Rule and leave the code cleaner than when we checked it out. We help protect the future value of the software. At the same time we minimize the risk of change by being careful and disciplined and patient and humble. Just enough.

Why isn't Risk Management included in Scrum?

There are a lot of fundamentally good ideas in Scrum, and a lot of teams seem to be having success with it, on more than trivial projects. I’m still concerned that there are some serious weaknesses in Scrum, especially “out of the box”, where Scrum leaves you to come up with your own engineering framework, asks you to come up with your own way to build good software.

I am happy to see that recent definitions (or reinterpretations) of Scrum, for example in Mike Cohn’s book Succeeding with Agile Software Development Using Scrum incorporate many of the ideas from Extreme Programming and other basic good management and engineering practices to fill in some of the gaps.

But I am not happy to see that risk management is still missing in Scrum. There is no discussion of risk management in the original definition of Scrum, or in the CSM training course (at least when I took it), the latest Scrum Guide (although with all of the political infighting in the Scrum community, I am not sure where to find the definitive definition if such a thing exists any more), not even in Mike Cohn’s new book.

I just don’t understand this.

I manage an experienced development and delivery team, and we follow an incremental, iterative development approach based on Scrum and XP. These are smart, senior people who understand what they are doing, they are talented and careful and disciplined, and we have good tools and we have a lot of controls built in to our SDLC and release processes. We work closely with our customer, we deliver software often (every 2-3 weeks to production, sometimes faster), we continuously review and learn and get better. But I still spend a lot of my time watching out for risks, trying to contain and plan for business and technical and operational risks and uncertainties.

Maybe I worry too much (some of my colleagues think so). But not worrying about the risks won’t make them go away. And neither will following Scrum by itself. As Ken Schwaber, one of the creators of Scrum, admits:

“Scrum purposefully has many gaps, holes, and bare spots where you are required to use best practices – such as risk management.”

My concern is that this should be made much more clear to people trying to follow the method. The contrast between Extreme Programming and Scrum is stark: XP confronts risk from the beginning, and the idea and importance of managing project and technical risks is carried throughout XP. Kent Beck, and later others including Martin Fowler, examine the problems of managing risk in building software, prioritizing work to take into account both business value and technical risk (dealing with the hard problems and most important features as early as possible), and outlining a set of disciplines and controls that need to be followed.

In Scrum, some management of risks is implicit, essentially burned-in to the approach of using short time-boxed sprints, regular meetings and reviews, more efficient communications, and a self-managing team working closely with the customer, inspecting and adapting to changes and new information. I definitely buy into this, and I’ve seen how much risk can be managed intrinsically by a good team following incremental, iterative development.

As I pointed out earlier, Scrum helps to minimize scope, schedule, quality, customer and personnel risks in software development through its driving principles and management practices.

But this is not enough, not even after you include good engineering practices from XP and Microsoft’s SDL and other sources. Some leaders in the Agile community have begun to recognize this, but there is an alarming lack of consensus within the community on how risks should be managed:

- by the team, intrinsically: like any other requirement or issue, members of the self-managing team will naturally recognize and take care of risks as part of their work. I’m not convinced that a development team is prepared to naturally manage risks, even an experienced team – software development is an essentially an optimistic effort, you need to believe that you can create something out of nothing, and most of the team’s attention will naturally be on what they need to do and how to best get it done rather than trying to forecast what could go wrong and how to prepare for it;

- by the team, explicitly: the team owns risk management, they are responsible for identifying and managing risks using a risk management framework (risk register, risk assessment and impact analysis, and so on);

- by the ScrumMaster, as another set of issues to consider when coaching and supporting the team, although how the ScrumMaster then ensures that risks are actually addressed is not clear, since they don’t control what work gets done and when;

- by the Product Owner: this idea has a lot of backing, supposedly because the Product Owner acting for the customer in the end understands the risk of failure to the business best. The real reason is because the team can resign themselves of the responsibility for risk management and hand it off to the customer. This makes me angry. It is unacceptable to suggest that someone representing the customer should be made responsible for assuming the risk for the project. The customer is paying your organization to do a job, and naturally expecting you as the expert to do this job professionally and competently. And yet you are asking this same customer to take responsibility for the risk that the project won’t be delivered successfully? Try selling that to any of the enterprise customers that I have worked with over the past 15 years…

Risk management for a Scrum project needs to take this into account, and take into account all of the risks and issues that are external to the project team:

- the political risks involved in the team’s relationship with the Product Owner, and the Product Owner’s position and influence within their own organization. Scrum places too much responsibility on the Product Owner and requires that this person is not only competent and committed to the project and deeply understands the needs of the business; but that they also always put the interests of the business ahead of their own, and of course this assumes that they are not in conflict with other important customer stakeholders over important issues. They also need to understand the importance of technical risk, and be willing to trade-off technical risk for business value in planning and scheduling work in the backlog. I am lucky in that I have the opportunity to work with a Product Owner who actually meets this profile. But I know how rare and precious this is;

- the political risks of managing executive sponsors and other key stakeholders within the customer and within your own organization - it's naive to think that delivering software regularly and keeping the Product Owner happy is enough to keep the support of key sponsors;

- financial and legal risks – making sure that the project satisfies the legal and business conditions of the contract, dealing with intellectual property and confidentiality issues, and financial governance;

- subcontractors and partners – making sure that they are not going to disappoint or surprise you, and that you are meeting your commitments to them;

- infrastructure, tooling, environmental factors – making sure that the team has everything that they need to get the job done properly;

- integration risks with other projects - requirements for program management, coordinating and communicating with other teams, managing interdependencies and resource conflicts, handling delays and changes between projects;

- rollout and implementation, packaging, training, marketing – managing the downstream requirements for distribution and deployment of the final product.

So don’t throw out that PMP.

It’s clear that you need real and active project management and risk management on Scrum projects, like any project: focused not so much inwards, but outwards. So much of what you read about Scrum, and Agile project management generally, is naive in that it assumes that the development work is being done in a vacuum, that there is nothing that concerns the project outside the immediate requirements of the development team. Even assuming that the team is dealing effectively with technical risks, and that other immediate delivery risks are contained by being burned-in to doing the work incrementally and following engineering best practices, you still have to monitor and manage the issues outside of the development work, the larger problems and issues that extend beyond the team and the next sprint, the issues that can make the difference between success and failure.

Software Project Manager's Bridge to Agility

The Software Project Manager’s Bridge to Agility by Michele Sliger and Stacia Broderick sets out to map the PMBOK project management practice framework to an Agile practice framework (essentially Scrum) for managing software development projects.

The book introduces Agile project management concepts, then explains how the PMBOK practices for Integration Management, Scope Management, Time Management, Cost Management, Quality Management, Human Resources Management, Communications Management, Risk Management and Procurement Management are realized in an Agile project management approach. This analysis is necessarily repetitive as Agile project management is essentially simple, solving a broad set of different problems using the same set of basic principles and techniques: a flat, collaborative team structure where the project manager primarily acts as a coach and facilitator; controlling scope and quality and managing risk through time-boxing and frequent reviews; maintaining a close working relationship with the customer through its representative product owner; and lightweight (just enough and no more) documentation.

The authors are certified PMPs and Agile consultants so they have a foot in both worlds, although it is clear where their bias (and their source of incomes) lays. The book that they have written is not a bad book – it offers an interesting analysis of two essentially different approaches to managing software projects, and reinforces the value of fundamental Agile development principles. But it falls short in a number of key areas:

Like too much writing on Agile management, the book offers a simplistic and naïve view of the scope of a project manager's responsibilities, and minimizes especially the need for preparation and planning and design. I suppose my concern here is not so much with the book, as with the weakness of the approach advocated: but to act as a bridge between the PMBOK and Agile management, the book should address the needs and concerns of project managers, especially for projects that are not trivial. In the Agile framework described in the book, there is an unfortunate lack of upfront planning and preparation: it essentially consists of some vague work on the part of the project manager to secure sponsorship for the project, and an upfront vision meeting where the team (who, naturally at this point, likely knows nothing at all about the domain or the problems to be solved) meets with the customer and together lays out the high-level boundaries of the project and the priorities and features (the Backlog), and agrees on how to describe the project’s vision in a few sentences (an Elevator Statement).

This raises a number of concerns of course, but one of the most fundamental is a bootstrapping problem: if you haven’t prepared, done some research and upfront analysis, if you don’t know anything about what you are being asked to try to build, then how do you decide who should be on the team? The fundamental assumption here is that the team is an initial constraint: you get a small group of cross-trained generalized specialists who can handle any kind of problem with minimal upfront work, and make do with that team. For simple business applications, I am sure this could work. For complex science or engineering problems that demand specific domain or technical (or both) experience to even understand what is going on and what the customer is talking about, never mind delivering what the customer needs…. umm, no.

Like others in the Agile community, the authors allow for an “Iteration Zero”: some brief upfront work to setup the environment, for ramp-up and discovery, approvals, design discussions, maybe some training for the team in the domain and tools. But the authors encourage the team to deliver working software in this first (zeroeth actually) iteration, again short changing time for upfront analysis and discovery.

I am a strong advocate of incremental, iterative design and development, and I fully appreciate the concerns of wasting time and resources in what Steve McConnell calls the “fuzzy front end” of a project. But rushing through the initial analysis and discovery and technical planning and integration planning steps before you get a chance to understand something about the domain and the technical requirements and the preparedness of the team would be suicide if you were managing a project that was part of a large program, or an embedded software development project, or building a safety-critical system, or really anything more than a trivial business system or maybe a web portal: and even here you are dependent on the team’s experience and preparedness, their technical competence to fill in the necessary gaps and catch up.

The Agile approach described in the book for project estimating and scheduling is fundamentally weak as well: there is no attempt (at least that I could find) to create a high-level schedule for the project. At the start of the project, in the project vision meeting

the scope is defined a very high level. It is not uncommon to leave the vision meeting with only a dozen or so features identified…

In the section on Time Management, the book explains how estimates are made “at the strategic level” – the level of release planning – through abstract story point estimates, although the method by which the team translates story points into useful measures of time and cost is not made clear (yes I know how it works, but you won’t understand it from this book). A further weakness here is the approach taken to create the estimates: the team picks a feature, estimates it in an abstract way based on their understanding of its complexity, then goes on to estimate the complexity of other features relative to the first one. Of course, the first feature is the least likely to be estimated correctly, as the team has the least amount of information available when preparing that estimate. All of the other estimates are calibrated to this first estimate, amplifying its inaccuracy.

The only time that the team does any substantial estimating is when they are working through the details at the start of each iteration inside a release. Although there is a reference to the team using a “prioritized, estimated product backlog” in release planning, it is not clear when these estimates are created, or how the project schedule itself is defined – i.e., how the team arrives at a date by which the project is scheduled to be completed.

Now I understand and agree that estimates done in initial planning of a project are notional only, that there is a wide variance in accuracy of these estimates, and that the accuracy of estimates will improve over time. This is commonly understood in software development, inside and outside of the Agile community. But there needs to be some attempt at estimating the scope of delivery as understood at the beginning so that the project can be chartered, so that money and people and supporting resources can be committed; and, if this is more than a trivial software development project, so that requirements of other teams working on related projects or programs can be understood, integration points and dependencies defined, and commitments made. This is a fault of the book and not of the method: there is no clear tie-in from project initiation to the planning of each release in the book, although there must be one.

The book also assumes a narrow scope for the project manager’s, and the team’s, responsibilities and concerns: that they are focused only on developing the software, and don’t need to be concerned with integration and coordination with other projects or programs. The authors do allow for an “Iteration H”, a bracketing iteration at the end of the project for “hardening” the system: preparing documentation, packaging, training operations and so on – oh and of course leave time for celebration. Ah, if project management was only that simple. While the book makes a brief reference to the need for the PM to take care of external dependencies, it does not explain how these are managed within the Agile context – as far as I can see, the answer is: they are not.

I was hoping that the book would offer some ideas on how to manage Agile projects with strict contracting models and fixed constraints. Unfortunately again, this concern is disregarded:

“Agile projects have the ability to always be on time, and within budget, if the scope is flexible”.

Umm, stating the obvious, yes this is true. But if you are not delivering what the customer needs, or asked for in the contract, when they need it, what is the point? The book doesn’t offer guidance to the PM on how to reconcile iterative delivery and elastic scope with the cost and time and other business constraints of the contract, and other constraints in a larger program: in any real project, the work of other teams is dependent on your team delivering an expected scope of work by an expected date. Instead, the book assumes that the team working with the customer’s product owner will be able to resolve changes in scope collaboratively, and presumably that there will be no need to worry about contractual terms and commitments and other business issues. In this world nobody appears to be accountable for managing the tie-in back to the contract: as long as working software is delivered, everyone will be happy and everyone will be paid.

It is disappointing, but not a surprise, that the authors don’t offer ideas on how to effectively manage fixed-price, fixed-schedule projects using Agile techniques. Ken Schwaber, one of the authors of the Scrum method, admits in Agile Project Management with Scrum that he does not know of a way to deal with this problem either, and suggests trying a different contracting model based on essentially conceding upfront that you won’t be able to meet the customer’s requirements, and selling them instead on an iterative, collaborative business model. If you are able to stay in business trying this, good on ya mate.

The book does address Risk Management, primarily relying on the intrinsic risk management practices in iterative, incremental development. I agree that iterative, incremental development in short time-boxes, coupled with solid engineering practices, significantly reduces many kinds of project risks – this is why we follow this approach. The book does go further to describe a basic model for those teams who decide that they need to do explicit risk management – reviewing risks and maintaining a Risk Board for each iteration. This is a good approach, but does not take into account larger or longer-term risks: the team is focused on dealing with risks and technical issues immediately in front of them.

My biggest concern with this book is that it takes such a fundamentally simplistic and unnecessarily critical view of “traditional plan-driven projects”. In these projects quality is short changed; project managers only go through the motions on risk management; and of course the planning process does not allow for iteration – all projects except for Agile projects follow a pure waterfall model and attempt to plan everything upfront in detail and fail manage to this plan. Where does this come from? Almost all large projects that I am aware of are managed through spiral or phased development SDLCs, and include prototyping and other iterative techniques. Rolling wave planning has been a recognized best practice in project management for a long time. It is absurd to take the position that PMs managing large, complex projects and programs are not aware of, and not following, these practices and that the only alternative is to follow Scrum.

It’s not clear to me who the audience for this book is. Project Managers (specifically PMPs) who don’t understand Agile methods? But how is this possible with all of the noise about Agile methods, and taking into account that Agile methods are essentially and purposefully simple – you can learn the Scrum approach in an afternoon, and a Certified Scrum Master course involves only a couple days of training.

The naïve and critical view that the authors take of planning and project management will not convince PMPs who manage large or complex projects that they will be more successful following Agile techniques. Or is the audience inexperienced managers who are trying to chose between the PMBOK and Agile methods, not understanding that they can (and should, and in fact if they are managing anything of any complexity, must) do both?