Just how effective is static analysis, what does it protect you from?
There is a lot of attention given to static analysis tools, especially from the software security community - and some serious venture capital money being thrown at static analysis tool providers such as Coverity.
The emphasis on using static analysis tools started with Cigital's CTO Gary McGraw in his definitive book on Software Security: Building Security In. In a recent interview with Jim Manico for OWASP (Jan 2009), Dr. McGraw went so far as to say that
“My belief is that everybody should be using static analysis tools today. And if you are not using them, then basically you are negligent, and you should prepare to be sued by the army of lawyers that have already hit the beach”.
Statements like this, from a thought leader in the software security community, certainly encourage companies to spend more money on static analysis tools, and of course should help Cigital’s ownership position in leading static analysis tool provider Fortify Software, which Cigital helped to found.
You can learn more about the important role that static analysis plays in building secure systems from Brian Chess, CTO of Fortify, in his book Secure Programming with Static Analysis.
Secure software development maturity models like SAMM and BSIMM emphasize the importance of code reviews to find bugs and vulnerabilities, but especially the use of static analysis tools, and OWASP has a number of free tools and projects in this area.
Now even Gartner has become interested in this the emerging emerging static analysis marketand its players – evidence that the hype is reaching, or has reached, a critical point. In Gartner’s study of what they call Static Application Security Testing (SAST) suppliers (available from Fortify), they state that
“...enterprises must adopt SAST technology and processes because the need is strategic. Enterprises should use a short-term, tactical approach to vendor selection and contract negotiation due to the relative immaturity of the market.” Well there you have it: whether the products are ready or not, you need to buy them.
Gartner’s analysis puts an emphasis on full-service offerings and suites: principally, I suppose, because CIOs at larger companies, who are Gartner’s customers, don’t want to spend a lot of time finding the best technology and prefer to work with broad solutions from strategic technology partners, like IBM or HP (neither of which has strong static analysis technology yet, so watch for acquisitions of the independents to fill out their security tool portfolios, as they did in the dynamic analysis space). Unfortunately, this has led vendors like Coverity to spend their time and money on filling out a larger ALM portfolio, building and buying in technology for build verification, software readiness (I still don’t understand who would use this) and architecture analysis rather than investing in their core static analysis technology. On their site, Coverity proudly references a recent story "Coverity: a new Mercury Interactive in the making?", which should make their investors happy and their customers nervous – as a former Mercury Interactive, now HP customer, I can attest that while the acquisition of Mercury by HP may have been good for HP and good for Mercury’s investors, it was not good for Mercury’s customers, at least the smaller ones.
The driver behind static analysis is its efficiency: you buy a tool, you run it, it scans thousands and thousands of lines of code and finds problems, you fix the problems, now you’re secure. Sounds good, right?
But how effective is static analysis? Do these tools find real security problems?
We have had success with static analysis tools, but it hasn’t been easy. Starting in 2006, some of our senior developers started working with FindBugs (we’re a Java shop) because it was free, it was easy to get started with, and it found some interesting, and real, problems right away. After getting to understand the tool and how it worked, some cleanup and a fair amount of time invested by a smart, diligent and senior engineer to investigate false positives and setup filters on some of the checkers, we added FindBugs checking to our automated build process, and it continues to be our first line of defense in static analysis. The developers are used to checking the results of the FindBugs analysis daily, and we take all of the warnings that it reports seriously.
Later in 2006, as part of our work with Cigital to build a software security roadmap, we conducted a bake-off of static analysis tool vendors including Fortify, Coverity, Klocwork (who would probably get more business if they didn't have a cutesy name that is so hard to remember), and a leading Java development tools provider whose pre-sales support was so irredeemably pathetic that we could not get their product installed, never mind working. We did not include Ounce Labs at the time because of pricing, and because we ran out of gas, although I understand that they have a strong product.
As the NIST SAMATE study confirms, working with different tool vendors is a confusing and challenging and time-consuming process: the engines work differently, which is good since they catch different types of problems, but there is no consistency in the way that warnings are reported or rated, and different terms are used by different vendors to describe the same problem. And there is the significant problem of dealing with noise: handling the large number of false positives that get reported by all of the tools (some are better than others), understanding what to take seriously.
At the time of our initial evaluation, some of the tools were immature, especially the C/C++ tools that were being extended into the Java code checking space (Coverity and Klocwork). Fortify was the most professional and prepared of the suppliers. However, we were not able to take advantage of Fortify’s data flow and control flow analysis (one of the tool’s most powerful analysis capabilities) because of some characteristics of our software architecture. We verified with Fortify and Cigital consultants that it was not possible to take advantage of the tool’s flow analysis, even with custom rules extensions, without fundamentally changing our code. This left us with relying on the tool’s simpler security pattern analysis checkers, which did not uncover any material vulnerabilities. We decided that with these limitations, the investment in the tool was not justified.
Coverity’s Java checkers were also limited at that time. However, by mid 2007 they had improved the coverage and accuracy of their checkers, especially for security issues and race conditions checking, and generally improved the quality of their Java analysis product. We purchased a license for Coverity Prevent, and over a few months worked our way through the same process of learning the tool, reviewing and suppressing false positives, and integrating it into our build process. We also evaluated an early release of Coverity’s Dynamic Thread Analysis tool for Java: unfortunately the trial failed, as the product was not stable – however, it has potential, and we will consider looking at it again in the future when it matures.
Some of the developers use Klocwork Developer for Java, now re-branded as Klocwork Solo, on a more ad hoc basis: for small teams, the price is attractive, and it comes integrated into Eclipse.
In our build process we have other static analysis checks, including code complexity checking and other metric trend analysis using an open source tool JavaNCSS to help identify complex (and therefore high high-risk) sections of code, and we have built proprietary package dependency analysis checks into our build to prevent violation of dependency rules. One of our senior developers has now started working with Structure101 to help us get a better understanding of our code and package structure and how it is changing over time. And other developers use PMD to help cleanup code, and the static analysis checkers included in IntelliJ.
Each tool takes different approaches and has different strengths, and we have seen some benefits in using more than one tool as part of a defense-in-depth approach. While we find by far the most significant issues in manual code reviews or exploratory testing or through our automated regression safety net, the static analysis tools have been helpful in finding real problems.
While FindBugs does only “simple, shallow analysis of network security vulnerabilities", and analysis of malicious code vulnerabilities as security checks, it is good at finding small, stupid coding mistakes that escape other checks, and the engine continues to improve over time. This open source project deserves more credit: it offers incredible value to the Java development community, and anyone building code in Java that who does not take advantage of it is a fool.
Coverity reports generally few false positives, and is especially good for finding potential thread safety problems and null pointer (null return and forward null) conditions. It also comes with a good management infrastructure for trend analysis and review of findings. Klocwork is the most excitable and noisiest of all of our tools, but it includes some interesting checkers that are not available in the other tools – although after manual code reviews and checks by the other static analysis tools, there is rarely anything of significance left for it to consider.
But more than the problems that the tools find directly, the tools help to identify areas where we may need to look deeper: where the code is complex, or too smarty pants fancy, or otherwise smelly, and requires followup review. In our experience, if a mature analysis tool like FindBugs reports warnings that don’t make sense, it is often because it is confused by the code, which in turn is a sign that the code needs to be cleaned up. We have also seen the number of warnings reported decline over time as developers react to the “nanny effect” of the tools’ warnings, and change and improve their coding practices to avoid being nagged. And the final benefit of using these tools is that this frees up the developers to concentrate on higher-value work in their code reviews: they don’t have to spend so much time looking out for fussy, low-level coding mistakes, because the tools have found them already, so the developers can concentrate on more important and more fundamental issues like correctness, proper input validation and error handling, optimization, simplicity and maintainability.
While we are happy with the toolset we have in place today, I sometimes wonder whether we should beef up our tool-based code checking. But is it worth it?
In a presentation at this year’s JavaOne conference, Prof. Bill Pugh, the Father of FindBugs says that
“static analysis, at best, might catch 5-10% of your software quality problems.”
He goes on to say, however, that static analysis is 80+% effective at finding specific defects and cheaper than other techniques for catching these same defects – silent, nasty bugs and programming mistakes.
Prof. Pugh emphasizes that static analysis tools have value in a defense-in-depth strategy for quality and security, combined with other techniques; that “each technique is more efficient at finding some mistakes than others”; and that “each technique is subject to diminishing returns”.
In his opinion, “testing is far more valuable than static analysis”, and “FindBugs might be more useful as an untested code detector than a bug detector”. If FindBugs finds a bug, you have to ask: “Did anyone test that code”?”. In our experience, Prof Pugh’s FindBugs findings can be applied to the other static analysis tools as well.
As technologists, we are susceptible to the belief that technology can solve our problems – the classic “silver bullet” problem. When it comes to static analysis tools, you’d be foolish not to use a tool at all, but at the same time you’d be foolish to expect too much from them – or pay too much for them.
Thursday, June 25, 2009
Wednesday, June 17, 2009
How long can this go on?
Our team delivers software iteratively and incrementally, and over the past 3 years we have experimented with longer (1-2 months) and shorter (1-2 week) iterations, adjusting to circumstances, looking for the proper balance between cost and control.
There are obvious costs in managing an iteration: startup activities (planning, prioritization, kickoff, securing the team's commitment to the goals of the release), technical overheads like source code and build management (branching and merging and associated controls), status reporting to stakeholders, end-to-end system and integration testing, and closure activities (retrospectives, resetting). We don’t just deliver “ship-quality” software at the end of an iteration: in almost every case we go all the way to releasing the code to production, so our costs also include packaging, change control, release management, security and operations reviews, documentation updates and release notes and training, certifications with partners, data conversion, rollback testing, and pre- and post-implementation operations support. Yep, that’s a lot of work.
All of these costs are balanced against control: our ability to manage and contain risks to the project, to the product, and to the organization. I explored how to manage risks through iterative, incremental development in an earlier post on risk management.
We’ve found that if an iteration is too long (a month or more), it is hard to defend the team from changes to priorities, to prevent new requirements from coming in and disrupting the team’s focus. And in a longer cycle, there are too many changes and fixes that need to be reviewed and tested, increasing the chance of mistakes or oversights or regressions.
Shorter releases are easier to manage because, of course, they are necessarily smaller. We can manage the pressure from the business-side for changes because of the fast delivery cycle (except for emergency hot fixes, we are usually able to convince our product owner, and ourselves, to wait for the next sprint since it is only a couple of weeks away) and it is easier for everyone to get their heads around what was changed in a release and how to verify it. And shorter cycles keep us closer to our customers, not only giving us faster feedback, but demonstrating constant value. I like to think of it as a “value pipeline”, continuously streaming business value to customers.
One of my favorite books on software project management, Johanna Rothman’s Manage It!, recommends making increments smaller to get faster feedback – feedback not just on the product, but on how you build it and how you can improve. The smaller the iteration, the easier to look at it from beginning-to-end and see where time is wasted, what works, what doesn’t, where time is being spent that isn’t expected.
Ms. Rothman recommends using the “Divide-by-Two Approach to Reduce Iteration Size”: if the iterations aren’t succeeding, divide the length in half, so 6 weeks becomes 3 weeks, and so on. Smaller iterations provide feedback - longer ones mask the problems.
Ms. Rothman also says that it is difficult to establish a rhythm for the team if iterations are too long. In “Selecting the Right Iteration Length for Your Software Development Process”, Mike Cohn of Mountain Goat Software examines the importance of establishing a rhythm in incremental development. He talks about the need for a sense of urgency: if an iteration is too long, it takes too much time for the team to “warm up” and take things seriously. Of course, this needs to be balanced against keeping the team in a constant state of emergency, and burning everyone out.
Some of the other factors that Mr. Cohn finds important in choosing an iteration length:
- how long can you go without introducing change – avoiding requirements churn during an iteration.
- if cycles are too short (for example, a week) small issues, like a key team member coming down with a cold, can throw the team’s rhythm off and impact delivery.
All of this supports our experience: shorter (but not too-short) cycles help establish a rhythm and build the team’s focus and commitment, constantly driving to delivering customer value. And shorter cycles help manage change and risk.
Now we are experimenting with an aggressive, fast-tracked delivery model: a 3-week end-to-end cycle, with software delivered to production every 2 weeks. The team starts work on designing and building the next release while the current release is in integration, packaging and rollout, overlapping development and release activities. Fast-tracking is difficult to manage, and can add risk if not done properly. But it does allow us to respond quickly to changing business demands and priorities, while giving us time for an intensive but efficient testing and release management process.
We'll review how this approach works over the next few months and change it as necessary, but we intend to continue with short increments. However, I am concerned about the longer-term risks, the potential future downsides to our rapid delivery model.
In The Decline and Fall of Agile James Shore argues that rapid cycling short cuts up-front design:
While Mr. Shore is specifically concerned about loose implementations of Scrum, and its lack of engineering practices compared with other approaches like XP (see also Martin Fowler of ThoughtWorks on the risks of incremental development without strong engineering discipline), the problem is a general one for teams working quickly, in short iterations: even with good engineering discipline, rapid cycling does not leave a lot of time for architecture, design and design reviews, test planning, security reviews... all of those quality gating activities that waterfall methods support. This is a challenge for secure software development, as there is little guidance available on effectively scaling software security SDLC practices to incremental, agile development methods, something that I will explore more later.
Trying to account for architecture and platform decisions and tooling and training in an upfront “iteration zero” isn’t enough, especially if your project is still going strong after 2 or 3 years. What I worry about (and I worry about a lot of things) is that, moving rapidly from sprint to sprint, the team cannot stop and look at the big picture, to properly re-assess architecture and platform technology decisions made earlier. Instead all the team has a chance to do is make incremental, smaller-scale improvements (tighten up the code here, clean up an interface there, upgrade some of the technology stack), which may leave fundamental questions unanswered, trading off short-term goals (deliver value, minimize the cost and risk of change) with longer-term costs and uncertainties.
One of the other factors that could affect quality in the longer term is the pressure on the team to deliver in a timebox. In Technical Debt: Warning Signs, Catherine Powell raises the concern that developers committing to a date may put schedule ahead of quality:
Finally, I am concerned that rapid cycling does not give the team sufficient opportunities to pause, to take a breath, to properly reset. If they are constantly moving heads down from one iteration to another, do team members really have a chance to reflect, understand and learn? One of the reasons that I maintain this blog is exactly for this: to explore problems and questions that my team and I face; to research, to look far back and far ahead, without having to focus on the goals and priorities of the next sprint.
These concerns, and others, are explored in Traps & Pitfalls of Agile Development - a Non-Contrarian View:
Our team’s challenge is not just to deliver software quickly: like other teams that follow these practices, we’ve proven that we can do that. Our challenge is to deliver value consistently, at an extremely high level of quality and reliability, on a continual and sustainable basis. Each design and implementation decision has to be made carefully: if your customer's business depends on you making changes quickly and perfectly, without impacting their day-to-day operations, how much risk can you afford to take on to make changes today so that the system may be simpler and easier to change tomorrow, especially in today's business environment? It is a high stakes game we're playing. I recognize that this is a problem of debt management, and I'll explore the problems of technical debt and design debt more later.
The practices that we have followed have worked well for us so far. But is there a point where rapid development cycles, even when following good engineering practices, provide diminishing returns? When does the accumulation of design decisions made under time pressure, and conscious decisions to minimize the risk of change, add up to bigger problems? Does developing software in short cycles, with a short decision-making horizon, necessarily result in long-term debt?
There are obvious costs in managing an iteration: startup activities (planning, prioritization, kickoff, securing the team's commitment to the goals of the release), technical overheads like source code and build management (branching and merging and associated controls), status reporting to stakeholders, end-to-end system and integration testing, and closure activities (retrospectives, resetting). We don’t just deliver “ship-quality” software at the end of an iteration: in almost every case we go all the way to releasing the code to production, so our costs also include packaging, change control, release management, security and operations reviews, documentation updates and release notes and training, certifications with partners, data conversion, rollback testing, and pre- and post-implementation operations support. Yep, that’s a lot of work.
All of these costs are balanced against control: our ability to manage and contain risks to the project, to the product, and to the organization. I explored how to manage risks through iterative, incremental development in an earlier post on risk management.
We’ve found that if an iteration is too long (a month or more), it is hard to defend the team from changes to priorities, to prevent new requirements from coming in and disrupting the team’s focus. And in a longer cycle, there are too many changes and fixes that need to be reviewed and tested, increasing the chance of mistakes or oversights or regressions.
Shorter releases are easier to manage because, of course, they are necessarily smaller. We can manage the pressure from the business-side for changes because of the fast delivery cycle (except for emergency hot fixes, we are usually able to convince our product owner, and ourselves, to wait for the next sprint since it is only a couple of weeks away) and it is easier for everyone to get their heads around what was changed in a release and how to verify it. And shorter cycles keep us closer to our customers, not only giving us faster feedback, but demonstrating constant value. I like to think of it as a “value pipeline”, continuously streaming business value to customers.
One of my favorite books on software project management, Johanna Rothman’s Manage It!, recommends making increments smaller to get faster feedback – feedback not just on the product, but on how you build it and how you can improve. The smaller the iteration, the easier to look at it from beginning-to-end and see where time is wasted, what works, what doesn’t, where time is being spent that isn’t expected.
“Shorter timeboxes will make the problems more obvious so you can solve them.”
Ms. Rothman recommends using the “Divide-by-Two Approach to Reduce Iteration Size”: if the iterations aren’t succeeding, divide the length in half, so 6 weeks becomes 3 weeks, and so on. Smaller iterations provide feedback - longer ones mask the problems.
Ms. Rothman also says that it is difficult to establish a rhythm for the team if iterations are too long. In “Selecting the Right Iteration Length for Your Software Development Process”, Mike Cohn of Mountain Goat Software examines the importance of establishing a rhythm in incremental development. He talks about the need for a sense of urgency: if an iteration is too long, it takes too much time for the team to “warm up” and take things seriously. Of course, this needs to be balanced against keeping the team in a constant state of emergency, and burning everyone out.
Some of the other factors that Mr. Cohn finds important in choosing an iteration length:
- how long can you go without introducing change – avoiding requirements churn during an iteration.
- if cycles are too short (for example, a week) small issues, like a key team member coming down with a cold, can throw the team’s rhythm off and impact delivery.
All of this supports our experience: shorter (but not too-short) cycles help establish a rhythm and build the team’s focus and commitment, constantly driving to delivering customer value. And shorter cycles help manage change and risk.
Now we are experimenting with an aggressive, fast-tracked delivery model: a 3-week end-to-end cycle, with software delivered to production every 2 weeks. The team starts work on designing and building the next release while the current release is in integration, packaging and rollout, overlapping development and release activities. Fast-tracking is difficult to manage, and can add risk if not done properly. But it does allow us to respond quickly to changing business demands and priorities, while giving us time for an intensive but efficient testing and release management process.
We'll review how this approach works over the next few months and change it as necessary, but we intend to continue with short increments. However, I am concerned about the longer-term risks, the potential future downsides to our rapid delivery model.
In The Decline and Fall of Agile James Shore argues that rapid cycling short cuts up-front design:
“Up-front design doesn't work when you're using short cycles, and Scrum doesn't provide a replacement. Without continuous, incremental design, Scrum teams quickly dig themselves a gigantic hole of technical debt. Two or three years later, I get a call--or one of my colleagues does. "Changes take too long and cost too much!" I hear. "Teach us about test-driven development, or pairing, or acceptance testing!" By that time, fixing the real problems requires paying back a lot of technical debt, and could take years.”
While Mr. Shore is specifically concerned about loose implementations of Scrum, and its lack of engineering practices compared with other approaches like XP (see also Martin Fowler of ThoughtWorks on the risks of incremental development without strong engineering discipline), the problem is a general one for teams working quickly, in short iterations: even with good engineering discipline, rapid cycling does not leave a lot of time for architecture, design and design reviews, test planning, security reviews... all of those quality gating activities that waterfall methods support. This is a challenge for secure software development, as there is little guidance available on effectively scaling software security SDLC practices to incremental, agile development methods, something that I will explore more later.
Trying to account for architecture and platform decisions and tooling and training in an upfront “iteration zero” isn’t enough, especially if your project is still going strong after 2 or 3 years. What I worry about (and I worry about a lot of things) is that, moving rapidly from sprint to sprint, the team cannot stop and look at the big picture, to properly re-assess architecture and platform technology decisions made earlier. Instead all the team has a chance to do is make incremental, smaller-scale improvements (tighten up the code here, clean up an interface there, upgrade some of the technology stack), which may leave fundamental questions unanswered, trading off short-term goals (deliver value, minimize the cost and risk of change) with longer-term costs and uncertainties.
One of the other factors that could affect quality in the longer term is the pressure on the team to deliver in a timebox. In Technical Debt: Warning Signs, Catherine Powell raises the concern that developers committing to a date may put schedule ahead of quality:
“Once you've committed to a release date and a feature set, it can be hard to change. And to change it because you really want to put a button on one more screen? Not likely. The "we have to ship on X because X is the date" mentality is very common (and rightly so - you can't be late forever because you're chasing perfection). However, to meet that date you're likely to cut corners, especially if you've underestimated how much time the feature really takes, or how much other stuff is going on.”
Finally, I am concerned that rapid cycling does not give the team sufficient opportunities to pause, to take a breath, to properly reset. If they are constantly moving heads down from one iteration to another, do team members really have a chance to reflect, understand and learn? One of the reasons that I maintain this blog is exactly for this: to explore problems and questions that my team and I face; to research, to look far back and far ahead, without having to focus on the goals and priorities of the next sprint.
These concerns, and others, are explored in Traps & Pitfalls of Agile Development - a Non-Contrarian View:
"Agile teams may be prone to rapid accumulation of technical debt. The accrual of technical debt can occur in a variety of ways. In a rush to completion, Iterative development is left out. Pieces get built (Incremental development) but rarely reworked. Design gets left out, possibly as a backlash to BDUF. In a rush to get started building software, sometimes preliminary design work is insufficient. Possibly too much hope is placed in refactoring. Refactoring gets left out. Refactoring is another form of rework that often is ignored in the rush to complete. In summary, the team may move too fast for it's own good."
Our team’s challenge is not just to deliver software quickly: like other teams that follow these practices, we’ve proven that we can do that. Our challenge is to deliver value consistently, at an extremely high level of quality and reliability, on a continual and sustainable basis. Each design and implementation decision has to be made carefully: if your customer's business depends on you making changes quickly and perfectly, without impacting their day-to-day operations, how much risk can you afford to take on to make changes today so that the system may be simpler and easier to change tomorrow, especially in today's business environment? It is a high stakes game we're playing. I recognize that this is a problem of debt management, and I'll explore the problems of technical debt and design debt more later.
The practices that we have followed have worked well for us so far. But is there a point where rapid development cycles, even when following good engineering practices, provide diminishing returns? When does the accumulation of design decisions made under time pressure, and conscious decisions to minimize the risk of change, add up to bigger problems? Does developing software in short cycles, with a short decision-making horizon, necessarily result in long-term debt?
Friday, May 8, 2009
OWASP SAMM Organizational Assessment
I recently completed a lightweight organizational assessment of our software security assurance program, using the OWASP SAMM assessment worksheets. This is the text of a message that I posted to the SAMM mailing list, summarizing my findings on working with the assessment model:
Completing the assessment was straightforward. I didn’t have problems understanding what was being asked or why; although there were a few questions for some Level 3 practices that were targeted to enterprise-level concerns (costing, central control, some of the auditing requirements, metrics) that I did not go into much, working from a small company perspective. Ok, I did have to look up "Likert" to verify what the metric ">3.0 Likert on usefulness of code review activities..." was measuring in CR1.
The assessment guided me in a structured way through a review of our application security assurance program, what we are doing well, where we need to invest more. Going through the assessment was also a good way to improve my understanding of the SAMM model, the practice areas and maturity ladder.
The findings from the assessment were generally in line with what we had already identified through our ad hoc internal analysis and our existing roadmap, at least in the areas of Construction and Verification practices.
I like that SAMM considers more than Construction and Verification, that it also addresses Governance (including training) in a simple and practical way (at least for level 1 and 2), as well as Deployment practices (vulnerability management, environment hardening, operational enablement), so I was able to take into account at least some of the requirements of operations without having to put on a different hat (ITIL, Visible Ops, SANS, and so on.). This should help teams work on the boundaries where software assurance ends and secure infrastructure management and operations begins, rather than throwing the problem over the fence to operations / your customer to take care of.
I also like that Vulnerability Management is considered an important part of the software assurance model. It makes sense to me that a consistent mechanism should be put into place for handling security risks, vulnerabilities and incidents, including incident management and root cause analysis, within the software development / project team structure; aligning this with what is done for secure deployment and operations of the system, and within the larger context of business risk management.
Using the assessment as a planning tool was also simple: building the assessment score card from the worksheets; identifying gaps and areas for development, and then referring to the roadmaps for examples, ideas where to prioritize effort. The sample roadmaps provided good coverage, at least for my purposes (building software for financial services customers). There are some good guidelines on handling outsourced scenarios in the roadmaps. It would be helpful if there were also guidelines on how to apply SAMM to agile development, for teams that follow lighter-weight methods and practices, maybe as part of the ISV roadmap. The case study example was also helpful.
The next challenge is hunting down resources from all of the good stuff available from OWASP to support improvement activities. I am happy to see that work has started to make this easier. I can see that there will be some interesting challenges in this. For example, Threat Assessment: SAMM TA1 recommends attack trees as a less formal alternative to STRIDE, Trike, etc., and provides a brief description of attack trees. Now I can google “attack trees” and find some helpful information out there in the wide world. But there is not much available in OWASP itself on attack trees, except for a note from some poor guy unsuccessfully searching for information on attack trees in CLASP at one time, and a thread in 2007 where Dinis Cruz said that Microsoft gave up on using attack trees in threat modelling because of complexity and difficulty reading them. Whoa pardner, this isn’t what I was hoping to find :-(
One of the keys to me, as a software development guy, not a security expert, is that I need clear and actionable guidance where possible. I have a lot of work to do. I really don’t want to get in the middle of the semantic war around Microsoft’s definition of Threat Modelling vs what other security experts define as Threat Modelling. And then there’s Attack Models and Architectural Risk Analysis (which don’t show up in SAMM, as far as I can see – indexing SAMM would help! - but of course are key to BSIMM).
Mapping resources from OWASP to SAMM practices in a consistent way will make a hard job, building secure software, much easier.
Completing the assessment was straightforward. I didn’t have problems understanding what was being asked or why; although there were a few questions for some Level 3 practices that were targeted to enterprise-level concerns (costing, central control, some of the auditing requirements, metrics) that I did not go into much, working from a small company perspective. Ok, I did have to look up "Likert" to verify what the metric ">3.0 Likert on usefulness of code review activities..." was measuring in CR1.
The assessment guided me in a structured way through a review of our application security assurance program, what we are doing well, where we need to invest more. Going through the assessment was also a good way to improve my understanding of the SAMM model, the practice areas and maturity ladder.
The findings from the assessment were generally in line with what we had already identified through our ad hoc internal analysis and our existing roadmap, at least in the areas of Construction and Verification practices.
I like that SAMM considers more than Construction and Verification, that it also addresses Governance (including training) in a simple and practical way (at least for level 1 and 2), as well as Deployment practices (vulnerability management, environment hardening, operational enablement), so I was able to take into account at least some of the requirements of operations without having to put on a different hat (ITIL, Visible Ops, SANS, and so on.). This should help teams work on the boundaries where software assurance ends and secure infrastructure management and operations begins, rather than throwing the problem over the fence to operations / your customer to take care of.
I also like that Vulnerability Management is considered an important part of the software assurance model. It makes sense to me that a consistent mechanism should be put into place for handling security risks, vulnerabilities and incidents, including incident management and root cause analysis, within the software development / project team structure; aligning this with what is done for secure deployment and operations of the system, and within the larger context of business risk management.
Using the assessment as a planning tool was also simple: building the assessment score card from the worksheets; identifying gaps and areas for development, and then referring to the roadmaps for examples, ideas where to prioritize effort. The sample roadmaps provided good coverage, at least for my purposes (building software for financial services customers). There are some good guidelines on handling outsourced scenarios in the roadmaps. It would be helpful if there were also guidelines on how to apply SAMM to agile development, for teams that follow lighter-weight methods and practices, maybe as part of the ISV roadmap. The case study example was also helpful.
The next challenge is hunting down resources from all of the good stuff available from OWASP to support improvement activities. I am happy to see that work has started to make this easier. I can see that there will be some interesting challenges in this. For example, Threat Assessment: SAMM TA1 recommends attack trees as a less formal alternative to STRIDE, Trike, etc., and provides a brief description of attack trees. Now I can google “attack trees” and find some helpful information out there in the wide world. But there is not much available in OWASP itself on attack trees, except for a note from some poor guy unsuccessfully searching for information on attack trees in CLASP at one time, and a thread in 2007 where Dinis Cruz said that Microsoft gave up on using attack trees in threat modelling because of complexity and difficulty reading them. Whoa pardner, this isn’t what I was hoping to find :-(
One of the keys to me, as a software development guy, not a security expert, is that I need clear and actionable guidance where possible. I have a lot of work to do. I really don’t want to get in the middle of the semantic war around Microsoft’s definition of Threat Modelling vs what other security experts define as Threat Modelling. And then there’s Attack Models and Architectural Risk Analysis (which don’t show up in SAMM, as far as I can see – indexing SAMM would help! - but of course are key to BSIMM).
Mapping resources from OWASP to SAMM practices in a consistent way will make a hard job, building secure software, much easier.
Sunday, May 3, 2009
Everything I needed to know about Maintenance...
So much has been written about software development: there are good books and blogs on software engineering, agile methods, design patterns, requirements requirement, software development lifecycles, testing, project management. But so little has been written about how most of us spend most of our careers in software: essentially maintaining and supporting legacy software, software that has already been built.
I learned most of what I needed to know about successful software maintenance and support a long time ago, when I worked at a small and successful software development company. At Robelle Solutions Technology we developed high-quality technical tools, mostly for other programmers: an IDE (although we didn’t call it an IDE back then, we called it an editor – but it was much more than that), database tools, and an email system, which were used by thousands of customers world wide. I worked there in the early 1990s as a technical support specialist – heck, I was half of the global technical support team at the time. Besides the two of us on support, there was three programmers building and maintaining the products, another developer building and supporting our internal tools, a technical writer (who moonlighted as a science fiction writer), and a small sales and administrative team. The developers worked from home – the rest of us worked out of a horse ranch in rural BC for most of the time I was there. The company succeeded because of strong leadership, talent, a close-knit team built on trust and respect, clear communications, and a focus on doing things right.
Looking back now I understand better what we did that worked so well, and how it can be applied today, with the same successful results.
We worked in small, distributed teams, using lightweight, iterative design and development methods; building and delivering working software to customers every month, responding to customer feedback immediately and constantly improving the design and quality of the product. Large changes were broken down into increments and delivered internally on a monthly basis. All code was peer reviewed. The developers were responsible for running their own set of tests, then code was passed on to the support team for more testing and reviews, and then on to customers for beta testing at the end of the monthly timebox. Back then we called this “Step by Step”.
Incremental design and delivery in timeboxes is perfectly suited to maintenance. Except for emergency patch releases, enhancement requests and bug fixes from the backlog can be prioritized, packaged up and delivered to customers on a regular basis. Timeboxing provides control and establishes a momentum to releases, and customers see continuous value.
We maintained a complete backlog of change requests, bug reports, and our product roadmaps in a beautiful, custom-built issue management system that was indexed for text lookup as well as all of the standard key searching: by customer, product, date, engineer, priority, issue type. Using this we could quickly and easily find information about a customer’s issue, check to see if we had ran into something similar before, if we had a workaround available or a fix scheduled in development. When a new version was ready, we could identify the customers who asked for fixes or changes and see if they were interested in a pre-release for testing. There are issue management systems today which still don’t have the capabilities that what we had back then.
Technical debt was carefully and continuously managed: of course we didn’t know about technical debt back then either. One of the developers’ guiding principles was that any piece of code could be around for years: in fact, some of the same code is still in use today, more than 25 years after the original products were developed! If you had to live with code for a long time, you better be happy with it, and be able to change it quickly and confidently. The programmers were careful in everything that they wrote or changed, and all code was peer reviewed, checking for clarity and consistency, encapsulation, error handling, portability, and performance optimization where that was important. The constraints of delivering within a timebox also focused the development team to come up with the simplest design and implementation possible, knowing that in some cases the solution may have to be thrown away entirely or rewritten in a later increment because it missed the requirement.
The same principles and practices apply today, taking advantage of improvements in engineering methods and technology, including automated unit testing, continuous integration, static analysis tools and refactoring capabilities in the IDE. All of this helps ensure the quality of the code; allows you to make changes with confidence; and helps you avoid falling into the trap of having to rewrite a system because you can no longer understand it or safely change it.
We knew our products in detail: on the support desk, we tested each release and worked with the developers and our technical writer to write and review the product documentation (which was updated in each timebox), and we used all of our own products to run our own business, although not pushing them to the same limit as our customers. Once or twice a year we could take a break from support and work in development for a timebox, getting a chance to do research or help with a part of the product that we felt needed improvement. All of the developers worked on the support desk a few times a year, getting a chance to hear directly from their customers, understand the kind of problems they were facing or what kind of improvements they wanted, and thinking about how to improve the quality of the products, how to make troubleshooting and debugging customer problems easier.
Since we delivered to customers as often as once per month, we had a well-practiced, well-documented release process, with release and distribution checklists, release notes, updated manuals, install instructions that were carefully tested in different environments. All of this was developed iteratively, constantly improved as we found problems or new tools or new ideas. Today teams can take advantage of the ITIL release management practice framework, books like Release It! and Visible Ops to build effective release management processes.
I have learned a lot since working at Robelle, but sometimes I find that I am relearning old lessons; only now truly understanding the importance, the value of these practices, and how can they be applied and improved on today.
I learned most of what I needed to know about successful software maintenance and support a long time ago, when I worked at a small and successful software development company. At Robelle Solutions Technology we developed high-quality technical tools, mostly for other programmers: an IDE (although we didn’t call it an IDE back then, we called it an editor – but it was much more than that), database tools, and an email system, which were used by thousands of customers world wide. I worked there in the early 1990s as a technical support specialist – heck, I was half of the global technical support team at the time. Besides the two of us on support, there was three programmers building and maintaining the products, another developer building and supporting our internal tools, a technical writer (who moonlighted as a science fiction writer), and a small sales and administrative team. The developers worked from home – the rest of us worked out of a horse ranch in rural BC for most of the time I was there. The company succeeded because of strong leadership, talent, a close-knit team built on trust and respect, clear communications, and a focus on doing things right.
Looking back now I understand better what we did that worked so well, and how it can be applied today, with the same successful results.
We worked in small, distributed teams, using lightweight, iterative design and development methods; building and delivering working software to customers every month, responding to customer feedback immediately and constantly improving the design and quality of the product. Large changes were broken down into increments and delivered internally on a monthly basis. All code was peer reviewed. The developers were responsible for running their own set of tests, then code was passed on to the support team for more testing and reviews, and then on to customers for beta testing at the end of the monthly timebox. Back then we called this “Step by Step”.
Incremental design and delivery in timeboxes is perfectly suited to maintenance. Except for emergency patch releases, enhancement requests and bug fixes from the backlog can be prioritized, packaged up and delivered to customers on a regular basis. Timeboxing provides control and establishes a momentum to releases, and customers see continuous value.
We maintained a complete backlog of change requests, bug reports, and our product roadmaps in a beautiful, custom-built issue management system that was indexed for text lookup as well as all of the standard key searching: by customer, product, date, engineer, priority, issue type. Using this we could quickly and easily find information about a customer’s issue, check to see if we had ran into something similar before, if we had a workaround available or a fix scheduled in development. When a new version was ready, we could identify the customers who asked for fixes or changes and see if they were interested in a pre-release for testing. There are issue management systems today which still don’t have the capabilities that what we had back then.
Technical debt was carefully and continuously managed: of course we didn’t know about technical debt back then either. One of the developers’ guiding principles was that any piece of code could be around for years: in fact, some of the same code is still in use today, more than 25 years after the original products were developed! If you had to live with code for a long time, you better be happy with it, and be able to change it quickly and confidently. The programmers were careful in everything that they wrote or changed, and all code was peer reviewed, checking for clarity and consistency, encapsulation, error handling, portability, and performance optimization where that was important. The constraints of delivering within a timebox also focused the development team to come up with the simplest design and implementation possible, knowing that in some cases the solution may have to be thrown away entirely or rewritten in a later increment because it missed the requirement.
The same principles and practices apply today, taking advantage of improvements in engineering methods and technology, including automated unit testing, continuous integration, static analysis tools and refactoring capabilities in the IDE. All of this helps ensure the quality of the code; allows you to make changes with confidence; and helps you avoid falling into the trap of having to rewrite a system because you can no longer understand it or safely change it.
We knew our products in detail: on the support desk, we tested each release and worked with the developers and our technical writer to write and review the product documentation (which was updated in each timebox), and we used all of our own products to run our own business, although not pushing them to the same limit as our customers. Once or twice a year we could take a break from support and work in development for a timebox, getting a chance to do research or help with a part of the product that we felt needed improvement. All of the developers worked on the support desk a few times a year, getting a chance to hear directly from their customers, understand the kind of problems they were facing or what kind of improvements they wanted, and thinking about how to improve the quality of the products, how to make troubleshooting and debugging customer problems easier.
Since we delivered to customers as often as once per month, we had a well-practiced, well-documented release process, with release and distribution checklists, release notes, updated manuals, install instructions that were carefully tested in different environments. All of this was developed iteratively, constantly improved as we found problems or new tools or new ideas. Today teams can take advantage of the ITIL release management practice framework, books like Release It! and Visible Ops to build effective release management processes.
I have learned a lot since working at Robelle, but sometimes I find that I am relearning old lessons; only now truly understanding the importance, the value of these practices, and how can they be applied and improved on today.
Friday, April 17, 2009
OpenSAMM shows a way
We have done a lot of work over the past 3 years to develop an effective software security program. We began working with Cigital’s Touchpoint model in 2006, starting with an internal vulnerability assessment. Touchpoints, described in Software Security: Building Security In by Gary McGraw, outline the required practices for secure software development. But to get real, applied value from the touch point model you have to break your own trail, or get consulting help from Cigital.
We brought consultants in from Secure Software (now part of Fortify) and Cigital’s software security practice to establish a baseline for the system and our organization, including a detailed tool-assisted code review and architecture assessment. With Cigital’s help we built a secure SDLC roadmap for the team, trained everyone in defensive coding, developed secure coding and code review guidelines, and added static analysis tools into our continuous build process.
Working with Cigital definitely kickstarted our security program. Cigital has some smart guys, and they know their stuff. However their approach became more heavyweight over time: their engagement model included assigning a practice manager and a project manager to what were otherwise small engagements, adding to cost and overhead while contributing little value. Their consulting model is clearly targeted towards bigger companies, with bigger consulting budgets and longer time horizons. We needed practical, concrete, immediate feedback; tools and deliverables that we could understand and use right away; and we worked hard with their team to get this.
As part of our security program we also work with another expert consulting firm, Foundstone Professional Services, who we contract for application vulnerability assessments and penetration tests. They have an efficient and well-defined engagement model, they are professional and thorough, and they’re fast. We get high-quality, clear and actionable feedback from their pen testing team – the results of Foundstone’s work not only validate the security of a release, they also provide a health check on the overall security posture of our team.
Based on this work, with the help of Cigital and Foundstone, we have a solid foundation in place and a high-level roadmap for continuous improvement. I am reviewing our next steps, looking to where we can get the most bang for our buck. To help with this I have been looking for a secure SDLC framework suitable for small companies – companies that cannot afford heavyweight process controls (with independent security teams and so on) and large expert consulting budgets, but still need to hold to a high standard of due diligence for secure software development and delivery.
I started working with the CLASP (the Comprehensive, Lightweight Application Security Process) framework from OWASP developed by Pravir Chandra while he was at Secure Software: Pravir then went on to Cigital (he was one of the consultants working with us) and is now independent. CLASP comes with a fair amount of resources, and is intended (as indicated by its name) to be lightweight. However it has not been well maintained since it was contributed, it is not aligned well with other OWASP initiatives; and there is little information on how to apply it and how to scale it. Just as I was growing frustrated at the amount of work it was going to take just to make CLASP useful, Pravir released “CLASP on steroids”, contributing a completely new framework, OpenSAMM to OWASP
OpenSAMM, a Software Assurance Maturity Model, offers a roadmap and well-defined maturity model for secure software development and deployment, with some good tools for self-assessment and planning. So far from my review, OpenSAMM seems more pragmatic than the “Building Security In Maturity Model” BSIMM developed out of the initial collaboration between Pravir, Gary McGraw at Cigital, and sponsored by Fortify Software.
I am working with OpenSAMM to see how it scales to small companies, with small teams using agile development methods. So far it looks promising and actionable. I have completed an internal assessment using the self-assessment tool, and I’m comparing the gap analysis findings to our roadmap plans. I will keep track of my experience with applying OpenSAMM and record my findings as I go along.
We brought consultants in from Secure Software (now part of Fortify) and Cigital’s software security practice to establish a baseline for the system and our organization, including a detailed tool-assisted code review and architecture assessment. With Cigital’s help we built a secure SDLC roadmap for the team, trained everyone in defensive coding, developed secure coding and code review guidelines, and added static analysis tools into our continuous build process.
Working with Cigital definitely kickstarted our security program. Cigital has some smart guys, and they know their stuff. However their approach became more heavyweight over time: their engagement model included assigning a practice manager and a project manager to what were otherwise small engagements, adding to cost and overhead while contributing little value. Their consulting model is clearly targeted towards bigger companies, with bigger consulting budgets and longer time horizons. We needed practical, concrete, immediate feedback; tools and deliverables that we could understand and use right away; and we worked hard with their team to get this.
As part of our security program we also work with another expert consulting firm, Foundstone Professional Services, who we contract for application vulnerability assessments and penetration tests. They have an efficient and well-defined engagement model, they are professional and thorough, and they’re fast. We get high-quality, clear and actionable feedback from their pen testing team – the results of Foundstone’s work not only validate the security of a release, they also provide a health check on the overall security posture of our team.
Based on this work, with the help of Cigital and Foundstone, we have a solid foundation in place and a high-level roadmap for continuous improvement. I am reviewing our next steps, looking to where we can get the most bang for our buck. To help with this I have been looking for a secure SDLC framework suitable for small companies – companies that cannot afford heavyweight process controls (with independent security teams and so on) and large expert consulting budgets, but still need to hold to a high standard of due diligence for secure software development and delivery.
I started working with the CLASP (the Comprehensive, Lightweight Application Security Process) framework from OWASP developed by Pravir Chandra while he was at Secure Software: Pravir then went on to Cigital (he was one of the consultants working with us) and is now independent. CLASP comes with a fair amount of resources, and is intended (as indicated by its name) to be lightweight. However it has not been well maintained since it was contributed, it is not aligned well with other OWASP initiatives; and there is little information on how to apply it and how to scale it. Just as I was growing frustrated at the amount of work it was going to take just to make CLASP useful, Pravir released “CLASP on steroids”, contributing a completely new framework, OpenSAMM to OWASP
OpenSAMM, a Software Assurance Maturity Model, offers a roadmap and well-defined maturity model for secure software development and deployment, with some good tools for self-assessment and planning. So far from my review, OpenSAMM seems more pragmatic than the “Building Security In Maturity Model” BSIMM developed out of the initial collaboration between Pravir, Gary McGraw at Cigital, and sponsored by Fortify Software.
I am working with OpenSAMM to see how it scales to small companies, with small teams using agile development methods. So far it looks promising and actionable. I have completed an internal assessment using the self-assessment tool, and I’m comparing the gap analysis findings to our roadmap plans. I will keep track of my experience with applying OpenSAMM and record my findings as I go along.
Thursday, April 16, 2009
Making Things Happen
I am reading an excellent book on project management called “Making Things Happen” by Scott Berkun, who used to run major projects at Microsoft, and then worked in their engineering excellence group. I quickly zeroed in on Chapter 13, titled “Making Things Happen” which explores what I believe project management is really about – doing whatever it takes to help the team get the job done.
How does a project manager “make things happen”?
First, one of the critical questions that should be asked when hiring a project manager, after you have checked out the candidate’s technical background, is: “If things were not going well on an important project, would I feel confident sending this person into that room, into that debate, and believe that he’d find a way to make it better, whatever the problem was?”. The team has to be convinced that the candidate can make a difference in tough situations.
The project manager’s job is to find out the priorities and manage to them. Make this list of priorities clear to everyone involved – the team must be focused on doing only what is important to success. “What wastes time on projects is confusion about which things should come before which other things.”
Set clear goals, make sure everyone understands them, followup and reinforce priorities. Everyone needs to understand what the “priority 1” list is: the list of things that must be done to succeed. Keep this list as small as possible.
Prevent miscommunications and missteps. Help people take secondary, minor things off of their plates. Resolve conflicts by driving back to the project’s priorities, the critical success factors.
Remove obstacles. Risk management is part of this of course: setup the project to minimize obstacles upfront, watch for things that could go wrong and manage them. Handle people problems. Fix the environment – make sure people can get their work done.
Be relentless. Don’t give up, don’t stop looking for alternatives. Berkun talks about the example of Apollo 13, where the team kept driving to fix unfixable problems and save the mission.
Question people (even powerful ones) and challenge assumptions. Believe that there is a solution to a problem – even if it means changing the definition of the problem. If you can’t find an answer that means that you haven’t looked hard enough.
Own the problem. Escalate, use your network, create options and alternatives. Be dead serious and fight to the end - there is always a way out.
All of this might sound over-done, over-dramatic, but I believe that this is what sets successful project managers apart – the sense of ownership, the ability, the discipline, the drive to execute.
How does a project manager “make things happen”?
First, one of the critical questions that should be asked when hiring a project manager, after you have checked out the candidate’s technical background, is: “If things were not going well on an important project, would I feel confident sending this person into that room, into that debate, and believe that he’d find a way to make it better, whatever the problem was?”. The team has to be convinced that the candidate can make a difference in tough situations.
The project manager’s job is to find out the priorities and manage to them. Make this list of priorities clear to everyone involved – the team must be focused on doing only what is important to success. “What wastes time on projects is confusion about which things should come before which other things.”
Set clear goals, make sure everyone understands them, followup and reinforce priorities. Everyone needs to understand what the “priority 1” list is: the list of things that must be done to succeed. Keep this list as small as possible.
Prevent miscommunications and missteps. Help people take secondary, minor things off of their plates. Resolve conflicts by driving back to the project’s priorities, the critical success factors.
Remove obstacles. Risk management is part of this of course: setup the project to minimize obstacles upfront, watch for things that could go wrong and manage them. Handle people problems. Fix the environment – make sure people can get their work done.
Be relentless. Don’t give up, don’t stop looking for alternatives. Berkun talks about the example of Apollo 13, where the team kept driving to fix unfixable problems and save the mission.
Question people (even powerful ones) and challenge assumptions. Believe that there is a solution to a problem – even if it means changing the definition of the problem. If you can’t find an answer that means that you haven’t looked hard enough.
Own the problem. Escalate, use your network, create options and alternatives. Be dead serious and fight to the end - there is always a way out.
All of this might sound over-done, over-dramatic, but I believe that this is what sets successful project managers apart – the sense of ownership, the ability, the discipline, the drive to execute.
Monday, March 23, 2009
What's Wrong with Sucking Less?
At the Agile 2008 conference in Toronto, David Douglas & Robin Dymond discussed their concerns that the majority of companies who adopt agile (and by “agile”, effectively meaning Scrum) practices were falling short of complete adoption. The companies that they were working with were satisfied with 1.5-2x performance improvements in quality and time-to-market gained by effectively cherry picking from the key agile, incremental software development practices. The authors were concerned that most, if not all companies adopting agile were content simply “to suck less” rather than transforming their businesses.
This was further explored in December of last year in a StickyMinds column “Little Scrum Pigs and the Big Bad Wolf” by Michelle Sliger, who expressed her concern that
“Indeed, many companies are refusing to view agile as anything other than a set of engineering practices.”
That surprised me. I thought that this was, in fact, the point of agile software development: for people to adopt more effective software engineering practices.
What is especially confusing is that Scrum, in particular, is really a project management approach and provides very little in the way of software engineering practices. It is intuitive and obvious and easy to implement, maybe too easy, which is why most agile projects today are based on Scrum: its strength, and weakness, is that it provides an effective framework for organizing and managing software projects by breaking them down into time boxed increments, but does not force the team to adopt specific engineering practices and disciplines, unlike other methods, and especially XP.
Martin Fowler of ThoughtWorks, one of the leading thinkers in the agile (in this case, XP, however, rather than Scrum) community, raises the concern that this lack of software engineering discipline can lead to Scrum teams building a lot of sloppy software, however quickly.
Back to the “Three Little Pigs” – the author goes on to express her dismay that
“They have not adopted the value system that is the underlying infrastructure of all agile approaches.”
and that companies who are simply interested in adopting good practices from scrum and XP, but who don’t buy into the complete philosophy and value set, therefore lack vision and lack commitment; and are at risk of failure.
I find the argument to be both elitist and dogmatic, and awfully awfully unclear. The author suggests that there is something hidden in the agile manifesto, that by surrendering to this mystery you will find the one true agile path to success in software development and anything less is conceding defeat, or at the least condemning yourself and your organization to mediocrity.
But what is this mystery, this ineffable something or somethings that organizations refuse to, or are somehow unable to, accept? Perhaps it is delivering working software incrementally, following a timeboxed approach – no, this can’t be it, this is a well understood engineering practice, one of those mere disciplines that the author suggests is insufficient for success. Maybe it is “continuous attention to technical excellence and good design”. That can’t be it: I don’t see why any organization would not accept this as axiomatic when building software. Or is it the emphasis on simplicity? Or that the team should have a good working environment and the trust and support of management? Or that developers and customers should work together? Or maybe that we need to create self-organizing teams? Just what is it that these companies, who are “teetering at the edge between mediocrity and high performance”, are failing to do?
Douglas and Dymond concede that there are too few real agile success stories: they point to Nokia and BMC Software and PatientKeeper of course; a small number of companies (very small, after 10 or so years of evangelism) who have had noted success in adopting Scrum in a fundamental way. But I would argue that there are a lot of success stories – all of those companies who are “sucking less”, who have started on a path to building better and better software: every day, working hard to suck less and still less and less and so on.
While there is a negative connotation to the term, I don't see what is wrong with "sucking less". With being practical, goal-focused, and incremental in improving software development practices. With delivering good, working software in time boxed, iterative releases. With building a stronger development team, and a better development environment. With following good engineering practices and management methods, and stopping bad ones. With constantly reviewing your failures and successes and finding new ways to improve. I don’t see this as building a “house of straw” – I see this as what we all have to do to succeed: constantly, ruthlessly get better and better together.
This was further explored in December of last year in a StickyMinds column “Little Scrum Pigs and the Big Bad Wolf” by Michelle Sliger, who expressed her concern that
“Indeed, many companies are refusing to view agile as anything other than a set of engineering practices.”
That surprised me. I thought that this was, in fact, the point of agile software development: for people to adopt more effective software engineering practices.
What is especially confusing is that Scrum, in particular, is really a project management approach and provides very little in the way of software engineering practices. It is intuitive and obvious and easy to implement, maybe too easy, which is why most agile projects today are based on Scrum: its strength, and weakness, is that it provides an effective framework for organizing and managing software projects by breaking them down into time boxed increments, but does not force the team to adopt specific engineering practices and disciplines, unlike other methods, and especially XP.
Martin Fowler of ThoughtWorks, one of the leading thinkers in the agile (in this case, XP, however, rather than Scrum) community, raises the concern that this lack of software engineering discipline can lead to Scrum teams building a lot of sloppy software, however quickly.
Back to the “Three Little Pigs” – the author goes on to express her dismay that
“They have not adopted the value system that is the underlying infrastructure of all agile approaches.”
and that companies who are simply interested in adopting good practices from scrum and XP, but who don’t buy into the complete philosophy and value set, therefore lack vision and lack commitment; and are at risk of failure.
I find the argument to be both elitist and dogmatic, and awfully awfully unclear. The author suggests that there is something hidden in the agile manifesto, that by surrendering to this mystery you will find the one true agile path to success in software development and anything less is conceding defeat, or at the least condemning yourself and your organization to mediocrity.
But what is this mystery, this ineffable something or somethings that organizations refuse to, or are somehow unable to, accept? Perhaps it is delivering working software incrementally, following a timeboxed approach – no, this can’t be it, this is a well understood engineering practice, one of those mere disciplines that the author suggests is insufficient for success. Maybe it is “continuous attention to technical excellence and good design”. That can’t be it: I don’t see why any organization would not accept this as axiomatic when building software. Or is it the emphasis on simplicity? Or that the team should have a good working environment and the trust and support of management? Or that developers and customers should work together? Or maybe that we need to create self-organizing teams? Just what is it that these companies, who are “teetering at the edge between mediocrity and high performance”, are failing to do?
Douglas and Dymond concede that there are too few real agile success stories: they point to Nokia and BMC Software and PatientKeeper of course; a small number of companies (very small, after 10 or so years of evangelism) who have had noted success in adopting Scrum in a fundamental way. But I would argue that there are a lot of success stories – all of those companies who are “sucking less”, who have started on a path to building better and better software: every day, working hard to suck less and still less and less and so on.
While there is a negative connotation to the term, I don't see what is wrong with "sucking less". With being practical, goal-focused, and incremental in improving software development practices. With delivering good, working software in time boxed, iterative releases. With building a stronger development team, and a better development environment. With following good engineering practices and management methods, and stopping bad ones. With constantly reviewing your failures and successes and finding new ways to improve. I don’t see this as building a “house of straw” – I see this as what we all have to do to succeed: constantly, ruthlessly get better and better together.
Subscribe to:
Posts (Atom)
