Last week I attended the OWASP AppSec USA conference in Minneapolis. It was my first time at an OWASP event, and it was an impressive show. More than 600 attendees, some high-quality speakers, lots of vendor representation.
There were several different tracks over the two days of the conference: attacks and defenses, security issues in Mobile computing and Cloud computing, thought leadership, OWASP tools, security patterns and secure SDLCs. And there were lots of opportunities to talk to vendors and smart people. Here were the hilights for me:
Jim Manico showed that there’s hope for developers to write secure code, at least when it comes to protecting web apps from XSS attacks, if they use common good sense and new frameworks and libraries like Google’s Caja, the technology behind OWASP’s Java HTML Sanitizer.
There is even some hope for writing secure Mobile apps, as long as developers are careful and informed. This is a more reassuring story than what I was hearing from security experts earlier in the year, when everyone was focused on how pathetically insecure the platforms and tools were for writing Mobile apps.
It's also possible to be secure in the Cloud as long as you take the time to understand the Cloud services that you are using and how they are implemented and what assumptions they make, and if you think through your design. Which a lot of people probably aren’t bothering to do.
There's too much code already out there
We’re still trying to find ways to build software that is secure. But a bigger problem for many organizations is that there is already too much code out there that is insecure. Jeff Williams at Aspect Security guesstimates that there are more than 1 trillion lines of code in the world, and around 15 million developers around the world writing more of it. A lot of this legacy code contains serious security problems – but it is too expensive to find these and fix these problems. We need better tools to do this, tools that scale up to enterprises and down to small companies.
Many organizations, especially big enterprises with thousands of applications, are relying heavily on scanning tools to find security vulnerabilities. But it takes too long and costs too much to work through all of the results and to find real risks from the long lists of findings and false positives – to figure out what problems need to be fixed and how to fix them.
HP thinks that they have an answer by correlating dynamic scanning and static code analysis results. The idea is to scan the source code, then run dynamic scans against an application under a run-time monitor which captures problems found by the dynamic scanner and records where the problems occurred in the code. By matching up these results, you can see which static analysis findings are exploitable, and it makes it easier to find and fix problems found in run-time testing. And combining these techniques also makes it possible to find new types of vulnerabilities that can’t be found using just one or other of the approaches. I heard about this idea earlier in the year when Brian Chess presented it at SANS Appsec, but it is interesting to see how far it has come since then. Hybrid testing is limited to HP technology for now (SPI Dynamics and Fortify) and it’s limited by what problems the scanners can find, but it looks like a good step forward in the test tool space.
Good Pen Testing and Bad Pen Testing
It’s clear talking to vendors that there are good pen tests and bad pen tests. In most cases, good pen tests are what they offer, and bad pen tests are what you would get from a competitor. Comparing one vendor to another isn't easy, since an application pen test means different things to different people, and the vendors all had different stories and offerings. Some stressed the quality of their tools, others stressed the experience and ingenuity of their testers. Some offered automated scans with some interpretation, which could work fine for plain-vanilla web apps. Some do combined source code review and scanning. Some don’t want help understanding the code or scoping an engagement – they have a fixed-price menu running from running scans to manual pen tests using scanners and fuzzers and attack proxies. Others expect to spend a lot of time scoping and understanding what to test, to make sure that they focus on what's important.
SSL is broken
Moxie Marlinspike’s keynote on the problems with SSL and the CA model was entertaining but scary. He showed that the circle of trust model that underlies SSL – I trust the host because my browser trusts the CA and the CA trusts the host – is fundamentally broken now that CAs like Comodo and DigiNotar (and maybe others) have been compromised.
His solution is Convergence, an alternative way to establish trust between parties, where the client is responsible for choosing trust providers (notaries) rather than being locked into a single trust provider selected by the host.
Metrics and numbers for fun
I like numbers and statistics, here are a few that I found especially interesting. Some of them were thrown out quickly, I hope that I got them right.
At a panel on secure development, one expert estimated that the tax for introducing a secure SDLC, including training people, getting tools and getting people to use them, was about 15% of the total development costs. Although he qualified this number by saying that it was from a pilot, another panelist agreed that it was reasonable, so it seems a good rule of thumb to use for now.
John Steven at Cigital says they have found that an inexperienced team can expect to have 38-40 security vulnerabilities per KLOC. A mature team, following good security hygiene, can get this down to 7-10 vulnerabilities per KLOC.
Chris Wysopal at Veracode presented some interesting numbers in his presentation on Application Security Debt – you can read more on his ideas on applying the technical debt metaphor to application security and building economic models for the costs of this debt on his blog. He referenced findings from the 2010 Verizon Breach study and tried to correlate these findings with analysis work that Veracode has done using data from testing customer applications. This showed that the most common application-related breach was through Backdoors or hidden Command Channels which I was surprised at, and so was he, since Veracode’s own analysis shows that application Backdoors are not that common. I assumed from this analysis that application Backdoors were so dangerous because they were so easily found and exploited by attackers. However, when I read the Verizon report again, what Verizon calls a Backdoor is a command channel opened up by malware, which is something quite different to a Backdoor left in an application, so the comparison doesn’t stand up. This leaves SQL Injection to be the number one application vulnerability, something that shouldn't surprise anyone.
The “A Word”
Most of the speakers and panelists at the conference were from enterprise companies or from vendors trying to sell to the enterprise. Not surprisingly, as a result there was little said about the needs and constraints of small companies, and at the sessions I attended, the speakers had not come to terms with the reality that most companies are adopting or have already adopted more incremental agile development practices. At one panel, Agile development was referred to as the “A Word”.
We urgently need tools and practices that small, fast-moving teams can use, because these teams are building and maintaining a lot of the world's software. It's time for OWASP and ISC2 and the rest of the application security community to accept this and understand what it means and offer help where it's really needed.
Dumbest thing said at the conference
This award goes to one vendor on a panel on security testing who said that it is irresponsible for developers to work with new technologies where they don't understand the security risks, that developers who work on the bleeding edge are putting their customers and businesses at unnecessary risk. Besides being blindingly obvious, it shows a complete lack of connection to the realities of software development and the businesses that rely on software (which is pretty much every business today). Dinis Cruz pointed out the absurdity of this statement, by admitting that we don’t fully understand the security risks of the common platforms that we use today like Java and .NET. This means that nobody should be writing software at all - or maybe it would be ok if we all went back to writing batch programs in COBOL and FORTRAN.
Impressed and Humbled
Overall I learned some good stuff, and I enjoyed the conference and the opportunity to meet some of the people behind OWASP. I was impressed and humbled by the commitment and accomplishments of the many OWASP contributors. Seeing how much they have done, talking to them, understanding how much they care, gives me some hope that we can all find a way to build better software together.
Monday, September 26, 2011
Wednesday, September 7, 2011
Standups – take ‘em or leave ‘em
We left ‘em.
Standup meetings are a core practice in Agile methods like Scrum and XP. Each day the team meets briefly to answer 3 questions: What did I get done yesterday? What am I going to do today? What is getting in my way?
Standups offer a quick check on what’s happening, what’s changed, who ‘s working on what, who needs help. The meeting is supposed to be short and sweet, no more than 15 minutes a day. Martin Fowler lists some good reasons to hold standups:
- Share commitment
- Communicate status
- Identify obstacles to be solved
- Set direction and focus
- Help to build a team
However, not everyone finds that standups are necessary and some people have started to question the value of standups over time and are looking for substitutes. A number of people have suffered through poorly-run standup meetings. And some people just plain don’t like them.
Our team has been successful following Scrum and XP practices as we transitioned from delivering phased releases (a release every 2-3 months) to 2-3 small releases a month. We looked closely at how other teams worked, spent time learning about incremental and iterative development methods, and took what made sense to us and tried it and adapted it. But one of the Agile practices that we did not take up was standups.
When we introduced the idea of standups to the team a few years ago, we were surprised by the response. The team was roughly split. Some people, especially people who were new to the team, or people who had worked on successful Scrum teams or XP teams before, liked the idea. But other people who had been on the team from the beginning were opposed, some of them strongly opposed. And they had some good reasons:
We were already running an operational business and building and delivering new software at the same time. Team members were helping to support the system, helping with day-to-day operations, and working closely with the business side. Some people were working late and weekends to get changes in or to do after-hours testing, others were working early to help with startup issues or to coordinate with partners, some people were working in different timezones, some people were working at home, others were at the beck and call of integration partners and important clients. It would not be possible to schedule a daily meeting, even a short one, which would work well for everyone on a sustained basis. This is one example (and there many others), where Agile development ideas which work well in the controlled, artificial reality of a development project need to be rethought and adapted to fit day-to-day operational demands and constraints.
Some of the team had their hands full with important changes or problems that had to be taken care of urgently. They met regularly with other team members to work through design issues and reviewed each other’s code. They got help from other team members or managers when they needed it – there was no need to wait for a daily meeting to bring up blockers or get whatever information they needed.
And there were a couple of introverts who hated all meetings and didn’t understand why a meeting that they had to go to everyday and standup at would be any better than any other meeting. They saw it as a half hour wasted out of every day – because they knew that a 15-minute meeting, when you include the time taken to save what you were working on, go to the meeting, standup, get out of the meeting and get back to working productively, would mean at least 30 minutes if not more time every day from what they were doing.
Rather than try to fight with the team to implement a practice that we weren’t sure was really needed, we decided to find other ways to get the same results.
Alternatives to standups
There are other ways to share status within (and outside of) the team. Because we have people working together in different countries and different groups, we rely heavily on our bug tracking system to manage the backlog of work, to schedule and plan work, and to radiate status information. The bug tracking system is used by and shared with everyone: developers, testers, support, business operations, systems engineering, IT, compliance and management. Everything to do with the software and systems operations is tracked here, including new features and bug fixes and security vulnerabilities and problems found in testing, and operational and infrastructural changes, compliance reviews and new clients being onboarded. Using the system’s dashboards and notification feeds everyone can easily tell what is happening and what will be happening and when. The team doesn’t need standups to know what is happening, who is working on what, what the schedule is or what needs to be worked on next.
Managers who need to know what’s going on and who want to make sure that everyone is on track can get all of this through MBWA. And regular one-on-one meetings help reinforce priorities and make sure that everyone gets face time and a chance to ask questions without wasting the rest of the team’s time.
Standups help at the start
I can see that standups are more useful in the early stages of a project, when the team is starting to come together and everyone is working through the design and learning the domain and the technology and feeling out each other, finding out each other’s strengths and quirks.
Without standups, it is harder for new people joining the team to understand what is going on and what’s important, and figure out where they fit in. Pairing up new team members with someone more experienced helps, but if you have a couple or more people joining a team, it makes sense to hold standups for a while at least to help them get up to speed.
Declining value over time
But when people have been working together for a while, standups offer declining value. Once the team has gelled and people know each other and can depend on each other, they don’t need to meet in a room every morning to talk about what they are working on. Like some other practices that are important and useful in the beginning, the team grows up or grows out of them.
And as you move more into maintenance and support work, when people on the team are working on smaller individual changes and fixes and the work tends to be more interrupt-driven, standups are a waste. What one person is working on doesn’t have much if anything to do with what somebody else is doing. They don’t need to, or want to, listen to each other talking about what they did and what they’re going to do, because if it was important they would already know about it anyways.
Stuffed Pigs, Nerf Balls and other Silly Games
One of the things I don’t like about standups is that they are fundamentally paternalistic – you’re treating the team like children, forcing them to get together and stand up in a room every day (they have to stand up because they can’t be trusted to sit down) and making them speak in turn for only a few minutes (but only of course if they are a pig, not a chicken). And on some teams, people go so far as to hand around a stuffed pig or a Nerf ball or follow some other awkward ritual to make sure that people don’t speak out of turn. If this sounds silly and childish, it is. You have a room full of children: team members who can’t ask questions, they can’t solve problems, they have to stand there and follow silly rituals. Day after day, week after week, month after month, year after year…
Rather than treating adults like little children, why not let people meet when they need to or want to – to solve problems, to review requirements or designs, to respond to incidents (Root Cause Analysis), to plan, and to make sure that people know when requirements or priorities are changing in a significant way.
Would we have had different results if we had implemented standups? Probably. Would the results have been better for the team and for the company? I’m not sure. We found other good ways for the team, and the rest of the company, to track what was going on and what needed to be done next. We’ve had some misunderstandings, and some people did get off track – standups might have helped prevent this. And standups would have helped new people joining the team, to come up to speed. But the rest of the team gelled quickly during our crazy startup days, and built up a high level of trust and openness and shared commitment without standups. Can you build high-performing, collaborative teams without standups? Of course you can.
Standups have a place, especially early on with people who don’t know each other, where everyone is learning and uncertain about what to do and about each other. But don’t do something, or keep doing something if you don’t think that you need to, just because somebody read it in a book or learned about it in a 3 day SCM class. It’s your team, and you can find your own way to succeed.
Standup meetings are a core practice in Agile methods like Scrum and XP. Each day the team meets briefly to answer 3 questions: What did I get done yesterday? What am I going to do today? What is getting in my way?
Standups offer a quick check on what’s happening, what’s changed, who ‘s working on what, who needs help. The meeting is supposed to be short and sweet, no more than 15 minutes a day. Martin Fowler lists some good reasons to hold standups:
- Share commitment
- Communicate status
- Identify obstacles to be solved
- Set direction and focus
- Help to build a team
However, not everyone finds that standups are necessary and some people have started to question the value of standups over time and are looking for substitutes. A number of people have suffered through poorly-run standup meetings. And some people just plain don’t like them.
Our team has been successful following Scrum and XP practices as we transitioned from delivering phased releases (a release every 2-3 months) to 2-3 small releases a month. We looked closely at how other teams worked, spent time learning about incremental and iterative development methods, and took what made sense to us and tried it and adapted it. But one of the Agile practices that we did not take up was standups.
When we introduced the idea of standups to the team a few years ago, we were surprised by the response. The team was roughly split. Some people, especially people who were new to the team, or people who had worked on successful Scrum teams or XP teams before, liked the idea. But other people who had been on the team from the beginning were opposed, some of them strongly opposed. And they had some good reasons:
We were already running an operational business and building and delivering new software at the same time. Team members were helping to support the system, helping with day-to-day operations, and working closely with the business side. Some people were working late and weekends to get changes in or to do after-hours testing, others were working early to help with startup issues or to coordinate with partners, some people were working in different timezones, some people were working at home, others were at the beck and call of integration partners and important clients. It would not be possible to schedule a daily meeting, even a short one, which would work well for everyone on a sustained basis. This is one example (and there many others), where Agile development ideas which work well in the controlled, artificial reality of a development project need to be rethought and adapted to fit day-to-day operational demands and constraints.
Some of the team had their hands full with important changes or problems that had to be taken care of urgently. They met regularly with other team members to work through design issues and reviewed each other’s code. They got help from other team members or managers when they needed it – there was no need to wait for a daily meeting to bring up blockers or get whatever information they needed.
And there were a couple of introverts who hated all meetings and didn’t understand why a meeting that they had to go to everyday and standup at would be any better than any other meeting. They saw it as a half hour wasted out of every day – because they knew that a 15-minute meeting, when you include the time taken to save what you were working on, go to the meeting, standup, get out of the meeting and get back to working productively, would mean at least 30 minutes if not more time every day from what they were doing.
Rather than try to fight with the team to implement a practice that we weren’t sure was really needed, we decided to find other ways to get the same results.
Alternatives to standups
There are other ways to share status within (and outside of) the team. Because we have people working together in different countries and different groups, we rely heavily on our bug tracking system to manage the backlog of work, to schedule and plan work, and to radiate status information. The bug tracking system is used by and shared with everyone: developers, testers, support, business operations, systems engineering, IT, compliance and management. Everything to do with the software and systems operations is tracked here, including new features and bug fixes and security vulnerabilities and problems found in testing, and operational and infrastructural changes, compliance reviews and new clients being onboarded. Using the system’s dashboards and notification feeds everyone can easily tell what is happening and what will be happening and when. The team doesn’t need standups to know what is happening, who is working on what, what the schedule is or what needs to be worked on next.
Managers who need to know what’s going on and who want to make sure that everyone is on track can get all of this through MBWA. And regular one-on-one meetings help reinforce priorities and make sure that everyone gets face time and a chance to ask questions without wasting the rest of the team’s time.
Standups help at the start
I can see that standups are more useful in the early stages of a project, when the team is starting to come together and everyone is working through the design and learning the domain and the technology and feeling out each other, finding out each other’s strengths and quirks.
Without standups, it is harder for new people joining the team to understand what is going on and what’s important, and figure out where they fit in. Pairing up new team members with someone more experienced helps, but if you have a couple or more people joining a team, it makes sense to hold standups for a while at least to help them get up to speed.
Declining value over time
But when people have been working together for a while, standups offer declining value. Once the team has gelled and people know each other and can depend on each other, they don’t need to meet in a room every morning to talk about what they are working on. Like some other practices that are important and useful in the beginning, the team grows up or grows out of them.
And as you move more into maintenance and support work, when people on the team are working on smaller individual changes and fixes and the work tends to be more interrupt-driven, standups are a waste. What one person is working on doesn’t have much if anything to do with what somebody else is doing. They don’t need to, or want to, listen to each other talking about what they did and what they’re going to do, because if it was important they would already know about it anyways.
Stuffed Pigs, Nerf Balls and other Silly Games
One of the things I don’t like about standups is that they are fundamentally paternalistic – you’re treating the team like children, forcing them to get together and stand up in a room every day (they have to stand up because they can’t be trusted to sit down) and making them speak in turn for only a few minutes (but only of course if they are a pig, not a chicken). And on some teams, people go so far as to hand around a stuffed pig or a Nerf ball or follow some other awkward ritual to make sure that people don’t speak out of turn. If this sounds silly and childish, it is. You have a room full of children: team members who can’t ask questions, they can’t solve problems, they have to stand there and follow silly rituals. Day after day, week after week, month after month, year after year…
Rather than treating adults like little children, why not let people meet when they need to or want to – to solve problems, to review requirements or designs, to respond to incidents (Root Cause Analysis), to plan, and to make sure that people know when requirements or priorities are changing in a significant way.
Would we have had different results if we had implemented standups? Probably. Would the results have been better for the team and for the company? I’m not sure. We found other good ways for the team, and the rest of the company, to track what was going on and what needed to be done next. We’ve had some misunderstandings, and some people did get off track – standups might have helped prevent this. And standups would have helped new people joining the team, to come up to speed. But the rest of the team gelled quickly during our crazy startup days, and built up a high level of trust and openness and shared commitment without standups. Can you build high-performing, collaborative teams without standups? Of course you can.
Standups have a place, especially early on with people who don’t know each other, where everyone is learning and uncertain about what to do and about each other. But don’t do something, or keep doing something if you don’t think that you need to, just because somebody read it in a book or learned about it in a 3 day SCM class. It’s your team, and you can find your own way to succeed.
Subscribe to:
Posts (Atom)
