Showing posts with label leadership. Show all posts
Showing posts with label leadership. Show all posts

Tuesday, November 15, 2011

Diminishing Returns in software development and maintenance

Everyone knows from reading The Mythical Man Month that as you add more people to a software development project you will see diminishing marginal returns.

When you add a person to a team, there’s a short-term hit as the rest of the team slows down to bring the new team member up to speed and adjusts to working with another person, making sure that they fit in and can contribute. There’s also a long-term cost. More people means more people who need to talk to each other (n x n-1 / 2), which means more opportunities for misunderstandings and mistakes and misdirections and missed handoffs, more chances for disagreements and conflicts, more bottleneck points.

As you continue to add people, the team needs to spend more time getting each new person up to speed and more time keeping everyone on the team in synch. Adding more people means that the team speeds up less and less, while people costs and communications costs and overhead costs keep going up. At some point negative returns set in – if you add more people, the team’s performance will decline and you will get less work done, not more.

Diminishing Returns from any One Practice

But adding too many people to a project isn’t the only case of diminishing returns in software development. If you work on a big enough project, or if you work in maintenance for long enough, you will run into problems of diminishing returns everywhere that you look.

Pushing too hard in one direction, depending too much on any tool or practice, will eventually yield diminishing returns. This applies to:
- Manual functional and acceptance testing
- Test automation
- Any single testing technique
- Code reviews
- Static analysis bug finding tools
- Penetration tests and other security reviews

Aiming for 100% code coverage on unit tests is a good example. Building a good automated regression safety net is important – as you wire in tests for key areas of the system, programmers get more confidence and can make more changes faster.

How many tests are enough? In Continuous Delivery, Jez Humble and David Farley set 80% coverage as a target for each of automated unit testing, functional testing and acceptance testing. You could get by with lower coverage in many areas, higher coverage in core areas. You need enough tests to catch common and important mistakes. But beyond this point, more tests get more difficult to write, and find fewer problems.

Unit testing can only find so many problems in the first place. In Code Complete, Steve McConnell explains that unit testing can only find between 15% and 50% (on average 30%) of the defects in your code. Rather than writing more unit tests, people’s time would be better spent on other approaches like exploratory system testing and code reviews or stress testing or fuzzing to find different kinds of errors.
Too much of anything is bad, but too much whiskey is enough.
Mark Twain, as quoted in Code Complete
Refactoring is important for maintaining and improving the structure and readability of code over time. It is intended to be a supporting practice – to help make changes and fixes simpler and clearer and safer. When refactoring becomes an end in itself or turns into Obsessive Refactoring Disorder, it not only adds unnecessary costs as programmers waste time over trivial details and style issues, it can also add unnecessary risks and create conflict in a team.

Make sure that refactoring is done in a disciplined way, and focus refactoring on those areas that need it the most: on code that is frequently changed, routines that are too big, too hard to read, too complex and error-prone. Putting most of your attention refactoring (or if necessary rewriting) this code will get you the highest returns.

Less and Less over Time

Diminishing returns also set in over time. The longer that you spend working the same way and with the same tools, the less benefits you will see. Even core practices that you’ve grown to depend on don’t pay back over time, and at some point may cost more than they are worth.

It’s time again for New Year’s resolutions – time to sign up at a gym and start lifting weights. If you stick with the same routine for a couple of months, you will start to see good results. But after a while your body will get used to the work – if you keep doing the same things the same way your performance will plateau and you will stop seeing gains. You will get bored and stop going to the gym, which will leave more room for people like me. If you do keep going, trying to push harder for returns, you will overtrain and injure yourself.

The same thing happens to software teams following the same practices, using the same tools. Some of this is due to inertia. Teams, organizations reach an equilibrium point and they want to stay there. Because it is comfortable, and it works – or at least they understand it. And because the better the team is working, the harder it is to get better – all the low-hanging fruit has been picked. People keep doing what worked for them in the past. They stop looking beyond their established routines, stop looking for new ideas. Competence and control lead to complacency and acceptance. Instead of trying to be as good as possible, they settle for being good enough.

This is the point of inspect-and-adapt in Scrum and other time boxed methods – asking the team to regularly re-evaluate what they are doing and how they are doing it, what’s going well and what isn’t, what they should do more of or less of, challenging the status quo and finding new ways to move forward. But even the act of assessing and improving is subject to diminishing returns. If you are building software in 2-week time boxes, and you’ve been doing this for 3, 4 or 5 years, then how much meaningful feedback should you really expect from so many superficial reviews? After a while the team finds themselves going over the same issues and problems and coming up with the same results. Reviews become an unnecessary and empty ritual, another waste of time.

The same thing happens with tools. When you first start using a static analysis bug checking tool for example, there’s a good chance that you will find some interesting problems that you didn’t know were in the code – maybe even more problems than you can deal with. But once you triage this and fix up the code and use the tool for a while, the tool will find fewer and fewer problems until it gets to the point where you are paying for insurance – it isn’t finding problems any more, but it might someday.

In "Has secure software development reached its limits?” William Jackson argues that SDLCs – all of them – eventually reach a point of diminishing returns from a quality and security standpoint, and that Microsoft and Oracle and other big shops are already seeing diminishing returns from their SDLCs. Their software won’t get any better – all they can do is to keep spending time and money to stay where they are. The same thing happens with Agile methods like Scrum or XP – at some point you’ve squeezed everything that you can from this way or working, and the team’s performance will plateau.

What can you do about diminishing returns?

First, understand and expect returns to diminish over time. Watch for the signs, and factor this into your expectations – that even if you maintain discipline and keep spending on tools, you will get less and less return for your time and money. Watch for the team’s velocity to plateau or decline.

Expect this to happen and be prepared to make changes, even force fundamental changes on the team. If the tools that you are using aren’t giving returns any more, then find new ones, or stop using them and see what happens.

Keep reviewing how the team is working, but do these reviews differently: review less often, make the reviews more focused on specific problems, involve different people from inside and outside of the team. Use problems or mistakes as an opportunity to shake things up and challenge the status quo. Dig deep using Root Cause Analysis and challenge the team’s way of thinking and working, look for something better. Don’t settle for simple answers or incremental improvements.

Remember the 80/20 rule. Most of your problems will happen in the same small number of areas, from a small number of common causes. And most of your gains will come from a few initiatives.

Change the team’s driving focus and key metrics, set new bars. Use Lean methods and Lean Thinking to identify and eliminate bottlenecks, delays and inefficiencies. Look at the controls and tests and checks that you have added over time, question whether you still need them, or find steps and checks that can be combined or automated or simplified. Focus on reducing cycle time and eliminating waste until you have squeezed out what you can. Then change your focus to quality and eliminating bugs, or to simplifying the release and deployment pipeline, or some other new focus that will push the team to improve in a meaningful way. And keep doing this and pushing until you see the team slowing down and results declining. Then start again, and push the team to improve again along another dimension. Keep watching, keep changing, keep moving ahead.

Tuesday, March 29, 2011

Scrum, XP and what about Self-Organizing teams?

A foundational principle of Scrum (and other Agile methods) is that we get better results from self-organizing teams. The Agile Manifesto says
“The best architectures, requirements, and designs emerge from self-organizing teams”
so it must be true.

A lot of Agile teams see improvements in productivity and quality. But how much of this comes from people working in self-organizing teams? And how much comes from better control and focus: from breaking work down into smaller pieces, working in timeboxes, holding frequent reviews. From faster feedback and more transparency with the customer. And how much from following disciplined technical practices, like developer testing, and code reviews or pairing, and refactoring.

How far do you need to go to get good, or even exceptional, results? Do you need to flatten down your organization, change your culture, change your management structure, change your HR practices and compensation model, change the way that you work with and communicate with the customer and with other teams… do you really need to change all of this to get better software out faster?

Learning from Survivor

In self-organizing teams, people are supposed to make decisions together, through consensus. Consensus works if you have a team of equals, or near-equals, and nobody (or buddies) can dominate the others. In reality, teams are dominated by stronger personalities or, if they are big enough, cliques.

We can learn some simple lessons from the TV show “Survivor”. Leave some people on an island, and pretty soon you will get politics. Some people will avoid responsibility, retreat from conflict: they don’t want to argue, they just want to get to work. Or they want to find ways to avoid working. Or they just don’t care. Some people will leave if they can find a way out. Others look forward to a good fight, get energized by it, but don’t have staying power or don’t know how to follow-up and get what they want done. So power and influence will go to people who are more articulate, more politically savvy, more confident, better at selling. Or to people who are more determined, more passionate about how work should be done, more willing to push for change, who have more to gain or less to lose. Or to people who are more manipulative and controlling.

These people will emerge as leaders in the group, and now you have replaced the role of management in the organization with new leaders who realize their agendas through cajoling, conning, convincing, ignoring, bullying, pressuring, outwitting, outplaying or outlasting the rest of the team into decisions.

I think that this is why some consultants are so strongly in favour of Agile teams and Agile methods: because in an Agile environment they can exert more influence, have more control over the outcome in a shorter time, play a bigger role on a team and have a chance to dominate it; rather than accepting, fitting into the existing organization and culture and “playing a role”. And if your team is having problems self-organizing and self-managing, well, there are consultants to help you with that too.

Swarm Intelligence?

I’m not convinced that the team needs to, or should, make all decisions together, and that the best decisions are always made by a “collective mind” or “swarm intelligence”. There is an important place for technical leads and experienced designers and technical specialists with valuable experience and skill. And there’s a good reason to let them do what they are good at. Even when you get a consensus-based democratic and fair decision from a team, as John Sonmez points out in When Scrum Hurts: Mob Architecture you tend to get average results, not exceptional results. If you have an expert on your team, why not let them make these kinds of decisions, so that you can aim for something better than average?

Management and Anti-Management

A common argument is that managers are not needed with self-organizing teams in Scrum – instead you have coaches who guide the team through their decision-making. This is part of what Uncle Bob Martin calls the anti-management bias in Scrum
“Scrum carries an anti-management undercurrent that is counter-productive. Scrum over-emphasizes the role of the team as self-managing. Self-organizing and self-managing teams are a good thing. But there is a limit to how much a team can self-X. Teams still need to be managed by someone who is responsible to the business. Scrum does not describe this with enough balance.”
Martin Fowler captured the importance of this balance in The New Methodology
“Such an approach requires a sharing of responsibility where developers and management have an equal place in the leadership of the project. Notice that I say equal. Management still plays a role, but recognizes the expertise of developers.”
The emphasis on equal is his, not mine.

Many people in the blogosphere take the position that if the teams are not self-organizing self-responsible self-directed (good), then the only alternative is hierarchical exploitative bureaucratic Command-and-Control (bad). But we need, and can have, a reasonable, middle path between extremes, a real-world and practical way of working.

Managers, good managers, respect and trust people to care about what they are doing and to do a good job, while still staying actively engaged. They offer direction, help define goals, help set priorities, step in and help when things are going wrong, handle conflict, and take responsibility when the business demands it. Good managers can help a team solve problems – or avoid them – because they have been in these situations before, and learned the hard lessons. A good manager understands that there are decisions that can be made and should be made by the team or by individuals on the team – and that there are other decisions that aren’t the team’s responsibility. That’s what managers get paid for.

Decisions about how to structure teams and how to get work done shouldn’t be about fear, or lack of trust. Or an organization’s culture, or inertia. Or politics. Or the current fashion. Instead, these decisions need to be about what’s best for the business. How to take advantage of all of the skills and capabilities that an organization has to offer – including management. And maximizing this advantage to the benefit of the team and of the customer.

Saturday, October 10, 2009

Dreaming in Code - A Failure in Leadership

Reading Scott Rosenberg’s Dreaming in Code gives you a sick feeling, the same sick feeling that you have watching a movie where the hero’s life is coming unraveled, or when you are involved in a project that is going nowhere fast, facing certain failure, and there is nothing that you can do to change the outcome. I made myself read it twice – there are some hard lessons for managing software development in this book.

Dreaming in Code tells the story of the failed Chandler project started by Mitch Kapor, software industry visionary and founder of Lotus Development Corp, currently Chairman of the Mozilla Foundation. Chandler began as an ambitious, “change the world” project to design and build a radically different kind of personal information manager, an Outlook-and-Exchange-killer that would flatten silos between types of data and offer people new ways of thinking about information; provide programmers a cross-platform, extensible open source platform for new development; and create new ways to share data safely, securely, cheaply and democratically, in a peer-to-peer model.

The project started in the spring of 2001. Because of Mitch Kapor’s reputation and the project’s ambitions, he was able to assemble an impressive team of alpha geeks. Smart people, led by a business visionary who had experienced success, interesting problems to solve, lots of money, lots of time to play with new technology and chart a new course: a “dream project”.

But Dreaming in Code is a story of wasted opportunities. Scott Rosenberg, the book’s author, followed the project for 3 years starting in 2003, but he gave up before the team had built something that worked – he had a book to publish, whether the story was finished or not. By January 2008, Mitch Kapor had called it quits and left the company that he founded. Finally, in August 2008, the surviving team released version 1.0, scaled back to a shadow of its original goals and of course no longer relevant to the market.

It is interesting, but sad, to map this project against Construx’s Classic Mistakes list, to see the mistakes that were made:

Unclear project vision. Lack of project sponsorship. Requirements gold-plating. Feature creep. Research-oriented development. Developer gold-plating. Silver bullet syndrome. Insufficient planning. Adding people to a late project. Overly optimistic schedules. Wishful thinking. Unrealistic expectations. Insufficient risk management. Wasted time in the “fuzzy front end”: the team spent years wondering and wandering around, playing with technology, building tools, exploring candidate designs, trying to figure out what the requirements were - and never understood the priorities. Shortchanged quality assurance… hold on, quality was important to this project. It wasn’t going to play out like other Silicon Valley startups. Then why did they wait almost 3 years before hiring a test team (of one person), while they faced continual problems from the start with unstable releases, unacceptable performance, developers spending days or weeks bug hunting.

The book begins with a meeting in July 2003, when the team should be well into design and development, where the project manager announces that the team is doomed, that they cannot hope to deliver to their commitments – and not for the first time. This is met with…. well, nothing. Nobody, not even Mitch Kapor, takes action, makes any decisions. It doesn't get any better from there.

This project was going to be different, it was going to be done on a “design-first” basis. But months, even years into the project, the team is still struggling to come up with a useful design. The team makes one attempt at time-boxed development, a single iteration, and then gives up. Senior team members leave because nothing is getting done. Volunteers stop showing up. People in the community stop downloading the software because it doesn’t work, or it does less than earlier versions – the team is not just standing still, they are moving backwards.

The project manager, after only a few months, quits. Mitch Kapor takes over. A few months later, the “fun draining out of his job”, he asks the team to redesign his job, and come up with a new management structure. OK, to be fair, he is rich and doesn’t have to work hard on this, but why start it all in the first place, why put everyone involved, even those of us who are going to read the book, through all of this?

The new management team works through a real planning exercise for this first time, making real trade-offs, real decisions based on data. It doesn’t help. They fail to deliver – again. And again with an alpha release. They then spend another 21 months putting together a beta release, significantly cutting back on features. By the time they deliver 1.0 nobody cares.

It’s a sad story. It’s also boring at times – the team spends hours, days in fruitless, philosophical bull sessions; in meaningless, abstract, circular arguments; asking the same questions, confronting the same problems, facing (or avoiding) the same decisions again and again. As the book says, “it’s Groundhog Day, again”. I hope that the writer may have not fully captured what happened in the design sessions and planning meetings – that, like the Survivor tv show, we only see what the camera wants us too, or happens to, an incomplete picture. That these people were actually more focused, more serious, more responsible than they come across.

The project failed on so many levels. A failure to set understandable, achievable goals. A failure to understand, or even attempt to articulate requirements. A failure to take advantage of talent. A failure to engage, to establish and maintain momentum. A failure to manage risks. A failure to learn from mistakes - their own, or others.

Most fundamentally, it was a failure of leadership. Mitch Kapor failed to hold himself and the team accountable. He failed to provide the team with meaningful direction, failed to understand and explain what was important. He failed to create an organization where people understood what it took to deliver something real and useful, where people cared about results, about the people who would, hopefully, someday, use what they were supposed to be building. And he gave up: he gave up well before 2008 when he left the company; he gave up almost from the start, he gave up when the hard, real work of building out his vision had actually begun.
Site Meter