Thursday, April 25, 2013

Code Ownership – Who Should Own the Code?

A key decision in building and managing any development team is agreeing on how ownership of the code will be divided up: who is going to work on what code; how much work can be, and should be, shared across the team; and who will be responsible for code quality. The approach that you take has immediate impact on the team’s performance and success, and a long-term impact on the shape and quality of the code.

Martin Fowler describes three different models for code ownership on a team:

  1. Strong code ownership – every module is owned exclusively by someone, developers can only change the code that they own, and if they need to change somebody else’s code, they need to talk to that owner and get the owner’s agreement first – except maybe in emergencies.

  2. Weak code ownership – where modules are still assigned to owners, but developers are allowed to change code owned by other people. Owners are expected to keep an eye on any changes that other people make, and developers are expected to ask for permission first before making changes to somebody else’s code.

    This can be thought of as a shared custody model, where an individual is forced to share ownership of their code with others; or Code Stewardship, where the team owns all of the code, but one person is held responsible for the quality of specific code, and for helping other people make changes to it, reviewing and approving all major changes, or pairing up with other developers as necessary. Brad Appleton says the job of a code steward is not to make all of the changes to a piece of code, but to “safeguard the integrity + consistency of that code (both conceptually and structurally) and to widely disseminate knowledge and expertise about it to others”.

  3. Collective Code Ownership – the code base is owned or shared by the entire team, and everyone is free to make whatever changes they need – or want – to make, including refactoring or rewriting code that somebody else originally wrote. This is a model that came out of Extreme Programming, where the Whole Team is responsible together for the quality and integrity of the code and for understanding and keeping the design.

Arguments against Strong/Individual Code Ownership

Fowler and other XP advocates such as Kent Beck don’t like strong individual code ownership, because it creates artificial barriers and dependencies inside the team. Work will stall and pause if you need to wait for somebody to make or even approve a change, and one owner can often become the critical path for the entire team. This could encourage developers to come up with their own workarounds and compromises. For example, instead of changing an API properly (which would involve a change to somebody else’s code), they might shoe horn in a change, like stuffing something into an existing field. Or they might take a copy of somebody’s code and add whatever they need to it, making maintenance harder in the future.

Other arguments against strong ownership are that it can lead to defensiveness and protectionism on the part of some developers (“hey, don’t touch my code!”), where they take any criticism of the code as a personal attack, creating tension on the team and discouraging reviewers from offering feedback and discouraging refactoring efforts; and local over-optimization, if developers are given too much time to spend to polish and perfect their precious code without thinking of the bigger picture.

And of course there is the “hit by a truck factor” to consider – the impact that a person leaving the team will have on productivity if they’re the only one who works on a piece of code.

Ward Cunningham. one of the original XPers, also believes that there is more pride of ownership when code is shared, because everyone’s work is always on display to everyone else on the team.

Arguments against Collective Code Ownership

But there are also arguments against Collective Code Ownership. A post by Mike Spille lists some problems that he has seen when teams try to “over-share” code:

  • Inconsistency. No overriding architecture is discernible, just individual solutions to individual problems. Lots of duplication of effort results, often leading to inconsistent behavior
  • Bugs. People "refactoring" code they don't really understand break something subtle in the original code.
  • Constant rounds of "The Blame Game". People have a knee jerk reaction to bugs, saying "It worked when I wrote it, but since Joe refactored it....well, that's his problem now.".
  • Slow delivery. Nobody has any expertise in any given domain, so people are spending more time trying to understand other people's code, less time writing new code.

Matthias Friedrich, in Thoughts on Collective Code Ownership believes that Collective Code Ownership can only work if you have the right conditions in place:

  • Team members are all on a similar skill level
  • Programmers work carefully and trust each other
  • The code base is in a good state
  • Unit tests are in place to detect problematic changes (although unit tests only go so far)

Remember that Collective Code Ownership came out of Extreme Programming. Successful team ownership depends on everyone sharing an understanding of the domain and the design, and maintaining a high-level of technical discipline: not only writing really good automated tests as a safety net, but everyone following consistent code conventions and standards across the code base, and working in pairs because hopefully one of you knows the code, or at least with two heads you can try to help each other understand it and make fewer mistakes.

Another problem with Collective Code Ownership is that ownership is spread so thin. Justin Hewlett talks about the Tragedy of the Commons problem: people will take care of their own yard, but how many people will pick up somebody else’s litter in the park, or on a street - even if they walk in that park or down that street everyday? If the code belongs to everyone, then there is always “someone else” who can take care of it – whoever that “someone else” may be. As a developer, you’re under pressure, and you may never touch this piece of code again, so why not get whatever you need to do as quickly as possible and get on to the next thing on your list, and let "somebody else" worry about refactoring or writing that extra unit test or...?

Code Ownership in the Real World

I've always worked on or with teams that follow individual (strong or weak) code ownership, except for an experiment in pure XP and Collective Code Ownership on one team over 10 years ago. One (or maybe two) people own different pieces of the code and do all or most of the heavy lifting work on that code. Because it only makes sense to have the people who understand the code best do most of the work, or the most important work. It’s not just because you want the work “done right” – sometimes you don’t really have a choice over who is going to do the work.

As Ralf Sudelbucher points out, Collective Code ownership assumes that all coding work is interchangeable within a team, which is not always true.

Some work isn't interchangeable because of technology: different parts of a system can be written in different languages, with different architectures. You have to learn the language and the framework before you can start to understand the other problems that need to be solved.

Or it might be because of the problem space. Sure, there is always coding on any project that is “just typing”: journeyman work that is well understood, like scaffolding work or writing another web form or another CRUD screen or fixing up a report or converting a file format, work that has to be done and can be taken on by anyone who has been on the team for a while and who understands where to find stuff and how things are done – or who pairs up with somebody who knows this.

But other software development involves solving hard domain problems and technical problems that require a lot of time to understand properly – where it can take days, weeks, months or sometimes even years to immerse yourself in the problem space well enough to know what to do, where anyone can’t just jump in and start coding, or even be of much help in a pair programming situation.

The worst disasters occur when you turn loose sorcerers' apprentices on code they don't understand. In a typical project, not everyone can know everything - except in some mature domains where there have been few business paradigm shifts in the past decade or two.
Jim Coplien, Code Ownership

I met someone who manages software development for a major computer animation studio. His team has a couple of expert developers who did their PHDs and post grad work in animating hair – that’s all that they do, and even if you are really smart you’ll need years of study and experience just to understand how they do what they do.

Lots of scientific and technical engineering domains are also like this – maybe not so deeply specialized, but they involve non-trivial work that can’t be easily or competently done by generalists, even competent generalists. Programming medical devices or avionics or robotics or weapons control; or any business domain where you are working at the leading edge of problem solving, applying advanced statistical models to big data analysis or financial trading algorithms or risk-management models; or supercomputing and high-scale computing and parallel programming, or writing an operating system kernel or solving cryptography problems or doing a really good job of User Experience (UX) design. Not everyone understands the problems that need to be solved, not everyone cares about the problems and not everyone can do a good job of solving them.

Ownership and Doing it Right

If you want the work done right, or need it to be done right the first time, it should be done by someone who has worked on the code before, who knows it and who has proven that they can get the job done. Not somebody who has only a superficial familiarity with the code. Research work by Microsoft and others have shown that as more people touch the same piece of code, there is more chance of misunderstandings and mistakes – and that the people who have done the most work on a piece of code are the ones who make the fewest mistakes.

Fowler comes back to this in a later post about “Shifting to Code Ownership” where he shares a story from a colleague who shifted a team from collective code ownership to weak individual code ownership because weaker or less experienced programmers were making mistakes in core parts of the code and impacting quality, velocity and the team’s morale. They changed their ownership model so anyone could work around the code base, but if they needed to change core code, they had to do this with the help of someone who knew that part of the code well.

In deciding on an an ownership approach, you have to make a trade-off between flexibility and quality, team ownership and individual ownership. With individual ownership you can have siloing problems and dependencies on critical people, and you’ll have to watch out for trucks. But you can get more done, faster, better and by fewer people.

7 comments:

  1. Hello,

    Nice post!

    While collective code ownership has some disadvantages as well, in my opinion, if it's managed well, it always wins from individual code ownership.

    Both have risks, as you correctly pointed out, but I think the difference is in the fact that the risk is spread out with collective ownership.

    What I mean by that is that during a project with collective ownership indeed there are always moments when things go slower than would be possible because of junior programmers touching advanced parts (or inexperienced programmers in that area for that matter). But this risk is measurable and you can account for it when you do planning and estimation. The introduction of bugs may also be higher, but again, this is solvable by a decent quality process. The fact is that you get a stable speed and a measurable quality.

    On the other hand, with strong individual code ownership, you will set a base standard for the velocity and quality standard of the team. This is good and constant until you have a situation where you do need someone to touch code that is not his (and this WILL happen). This will cause a lot of fluctuations in quality and speed.

    In my opinion the overall speed and quality are not necessarily affected by either one. It's just that with collective ownership this is measurable and stable, whereas individual ownership usually brings surprises and fluctuations in both, making it hard to estimate and plan.

    ReplyDelete
  2. Jim,

    Nice post on ownership. I saw that you referenced my paper on ownership at Microsoft. You may be interested to know that I replicated the study on Firefox and Eclipse and found differing results.

    The results are here (this paper is basically the paper you cited plus some results for OSS, so just skim to the good parts)

    http://research.microsoft.com/apps/pubs/default.aspx?id=140711

    We've found that when people who have never touched the code before make a change, they are more likely to make mistakes. But they can't become experts unless they're working in that part of the code, so that presents something of a conundrum.

    I've been wondering if there's a way to transfer the knowledge required to be an expert in a part of the system without requiring them to actually make changes (which would introduce risk, since they aren't owners). Involving developers in reviews of changes to code may be one way to get them to invest the time to gain more than surface familiarity with the code and how it fits into the system with minimal risk. At MS, we do a lot of review, but I'm wondering how many other projects use review heavily?

    ReplyDelete
  3. Christian,

    Your point about knowledge transfer through reviews is a good one. My experience is that you can effectively transfer through reviews, as long as the same reviewers get to see the code often enough to build a real understanding of what the code does, and of how the code has changed over time.

    As the same reviewers work on the the same code again and again, the quality and nature of their reviews change too. As they get to know more about the code and the coder, they look at and for different things. For example, they learn that one developer forgets to validate API return values carefully, or that they tend to make some other kind of mistake or oversight, so they watch out specifically for these weaknesses and tendencies. Reviewers are also able to make better suggestions on how to simplify the code once they have had a chance to work with the code more than once; and they become less tolerant of complexity and ambiguity after they have had to look at the same code a few times, pushing harder to make the code simpler and clearer. Which means that in the next review, they should be able to see and understand more, in a self-reinforcing way.

    ReplyDelete
  4. Kenneth,

    You make a good point that collective code ownership is more predictable - that quality and productivity average out over time across the team. I can see advantages to that.

    What I have struggled with is that on a lot of projects, there is at least some work (and sometimes a lot of work) that can't easily be shared, work that requires specialist technical or domain knowledge to do properly. So even if you have a team of All Stars, all of the work isn't interchangeable. Unless maybe you have had all of the same people sharing everything since the beginning, everyone with equal ability building up equal familiarity with every part of the system together. That's an optimal situation, I wonder how many teams have been able to achieve that, and sustain that, in complex domains?

    ReplyDelete
  5. What if we used distributed version control and a code lieutenant model. You get freedom to refactor and like Linus on the kernel, your patch either gets in or not yet, or is kept out of the final shipped code base until issues are sorted out. Everyone owns their own repo which is a branch and the team or its lieutenants are assigned to approve and Merge up feature branches to trunk.

    ReplyDelete
  6. The analysis matches with my experience with both strong individual ownership and collective ownership. What I don't like about collective ownership is that there isn't much motivation to fully understand/refactor the code when you are under schedule pressure and the quality is a real concern.

    ReplyDelete
  7. Balanced and thoughtful — a rare property for an internet post. Thanks for the tour of ideas.

    ReplyDelete

Note: Only a member of this blog may post a comment.