If you could only do one thing to make better software, what would it be?

Good technical practices are what we have to do to make good software – this is the engineering part of software engineering. Design. Coding. Testing and Reviews.

If you could do only one thing to make better software, what would it be? Where would you get the most bang for your buck?

Continuous Integration – Making Code Run

Continuous Integration is an obvious place to start. You need to build the software and get it running before you can do anything useful with it.

Getting developers to check in and sync up with each other more often. Building the system more often – at least once a day to start, then on every check in. Which means simplifying and automating the steps to build the system. Making sure that the system builds successfully every time – without errors or warnings. Which means that people can run it and try it out whenever they want. Make sure that it will run correctly. Which means adding tests and checks as part of the build and deploy steps. Building information radiators so that everyone knows the status of the build and when the build is broken.

You can’t be Agile without Continuous Integration, and you need Continuous Integration in place before you can go down the Devops path to Continuous Delivery or Continuous Deployment.

And Continuous Integration works in sequential Waterfall delivery too. Developers in these environments might check in more code less often, but there is still real value in knowing that you can build and run the system and see it working sooner rather than later, especially in big enterprise systems and big programs where getting dependencies worked out and all the pieces working together is a huge challenge.

Developers testing their own work – Making Code Work

Making developers responsible for testing their own work, automating this as much as possible by building on Continuous Integration, is the only way to deliver software faster and keep costs down – depending too much on manual testing and hand-offs to a test team will slow you down too much.

Almost every organization that I have talked to over the past couple of years is pushing more responsibilities for testing onto developers, and pushing more testers into development teams (or out of the organization altogether), following the lead of Google and now Microsoft, to become "more Agile".

This means relying more on developers to write good automated tests (unit tests, basic UI regression using Selenium or Watir) and static analysis checking in Continuous Integration or the developer’s IDE to find common coding mistakes and security vulnerabilities.

But there are limits to what developers will catch in their testing, even good developers. Once you get developers to write tests (before, or after they write the code, it doesn't matter, now that TDD is dead), you’ll end up with mostly simple unit tests or UI regression tests that don’t stray far from the happy path, proving that the code does what the developer thinks it is supposed to do – because that is what they need to get their work done. Their assumptions and blind spots will be reflected in the tests as well as the code. Little or no negative testing. Or usability testing. Or security testing. Or stress testing. Or system-level integration testing. All of which still has to be done by somebody - unless you expect your customers to find your bugs for you.

It will take a long time before the team learns how to write good, efficient tests, and before they build up a set of tests that will catch real bugs, rather than just getting in the way. But if you do this right, you can deliver good code while still moving fast, and get better value out of testing.

Code Reviews or Pairing – Making Code Good

Another way to get better code is by getting developers to do code reviews.

Code reviews should be about finding problems in the code first – checking for correctness, defensive coding protection (error handling and API contracts and thread safety and data validation), security (using security libraries correctly for access control and output encoding, protecting confidential data, logging and auditing…). And about making the code better – more understandable, safer and easier to change.

Code reviews are expensive, so do them right: lightweight, risk-based, using static analysis first to catch low-level mistakes and bad coding practices so that reviewers can spend their time looking for more important problems.

Instead of code reviews, you could try pairing as a way to get another pair of eyes on the code.

Pairing isn’t the same as reviews – the goals and priorities are different. A good reviewer will find problems even in code developed through pair programming, because reviewers look for different things. But research proves that disciplined pair programming will give you better structured, cleaner code, with fewer bugs. And pairing is a much better way to teach programmers about the system than code reviews are.

The downsides of pair programming? The cost of having two people do the work of one person – a good pair will work faster than one person on their own, but the less experienced or less skilled team member will slow the pair down to what they can deal with. Focus fatigue. Pairing can be exhausting, which means people can’t do it for too long at a stretch, before their work becomes superficial or strained. And social problems. People who like it, like it a lot. But people who don’t like it won’t do it at all.

Refactoring – Making Code – and Design – Last

What about design? Collaborative design workshops? Design reviews? Threat modeling in design to take care of security and operational risks?

We do all of these things. But as we continue to iterate through the design and as our code base grows, refactoring – to retain, and sometimes restore, the design, and to keep the code maintainable – is becoming more and more important.

It’s easy to learn your IDE’s refactoring tools and the basic ideas behind refactoring. But it’s not easy to learn how do refactoring right (although you can learn a lot in a short time from Woody Zuill and Llewellyn Falco in their “2 Minutes to Better Code” video). Understanding why some refactoring approaches are better than others. How to save time refactoring. How to do it safely.

Mariusz Sieraczkiewicz does a good job of explaining how and when to do "everyday refactoring" using a matrix built on Michael Feathers’ work on brutal refactoring and the biology of code:

1. Start by reading and annotating the code, maybe do some scratch (rapid, throwaway) refactoring to understand it better
2. Find meaningful names for variables and conditionals
3. Extract methods to break down big chunks of code and express the algorithm
4. Get rid of obvious duplication
5. Move methods and extract classes to isolate responsibilities.

I agree with Sieraczkiewicz that these simple steps “would heal most code bases on this planet”. He then goes on to describe larger and more fundamental “strategic refactoring” (aka “root canal refactoring"): refactoring to patterns, introducing new architectural constructs. Work that carries much higher risks and costs. This is where refactoring ends, and re-design and re-architecture starts.

What would you do, to make Better Software?

Continuous Integration can pay off quickly: the change in transparency and in the team’s focus is almost immediate.

Developer testing is a journey, not a goal. It will take a long time for most developers to get good at it, and a long time to build up a good set of tests that you can rely on. The sooner you start, the better.

Code reviews can also take a long time to pay off. Developers – and managers – need to make the time for reviews to be done and build the discipline, and developers need time to learn how to review code properly, and how to give and accept criticism. But code reviews – or pairing – will give you better code.

Refactoring is more of a compounding investment – you pay a little bit today to save a lot in the future.

If there is only one thing that you could do to make better software, what would it be? Where would you start?

Don’t Waste Time Writing Perfect Code

A system can last for 5 or 10 or even 20 or more years. But the life of specific lines of code, even of designs, is often much shorter: months or days or even minutes when you’re iterating through different approaches to a solution.

Some code matters more than other code

Researching how code changes over time, Michael Feathers has identified a power curve in code bases. Every system has code, often a lot of it, that is written once and is never changed. But a small amount of code, including the code that is most important and useful, is changed over and over again, refactored or rewritten from scratch several times.

As you get more experience with a system, or with a problem domain or an architectural approach, it should get easier to know and to predict what code will change all the time, and what code will never change: what code matters, and what code doesn’t.

Should we try to write Perfect Code?

We know that we should write clean code, code that is consistent, obvious and as simple as possible.

Some people take this to extremes, and push themselves to write code that is as beautiful and elegant and as close to perfect as they can get, obsessively refactoring and agonizing over each detail.

But if code is only going to be written once and never changed, or at the other extreme if it is changing all the time, isn’t writing perfect code as wasteful and unnecessary (and impossible to achieve) as trying to write perfect requirements or trying to come up with a perfect design upfront?

You Can't Write Perfect Software. Did that hurt? It shouldn't. Accept it as an axiom of life. Embrace it. Celebrate it. Because perfect software doesn't exist. No one in the brief history of computing has ever written a piece of perfect software. It's unlikely that you'll be the first. And unless you accept this as a fact, you'll end up wasting time and energy chasing an impossible dream.”
Andrew Hunt, The Pragmatic Programmer: from Journeyman to Master

Code that is written once doesn’t need to be beautiful and elegant. It has to be correct. It has to be understandable – because code that is never changed may still be read many times over the life of the system. It doesn't have to be clean and tight – just clean enough. Copy and paste and other short cuts in this code can be allowed, at least up to a point. This is code that never needs to be polished. This is code that doesn't need to be refactored (until and unless you need to change it), even if other code around it is changing. This is code that isn't worth spending extra time on.

What about the code that you are changing all of the time? Agonizing over style and coming up with the most elegant solution is a waste of time, because this code will probably be changed again, maybe even rewritten, in a few days or weeks. And so is obsessively refactoring code each time that you make a change, or refactoring code that you aren't changing because it could be better. Code can always be better. But that’s not important.

What matters is: Does the code do what it is supposed to do – is it correct and usable and efficient? Can it handle errors and bad data without crashing – or at least fail safely? Is it easy to debug? Is it easy and safe to change? These aren't subjective aspects of beauty. These are practical measures that make the difference between success and failure.

Pragmatic Coding and Refactoring

The core idea of Lean Development is: don’t waste time on things that aren't important. This should inform how we write code, and how we refactor it, how we review it, how we test it.

Only refactor what you need to, in order to get the job done - what Martin Fowler calls opportunistic refactoring (comprehension, cleanup, Boy Scout rule stuff) and preparatory refactoring. Enough to make a change easier and safer, and no more. If you’re not changing the code, it doesn't really matter what it looks like.

In code reviews, focus only on what is important. Is the code correct? Is it defensive? Is it secure? Can you follow it? Is it safe to change?

Forget about style (unless style gets in the way of understandability). Let your IDE take care of formatting. No arguments over whether the code could be “more OO”. It doesn’t matter if it properly follows this or that pattern as long as it makes sense. It doesn't matter if you like it or not. Whether you could have done it in a nicer way isn’t important – unless you’re teaching someone who is new to the platform and the language, and you’re expected to do some mentoring as part of code review.

Write tests that matter. Tests that cover the main paths and the important exception cases. Tests that give you the most information and the most confidence with the least amount of work. Big fat tests, or small focused tests – it doesn't matter, and it doesn't matter if you write the tests before you write the code or after, as long as they do the job.

It’s not (Just) About the Code

The architectural and engineering metaphors have never been valid for software. We aren’t designing and building bridges or skyscrapers that will stay essentially the same for years or generations. We’re building something much more plastic and abstract, more ephemeral. Code is written to be changed – that is why it’s called “software”.

“After five years of use and modification, the source for a successful software program is often completely unrecognizable from its original form, while a successful building after five years is virtually untouched.”
Kevin Tate, Sustainable Software Development

We need to look at code as a temporary artefact of our work:

…we're led to fetishize code, sometimes in the face of more important things. Often we suffer under the illusion that the valuable thing produced in shipping a product is the code, when it might actually be an understanding of the problem domain, progress on design conundrums, or even customer feedback.
Dan Grover, Code and Creative Destruction

Iterative development teaches us to experiment and examine the results of our work – did we solve the problem, if we didn’t, what did we learn, how can we improve? The software that we are building is never done. Even if the design and the code are right, they may only be right for a while, until circumstances demand that they be changed again or replaced with something else that fits better.

We need to write good code: code that is understandable, correct, safe and secure. We need to refactor and review it, and write good useful tests, all the while knowing that some of this code, or maybe all of it, could be thrown out soon, or that it may never be looked at again, or that it may not get used at all. We need to recognize that some of our work will necessarily be wasted, and optimize for this. Do what needs to be done, and no more. Don’t waste time trying to write perfect code.

Automated Tests as Documentation

One of the arguments for writing automated tests is that tests can act as useful documentation for a system. But what do tests document? And who will find this documentation useful?

Most developers don’t rely on system documentation because there isn't enough documentation to give them a complete idea of how the system works, or because there’s too much of it to read, or because it’s not well written, or because it’s not up to date, or because they don’t believe it’s up to date.

But a good set of automated tests can tell you how the system really works today – not how somebody thought it works or how they thought it was supposed to work or how it used to work, if anybody bothered to write any of this down.

“Tests are no more and no less than executable documentation. When writing new code, it makes sense that it should be documented by the person closest to it: the author. That’s my rationale for developer testing.

Comment “doc blocks” and wiki pages, are always found to be inaccurate to some extent. By contrast automated tests fail noisily whenever they go out of date. So with regard to documentation, automated tests are uniquely advantageous in that (given a CI server) you at least have the option of keeping your documentation up to date whenever it starts to drift away from reality.”
Noah Sussman

You might be able to use tests as documentation, if…

To be useful as documentation, tests have to be:

1. comprehensive – they have to cover all of the important areas of the code and functions of the system;
2. run often and work – run on every check-in, or least often enough to give everyone confidence that they are up to date; and the tests have to pass – you can’t leave tests broken or failing;
3. written to be read – writing a test that works, and writing a test that can be used as documentation, are two different things;
4. at the right level of abstraction – most people when they talk about automated tests mean unit tests. …

Unit tests constitute design documentation that evolves naturally with a system. Read that again. This is the Holy Grail of software development, documentation that evolves naturally with a system. What better way to document a class than to provide a coded set of use cases. That's what these unit tests are: a set of coded use cases that document what a class does, given a controlled set of inputs. As such, this design document is always up-to-date because the unit tests always have to pass.
Jeff Canna Testing, fun? Really?

Of course even tests that run frequently and pass in Continuous Integration or Continuous Delivery can still contain mistakes. There will be tests that pass but shouldn't, or tests that tell you what the system does, but what the system does is not what what it is supposed to do (one of the risks of having developers writing tests is that if they misunderstood the requirement and got the code wrong, they got the tests wrong too). But on the whole, tests that run often should be more accurate than some document that may or may not have been right in the first place and that probably hasn't been kept up to date since. And through code coverage analysis, you can at least understand what parts of the system are described by the tests, and what parts aren't.

Tests have to be written to be read

One problem with using tests as documentation, is that tests are only a means to an end – even tests written up front in TDD. Tests are just another part of the supporting infrastructure and tooling that developers rely on to write code (and it's all about the code). Tests are tools to help developers think about what code they need to write (if they are following TDD), to prove that the code does what it was supposed to do, and to catch mistakes when people make changes in the future.

As a result, developers don’t put the same amount of attention and discipline into designing and implementing tests as they do with the code itself. As long as the tests run quickly and they cover the code (or at least the important parts of the code), they’ve done the job. Nobody has to care that much about how the tests are named or what they look like inside. Few teams peer review unit tests, and even then most reviewers only check to see that somebody wrote a test, and not that every test is correct, or that each test has a nice meaningful and consistent name and is easy to understand. There aren't a lot of developers who understand xUnit Test Patterns or spend extra time – or can afford to spend extra time – refactoring tests. So, a lot of automated tests aren't clean, consistent or easy to follow.

Another thing that makes unit tests hard to follow as documentation, is the way that tests are usually organized. Common conventions for structuring and naming unit tests are that developers will write a test class/module for every code class/module in the system, with test methods to assert specific behaviour in that code module.

Assuming that whoever wrote the code wrote a comprehensive set of tests and followed a consistent, well-defined structure and naming approach, and that they came up with good, descriptive names for each test class and test method and that everyone who worked on the code over time understood all of this and took the trouble to keep all of it up as they changed and refactored the code and moved responsibilities around and wrote their own tests (which is assuming a lot), then you should be able to get a good idea of what’s happening inside each piece of code by following the tests.

Can you really understand a system from Unit Tests?

But you’re still not going to be able to read a nice story about how a system is designed or what the system does or how it does it by reading unit tests.

Even if unit tests are complete and up-to-date and well written and well organized and have good names (if, if, if, if, if), the accumulation of details built up from looking at all of these tests is overwhelming. Unit tests are too close to the metal, sometimes obscured by fixtures and other test plumbing, and too far removed from the important aspects of the business logic or the design.

UnitTests include a lot of testing of lower level processes that have no direct connection to the stories.
Steve Jorgensen, comment in Unit Test as Documentation

Tests follow the code. You can’t understand the tests without understanding the code. So why not read the code instead? You’ll have to do this eventually to make sure that you really know what’s going on, and because without reading the code, you can’t know if the tests are well-written in the first place.

One place where low-level developer tests may be useful as documentation is describing how to use an API – provided that tests are comprehensive, expressive and “named in such a way that the behavior they validate is evident”.

A good set of unit tests like this can act as a reference implementation, showing how the API is supposed to be used, documenting common usage details:

If you are looking for authoritative answers on how to use the Rails API, look no further than the Rails unit tests. Those tests provide excellent documentation on what's supported by the API and what isn't, how the API is intended to be used, the kind of use cases and domain specific problems that drove the API, and also what API usage is most likely to work in future versions of Rails.
Peter Marklund, Rails Tip: Use the Unit Tests as Documentation

But again, this can only work if you take the time to design, organize and write the tests to do “Double Duty” as tests and as documentation, (Double Duty: How to repurpose the unit tests you’re doing to help create the documentation you’re not, by Brian Button) and make your tests “as understandable as humanly possible for as many different readers as possible”.

Tests as documentation?

I like the idea that automated tests can serve as documentation – we’d all save time and money this way. But who is this documentation for, and how is it supposed to be used?

I don’t know any developers who would start by reading tests cases in order to understand the design of a system. A good developer knows that they can’t trust documents or pictures, or tests, or even what other programmers tell them, or comments in the code. The only thing that they can trust is the code.

Tests might be more useful as documentation to testers. After all, it’s a tester’s job to understand the tests in order to maintain them or add to them. But most testers aren't going to learn much from most tests. When it comes to unit tests, it is the same for testers as it is for developers: unit tests aren't useful unless you understand the code – and if you understand the code, then you should read it instead. Higher-level acceptance tests are easier to understand and more useful to look at especially for non-technical people. It should be easier to tell a story and to follow a story about what a system does through high-level functional and integration scenarios (big fat tests), or acceptance tests captured in a tool like Fitnesse. Rather than asserting detailed implementation-specific conditions, these tests describe technical scenarios, or business rules and business workflows and other requirements that somebody thought were important enough to test for.

But even if you can follow these tests, there’s no way to know how well they describe what the system does without spending time talking to people, learning about the domain, testing the system yourself, and… reading the code.

I asked Jonathan Kohl, one of the smartest testers I know, about his experience using automated tests as documentation:

Back in '03 and '04, Brian Marick and I were looking into this. We held an experimental workshop at XP/Agile Universe in 2004 with testers, developers and other project stakeholders to see what would happen if people who were unfamiliar with the program code (both the code and the automated unit tests) could get something useful from the automated tests as documentation. It was a complete and utter failure. No one really got any value whatsoever from the automated unit tests from a doc perspective. We had to explain what was going on afterwards…

Marick and I essentially tossed the idea aside after that experience. I gave up on the whole double duty thing. Documentation is really good at documenting and explaining, while code is good at program creation and executing. I go for the right tool for the job now, rather than try to overload concepts.

Bottom line, I do not find tests useful as documentation at all. When done well, I do find them useful as examples of implementation of interfaces, etc. when I am new to a system, but nothing replaces a good written doc, especially when coupled with some face-to-face brainstorming and explanations.

There are a lot of benefits to automated testing, especially once you have a good automated suite in place. But documentation is not one of them.

7 Agile Best Practices that You Don’t Need to Follow

There are many good ideas and practices in Agile development, ideas and practices that definitely work: breaking projects into Small Releases to manage risk and accelerate feedback; time-boxing to limit WIP and keep everyone focused; relying only on working software as the measure of progress; simple estimating and using velocity to forecast team performance; working closely and constantly with the customer; and Continuous Integration – and Continuous Delivery – to ensure that code is always working and stable.

But there are other commonly accepted ideas and best practices that aren’t important: if you don’t follow them, nothing bad will happen to you and your project will still succeed. And there are a couple that you are better off not following at all.

Test-Driven Development

Teams that need to move quickly need to depend on a fast, efficient testing safety net. With Test First Development or Test-Driven Development (TDD), there’s no excuse for not writing tests – after all, you have to write a failing test before you write the code. So you end up with a good set of working automated tests that ensure a high level of coverage and regression protection.

TDD is not only a way of ensuring that developers test their code. It is also advocated as a design technique that leads to better quality code and a simpler, cleaner design.

A study of teams at Microsoft and IBM (Realizing Quality Improvement through Test Driven Development, Microsoft Research, 2008) found that while TDD increased upfront development costs between 15-35% (TDD demands developers change the way that they think and work, which slows developers down, at least at first), it reduced defect density by 40% (IBM) or as much as 60-90% (Microsoft) over teams that did not follow disciplined unit testing.

But in Making Software Chapter 12 “How Effective is Test-Driven Development” researchers led by Burak Turhan found that while TDD improves external quality (measured by one or more of test cases passed, number of defects, defect density, defects per test, effort required to fix defects, change density, % of preventative changes) and can improve the quality of the tests (fewer mistakes in the tests, tests that are easier to maintain), TDD does not consistently improve the quality of the design. TDD seems to reduce code complexity and improve reuse, however it also negatively impacts coupling and cohesion. And while method and class-level complexity is better in code developed using TDD, project/package level complexity is worse.

People who like TDD like it a lot, so if you like it, do it. And even if you are not TDD-infected, there are times when working test first is natural – when you have to solve a specific problem in a specific way, or if you’re fixing a bug where the failing test case is already written up for you. But the important thing is that you write a good set of tests and keep them up to date and run them frequently – it doesn't matter if you write them before, or after, you write the code.

Pair Programming

According to the VersionOne State of Agile Development Survey 2012, almost 1/3 of teams follow pair programming – a surprisingly high number, given how disciplined pair programming is, and how few teams follow XP (2%) or Scrum/XP Hybrid (11%) methods where pair programming would be prescribed.

There are good reasons for pairing: information sharing and improving code quality through continuous, informal code reviews as developers work together. And there are natural times to pair developers, or sometimes developers and testers, together: when you’re working through a hard design problem; or on code that you’ve never seen before and somebody who has worked on it is available to help; or when you’re over your head in troubleshooting a high-pressure problem; or testing a difficult part of the system; or when a new person joins the team and needs to learn about the code and coding practices.

Some (extroverted) people enjoy pairing up, the energy it creates and the opportunities it provides to get to know others on the team. But forcing people who prefer working on their own or who don’t like each other to work closely together is definitely not a good idea. There are real social costs in pairing: you have to be careful to pair people up by skill, experience, style, personality type and work ethic. And sustained pair programming can be exhausting, especially over the long term – one study (Vanhanen and Lassenius 2007) found that people only pair between 1.5 and 4 hours a day on average, because it’s too intense to do all day long.

In Pair Programming Considered Harmful? Jon Evans says that pairing can have also negative effects on creativity:

Research strongly suggests that people are more creative when they enjoy privacy and freedom from interruption … What distinguished programmers at the top-performing companies wasn’t greater experience or better pay. It was how much privacy, personal workspace and freedom from interruption they enjoyed,” says a New York Times article castigating “the new groupthink”.

And in “Still Questioning Extreme Programming” Pete McBreen points out some other disadvantages and weaknesses of pair programming:

• Exploration of ideas is not encouraged, pairing makes a developer focus on writing the code, so unless there is time in the day for solo exploration the team gets a very superficial level of understanding of the code.
• Developers can come to rely too much on the unit tests, assuming that if the tests pass then the code is OK. (This follows on from the lack of exploration.)
• Corner cases and edge cases are not investigated in detail, especially if they are hard to write tests for.
• Code that requires detail thinking about the design is hard to do when pairing unless one partner completely dominates the session. With the usual tradeoff between partners, it is hard to build technically complex designs unless they have been already been worked out in a solo session.
• Personal styles matter when pairing, and not all pairings are as productive as others.
• Pairs with different typing skills and proficiencies often result in the better typist doing all of the coding with the other partner being purely passive.

And of course pairing in distributed teams doesn't work well if at all (depending on distance, differences in time zones, culture, working styles, language), although some people still try.

While pairing does improve code quality over solo programming, you can get the same improvements in code quality, and at least some of the information sharing advantages, through code reviews, at less cost. Code reviews – especially lightweight, offline reviews – are easier to schedule, less expensive and less intrusive than pairing. And as Jason Cohen points out even if developers are pair programming, you may still need to do code reviews, because pair programming is really about joint problem solving, and doesn’t cover all of the issues that a code review would.

Back to Jon Evans for the final word on pair programming:

The true answer is that there is no one answer; that what works best is a dynamic combination of solitary, pair, and group work, depending on the context, using your best judgement. Paired programming definitely has its place. (Betteridge’s Law strikes again!) In some cases that place may even be “much of most days.” But insisting on 100 percent pairing is mindless dogma, and like all mindless dogma, ultimately counterproductive.

Emergent Design and Metaphor

Incremental development works, and trying to keep design simple makes good sense, but attempting to define an architecture on the fly is foolish and impractical. There’s a reason that almost nobody actually follows Emergent Design: it doesn't work.

Relying on a high-level metaphor (the system is an "assembly line" or a "bill of materials" or a "hive of bees") shared by the team as some kind of substitute for architecture is even more ridiculous. Research from Carnegie Mellon University found that

… natural language metaphors are relatively useless for either fostering communication among technical and non-technical project members or in developing architecture.

Almost no one understands what a system metaphor is any ways, or how it is to be used, or how to choose a meaningful metaphor or how to change it if you got it wrong (and how you would know if you got it wrong), including one of the people who helped come up with the idea:

Okay I might as well say it publicly - I still haven't got the hang of this metaphor thing. I saw it work, and work well on the C3 project, but it doesn't mean I have any idea how to do it, let alone how to explain how to do it.
Martin Fowler, Is Design Dead?

Agile development methods have improved development success and shown better ways to approach many different software development problems – but not architecture and design.

Daily Standups

When you have a new team and everyone needs to get to know each other and more time to understand what the project is about; or when the team is working under emergency conditions trying to fix something or finish something under extreme pressure, then getting everyone together in regular meetings, maybe even more than once a day, is necessary and valuable. But whether everyone stands up or sits down and what they end up talking about in a meeting should be up to you.

If your team has been working well together for a while and everyone knows each other and knows what they are working on, and if developers update cards on a task board or a Kanban board or the status in an electronic system as they get things done, and if they are grown up enough to ask for help when they need it, then you don’t need to make them all stand up in a room every morning.

Collective Code Ownership

Letting everyone work on all of the code isn't always practical (because not everyone on the team has the requisite knowledge or experience to work on every problem) and collective code ownership can have negative effects on code quality.

Share code where it makes sense to do so, but realize that not everybody can – or should – work on every part of the system.

Writing All Requirements as Stories

The idea that every requirement specification can be written as User Stories in 1 or 2 lines on cards, that requirements should be too short on purpose (so that the developer has to talk to someone to explain what’s really needed) and insisting that they should all be in the same template form

“As a type of user I want some goal so that some reason…”

is silly and unnecessary. This is the same kind of simple minded orthodoxy that led everyone to try to capture all requirements in UML Use Case format with stick men and bubbles 15 years ago.

There are many different ways to effectively express requirements. Sometimes requirements need to be specified in detail (when you have to meet regulatory compliance or comply with a standard or integrate with an existing system or implement a specific algorithm or…). Sometimes it’s better to work from a test case or a detailed use case scenario or a wire frame or some other kind of model, because somebody who knows what’s going on has already worked out the details for you. So pick the format and level of detail that works best and get to work.

Relying on a Product Owner

Relying on one person as the Product Owner, as the single solitary voice of the customer and the “one throat to choke” when the project fails, doesn't scale, doesn't last, and puts the team and the project and eventually the business at risk. It’s a naïve, dangerous approach to designing a product and to managing a development project, and it causes more problems than it solves.

Many teams have realized this and are trying to work around the Product Owner idea because they have to. To succeed, a team needs real and sustained customer engagement at multiple levels, and they should take responsibility themselves for making sure that they get what they need, rather than relying on one person to do it all.

What’s better – Big Fat Tests or Little Tests?

Like most startups, we built a lot of prototypes and wrote and threw out a lot of code as we tried out different ideas. Because we were throwing out the code anyways, we didn't bother writing tests - why write tests that you'll just throw away too?

But as we ramped the team up to build the prototype out into a working system, we got into trouble early. We were pushing our small test team too hard trying to keep up with changes and new features, while still trying to make sure that the core system was working properly. We needed to get a good automated test capability in place fast.

The quickest way to do this was by writing what Michael Feathers calls “Characterization Tests”: automated tests – written at inflection points in an existing code base – that capture the behavior of parts of a system, so that you know if you’ve affected existing behavior when you change or fix something. Once you’ve reviewed these tests to make sure that what the system is doing is actually what it is supposed to be doing, the tests become an effective regression tool.

The tests that we wrote to do this are bigger and broader than unit tests – they’re fat developer-facing tests that run beneath the UI and validate a business function or a business rule involving one or more system components or subsystems. Unlike customer-facing functional tests, they don't require manual setup or verification. Most of these tests are positive, happy path tests that make sure that important functions in the system are working properly, and that test validation functions.

Using fat and happy tests as a starting point for test automation is described in the Continuous Delivery book. The idea is to automate high-value high-risk test scenarios that cover as much of the important parts of the system as you can with a small number of tests. This gives you a “smoke test” to start, and the core of a test suite.

Today we have thousands of automated tests that run in our Continuous Integration environment. Developers write small unit tests, especially in new parts of the code and where we need to test through a lot of different logical paths and variations quickly. But a big part of our automated tests are still fat, or at least chubby, functional component tests and linked integration tests that explore different paths through the main parts of the system.

We use code coverage analysis to identify weak spots, areas where we need to add more automated tests or do more manual testing. Using a combination of unit tests and component tests we get high (90%+) test coverage in core parts of the application, and we exercise a lot of the general plumbing of the system regularly.

It’s easy to test server-side services this way, using a common pattern: set up initial state in a database or memory, perform some action using a message or API call, verify the expected results (including messages and database changes and in-memory state) and then roll-back state and prepare for the next test.

We also have hundreds of much bigger and fatter integration and acceptance tests that test client UI functions and client API functions through to the server. These “really big fat” tests involve a lot more setup work and have more moving parts, are harder to write and require more maintenance, and take longer to run. They are also more fragile and need to be changed more often. But they test real end-to-end scenarios that can catch real problems like intermittent system race conditions as well as regressions.

What’s good and bad about fat tests?

There are advantages and disadvantages in relying on fat tests.

First, bigger tests have more dependencies. They need more setup work and more test infrastructure, they have more steps, and they take longer to run than unit tests. You need to take time to design a test approach and to create templates and utilities to make it easy to write and maintain bigger tests.

You’ll end up with more waste and overlap: common code that gets exercised over and over, just like in the real world. You’ll have to put in better hardware to run the tests, and testing pipelines so that more expensive testing (like the really fat integration and acceptance testing) is done later and less often.

Feedback from big tests isn’t as fast or as direct when tests fail. Gerard Meszaros points out that the bigger the test, the harder is to understand what actually broke – you know that there is a real problem, but you have more digging to figure out where the problem is. Feedback to the developer is less immediate: bigger tests run slower than small tests and you have more debugging work to do. We’ve done a lot of work on providing contextual information when tests fail so that programmers can move faster to figuring out what’s broken. And from a regression test standpoint, it’s usually obvious that whatever broke the system is whatever you just changed, so….

As you work more on a large system, it is less important to get immediate and local feedback on the change that you just made and more important to make sure that you didn’t break something else somewhere else, that you didn’t make an incorrect assumption or break a contract of some kind, or introduce a side-effect. Big component tests and interaction tests help catch important problems faster. They tell you more about the state of the system, how healthy it is. You can have a lot of small unit tests that are passing, but that won’t give you as much confidence as a smaller number of fat tests that tell you that the core functions of the system are working correctly.

Bigger tests also tell you more about what the system does and how it works. I don’t buy the idea that tests make for good documentation of a system – at least unit tests don’t. It’s unrealistic to expect a developer to pick up how a system works from looking at hundreds or thousands of unit tests. But new people joining a team can look at functional tests to understand the important functions of the system and what the rules of the system are. And testers, even non-technical manual testers, can read the tests and understand what tests scenarios are covered and what aren’t, and use this to guide their own testing and review work.

Meszaros also explains that good automated developer tests, even tests at the class or method level, should always be black box tests, so that if you need to change the implementation in refactoring or for optimization, you can do this without breaking a lot of tests. Fat tests make these black boxes bigger, raising it to a component or service level. This makes it even easier to change implementation details without having to fix tests – as long as you don’t change public interfaces and public behavior, (which are dangerous changes to make anyways), the tests will still run fine.

But this also means that you can make mistakes in implementation that won’t be caught by functional tests – behavior outside of the box hasn’t changed, but something inside the box might still be wrong, a mistake that won’t trip you up until later. Fat tests won’t find these kinds of mistakes, and they won’t catch other detailed mistakes like missing some validation.

It’s harder to write negative tests and to test error handling code this way, because the internal exception paths are often blocked at a higher level. You’ll need other kinds of testing, including unit tests and manual exploratory testing and destructive testing to check edge cases and catch problems in exception handling.

Would we do it this way again?

I’d like to think that if we started something brand new again, we’d start off in a more disciplined way, test first and all that. But I can’t promise. When you are trying to get to the right idea as quickly as possible, anything that gets in the way and slows down thinking and feedback is going to be put aside. It's once you’ve got something that is close-to-right and close-to-working and you need to make sure that it keeps working, that testing becomes an imperative.

You need both small unit tests and chubby functional tests and some big fat integration and end-to-end tests to do a proper job of automated testing. It’s not an either/or argument.

But writing fat, functional and interaction tests will pay back faster in the short-term, because you can cover more of the important scenarios faster with fewer tests. And they pay back over time in regression, because you always know that aren’t breaking anything important, and you know that you are exercising the paths and scenarios that your customers are or will be – the paths and scenarios that should be tested all of the time. When it comes to automated testing, some extra fat is a good thing.