Thursday, April 19, 2012

You don’t need Testers – Or do you?

I talk to a lot of people in both big and small software development organizations about how they manage software development, how they’re organized, what practices they follow and what practices actually work. Most people working on small teams that I talk to can’t justify having someone to just test their apps, because testers don’t actually build software, so they’re considered overhead. That means that developers need to test the software themselves – or the customer will have to do it.

What do testers do on an Agile team?

Quite a few Agile teams believe that you don’t need testers to deliver working software. Testers are looked upon as a relic from the waterfall days (requirements, design, code, then pass off to test). On XP teams, everyone is a developer, and developers are responsible and accountable for testing their own code, writing automated unit tests and then automating the acceptance tests that the Customer has defined. Scrum doesn’t explain how testing is done at all – the team will find a way to figure it out as they “inspect and adapt” themselves towards good practices.

If developers are already testing their own code (and maybe even pairing up to review code as it is written), then what do you need testers for?

Janet Gregory and Lisa Crispin wrote a big book to justify the role of testers on Agile teams and to explain to programmers and testers how testers can fit into Agile development, but this hasn’t changed the attitude of many teams, especially in “engineering-driven cultures” (startups founded by programmers).

One of their arguments is that Agile teams move too fast for testers, that black box testers writing up test plans and working through manual test scripts or constantly updating their Quality Center or Selenium UI regression tests can never catch up to a team delivering new features in short sprints. If the testers don’t have the technical skills to at least write acceptance tests in something like Fitnesse or Cucumber, or if they don’t have the business domain knowledge to help fill in for the Customer/Product Owner and answer developer questions, what are they good for?

This is taken to the extreme in Continuous Deployment,a practice made popular by companies like IMVU and Facebook where developers review their work, write automated tests, check the code and tests in and if the tests pass, the changes are immediately and automatically pushed to production.

Letting Customers test your work

Some shops look at Continuous Deployment as a chance to “crowdsource” their testing – by getting their customers to do their testing for them. It’s actually promoted as a competitive advantage. But it’s really hard – maybe impossible – to write secure and reliable software this way, as I have looked at before. For a critical review of the quality of a system continuously deployed to customers, read James Bach’s fascinating post on 20 minutes spent testing one of the poster child apps for Continuous Deployment and the problems that they found in the app in just a short period of time.

Other Continuous Deployment shops are more careful and follow Etsy/Flickr’s approach of dark launching: deploying changes continuously, but testing and reviewing them before turning them on progressively for customers and closely monitoring the outcome.

Regardless, it’s important to remember that there are some things that customers can test and in fact only customers should test: whether a feature is useful or not, whether a feature is usable, what kind of information they need to do a task properly, what the optimal workflow is. This is what A/B split testing is supposed to be about – experimenting with ideas and features and workflows, collecting usage data and finding out what customers use or like best and what they don’t. To evaluate alternatives and get feedback.

But you don’t ask your customers to test whether something is finished or not, whether the code works or not, whether the system is stable and secure or whether it will perform under load.

What do you need from your test team?

Even the best, most responsible and experienced developers make mistakes. In our shop, everyone is an experienced developer – some of them have been working in this domain for 10-15 years or more. They carefully test their own work and update the automated unit/functional test suite for every check-in. These tests and static analysis checks are run in Continuous Integration – we’ve learned to lean heavily on the test suite (there are thousands and thousands of tests now with a high level of coverage) and on static analysis bug checking and security vulnerability checking tools to find common coding mistakes. All code changes are also reviewed by another senior developer – without exception.

Even with good discipline and good tools, good programmers still make mistakes: some subtle (inconsistencies, look-and-feel problems, data conversion and setup, missing edits) and some fundamental (run-time failures under load, concurrency problems, missed requirements, mistakes in rules, errors in error handling). I want to make sure that we find most (if not all) of these mistakes before the customers do. And so do the developers.

That’s where our test team comes in. We have a small, experienced and highly-specialized test team. One tester focuses on acceptance testing, validating functional requirements and usability and workflow with the business. Another tester works on functional regression and business rules correctness and coverage, looking for missing rules and for holes in the developer’s test suites, and automating our integration tests at the API level. And the other tester’s main thing is operational testing, stress testing for spikes and demand shocks and soak testing to look for leaks and GC issues, destructive system testing and bug hunting – actively trying to break the system. They all know enough to fill in for each other when someone is away, but they each have their own unique knowledge and skills and strengths, and their own ways of approaching problems.

When we were first building the system we started with a larger test team focused more on coverage and assurance, with test planning and traceability and detailed manual testing checklists, and writing automated regression tests at the UI. But there was a lot of wasted time and effort working this way.

Now we depend more on automated tests written by the developers underneath the UI for functional coverage and regression protection. Our test team puts most of their effort into exploratory functional and system and operational testing, risk-based and customer-focused targeted tests to find the most important bugs, to find weaknesses and exploit them. They like this approach, I like it, and developers like it, because we find real and important bugs in testing, the kinds of problems that escape code reviews and unit testing.

They smoke test changes as soon as developers check them in, in different customer configurations. They pair up with developers to test through new features and run war games and simulations with the developers to try to find run-time errors and race conditions and timing issues and workflow problems under “real-world” conditions. They fail the system to make sure that the failure-detection and recovery mechanisms work. They test security features and setup and manage pen tests with consultants. They run the system through an operational day. Together with Ops they also handle integration certification with new customers and partners. They do this in short sprints with the rest of the team, releasing to production every 2 weeks (and sometimes more often).

The test team is also responsible for getting the software into production. They put together each release, check the dependencies, they decide when the release is done, what will make it into a release and what won’t, they check that we have done all of the reviews that the team agreed to, they test the roll-back and data conversion routines and then they work with Ops to deploy the release through to production.
They don’t slow the team down, they don’t keep us from delivering software. They help us make sure that the software works and that it gets into production safely.

Testers find more than bugs

I’ve worked for a long time in high-assurance, high-integrity businesses where not having testers isn’t an option – the stakes of making mistakes are too high. But I don’t think that you can build real software without someone helping to test it. Unless you are an early stage startup pounding out a proof of concept, or you are a small team building something trivial for internal use (but then you probably won’t read this), you need help testing the system to make sure that it works.

It doesn’t matter how you are working, what method you follow - Agile or Waterfall doesn’t change the need for testers. If you’re moving fast and light, testers need to adapt to the pace and to the way that they get and share information. That’s ok. Good testers can do that.

I’m not naïve enough (any more) to think that the test team will find all of the bugs that might be in the system – or that this is their job. Of course, I hope that the testers will find any important or obvious bugs before customers do.
What I need for them to do is to help us to answer some important questions: Are we ready to release? What’s too rough or unstable or incomplete, what needs to be backed-out, or what needs further review, or maybe a rewrite? What’s weak in the design? Where are we missing automated tests? Where do we need better test tools? What features are too hard to understand, or inconsistent, or too hard to setup? What error messages are missing or misleading? Are we trying to do too much, too fast? What do we need to change in the design, or the code, or the way that we design or code the system to make it better, more reliable?

Testing doesn’t provide all possible information, but it provides some. Good testing will provide lots of useful information.
James Bach (Satisfice)

Without testers, not only do you put out code that you shouldn’t with bugs that you should have caught – you also lose a lot of important information about how good your software really is and what you need to do to make it better. If you care about building good software, this is an opportunity that you cannot pass up.


Kane Barton said...

Thank you so much for taking the time to write an honest, comprehensive blog post on an extremely important aspect of our industry.

Anonymous said...

Really nice post which states tester importance

Rosie Sherry said...

Awesome post and pretty spot on I think. Not often I read through a whole blog post :)

From what I've seen it would appear that people want to say whether test is dead or alive. No inbetweens. There's been a fair bit of debate about this recently.

Testing is changing. Perhaps the job title should change. Or perhaps in earlier years people expected testers to do too much of the testing, where much of the coded/automated/unit stuff should have been done by developers all along.

There are developers/testers. There are testers/developers. There are support/testers. There are designers/developers. Etc.

Why is it assumed that everyone should be confined to a strict job title?

Testing is not dead, we just need to see and show that testers can do more than just test & find bugs. There needs to be the passion and interest in the business, in creating a great company, in innovation...from everyone, not just testers.

Maxim Shulga (aka MaxBeard) said...

Very interesting post. Thank you. So testers are not for bugs, but for expertise, for exploration and to provide you first feedback.

Jim Bird said...


The job of testers is to find bugs AND provide feedback to developers and management. AND to help understand risks that the developers aren't looking for. If you're not getting good information from the test team, or not using this information, you're not doing it properly.

A tester finds a bug. The developer can a) fix the bug and go on to the next thing; or b) look at what the tester did, try to understand what kind of mistake they made, why they didn't catch it in their own testing, what other mistakes like this could be in the code, what help they need to prevent these mistakes in the future. Mature teams and developers follow the second approach.

Jim Bird said...


Thank you. Yes when I talk to development teams and people who do testing for a living, it does seem that there is a line between organizations that either "can't afford" or don't see the need for testing (relying on good developers and good tools, or relying on their customers) and organizations that spend a lot on testing and aren't always sure what they are getting out of it. So then they look to offshore the testing work to reduce costs, with mixed results but at least it costs less.

I do agree with you that people can play multiple roles, especially in smaller organizations. I worked as a "support/tester" in one small development shop years ago. Developers reviewed and tested their code, and then the support team (me and my buddy) tested it, and our overseas distributors tested it when they got updates. Everybody helped in testing.

Jesper L. Ottosen said...


Panna said...

Thank you so much for writing this up.

I am in full support for "testers are not for bugs....." as put by Maxim Shulga.....

Erich said...

This is a good article, and continues to get the idea out that testers are ultimately information providers.

I worked as a hardware engineer for a few years, and while software is a different animal (of sorts), there are some strong parallels. Hardware tends to have cleaner requirements (under the hood at least) - input set A results in output B. But it seems like parallels can still be drawn.

In the hardware world, the engineers (developers analog) would spend a LOT of time making sure the functional requirements were met. At most, I would write up extensive test procedures, then hand them off to a tech to go through (there were also fixtures for test automation). So I still think that the developers have a vested interest in producing quality - making sure their software does the best it can.

Where the QA team came into play was regarding areas of failure that required additional expertise and some level of acceptance/integration testing. The test team could centralize the resources to perform regulatory compliance testing and reliability/performance testing.

It seems that can work in Software, too. The developer should be able to render the requirements into a clean set of development goals and code to those goals. Then the test team can focus on things like Security, Performance, Stability, all those other "-ities." We can also focus on the overall user experience (customer acceptance). When I am spending my time writing low-level functional tests, I have to spend time reading and understanding the code (which the devs already do), and it takes time away from these other areas of concern. In that case, it lowers the quality in both areas, since I do not have the time for comprehensive Unit/Functional Testing, and I have lost time from Acceptance/Compliance testing.

Erich said...

Thanks for the article - it does a good job of describing where testers fit into the agile software process, and re-iterates the idea that QA/Test people are the measurers and information providers regarding the software product.

My early career was in hardware, and I found there that the roles for design and QA were fairly well-understood. As the hardware designer, it was my responsibility to see the production through design, prototyping, etc..., and to make sure the functional specifications were met. QA came in to measure things that were generally outside my expertise (regulatory compliance, environmental issues) or that required test environments on a larger scale than was practical for the individual engineer.

It seems that software should be similar. As testers, we are better suited to identifying the larger issues such as security, performance, reliability, etc.... We also are valuable as a resource for test environments that cannot easily be set up by the individual developer. I am frustrated when a developer "hands off" code with the attitude of "Please find my bugs." I would rather see them say "I defy you to find a functional bug in this code!" and expect feedback at the systems level. If I have to write functional automation, then I end up spending time reading the code and doing branch analysis, which the developers should have already done, since they are the code experts.

Thanks again for the article!

alejo699 said...

Testers ARE there to find bugs - trust me, I've found thousands - but we are also valuable for our perspective, particularly in game development. I've been playing video games my whole life, and even though I can't tell you how to fix something, I can tell you why it doesn't work. Too often developers and designers are too busy to be able to step back from their work and look at it the way a user would. That's where we come in.

Mike said...

Very nice article.

This might not apply to everyone but where I work, Testers are the "encyclopedias" of the company.

Developers are very Project-focused. They can tell you every bit of detail about a Project that they have worked on. But they cannot tell you about an overall Product. They might not have been involved in all changes to the Product.

Testers, however, are the ones who collect business knowledge, immerse themselves in it. A Tester will be able to describe the entire functionality of a Product. A Developer typically won't be able to without looking at the code.

A Tester will also be interesting in the "Why" (Why does this button need clicking twice?) rather than just the "How" that Developers are typically only interested in.

aya said...

@Jim Bird

You shared the details about your test team of 3 members.
How many developers were working on the project?


Mobile App Development Company in India said...

Thanks for sharing such a great blog... I am impressed with you taking time to post a nice info.
Website Development Company in Delhi
Website Designing Company in Delhi
Mobile App Development Company
Mobile App Development Company in India

Stephanica said...

Thank you for the nice article here. Really nice and keep update to explore more gaming tips and ideas.

AR/VR Game Testing

Gameplay Testing Services

Video Game Testing Company

Video Game Testing Company

Anonymous said...

If you are facing any issue with downloading and using QuickBooks connection diagnostic tool, get in touch with our QuickBooks ProAdvisors. Call us on our QuickBooks desktop error support number +1-877-263-2742 to get the best solutions.

combination pharmaceutical development said...

The combinations products are also considered with the relationship the development of devices and establishments of regular sources and also determine with understand the lots of needs. However, the Combination of pharmaceutical development and devices the different variation begin with clear customer requirements. In the main factor, we can understand the use of products as well as it is very safe and secure the high-quality products with meet your customer requirements. There are possible to get the development of your regulated added the complexity in the development of process and select the life cycle management.

Kartik said...

If need any technical assistance for queries related to QuickBooks, browse

write off bad debt in QuickBooks
how to rebuild data in QuickBooks
QuickBooks Error 6177
QuickBooks error code 6000 77
QuickBooks is unable to send your emails to outlook

QuickBooks Install Diagnostic Tool said...

Your Blog is very much Informative. Now I would like to tell you about QuickBooks Install Diagnostic Tool.

QuickBooks said...

Quickbooks Error 80029c4a
Quickbooks Error Code 80029c4a

Athiya said...

Thanks for sharing such informative blog.

QuickBooks balance sheet does not balance
how to print pay stubs in QuickBookss
update QuickBook Software
qbwc1085 error
QuickBooks would not open company file
QuickBooks online bank reconciliation
email setup in QuickBooks
QuickBooks pdf converter Tool

Pharmaceutical Development Group said...

PharmDevGroup, or as they are popularly known, PDG, has provided the world with some of the finest regulatory services, often being called the best Pharmaceutical Regulatory Consultants by some of the leading Research and Development companies and teams in their world. One is encouraged to call PDG at +1 (813) 963-3062, and to email them at, where one can learn more about the services that PDG offers its clients, and witness for themselves the efficiency PFG shows at their job.

Other Services:

FDA regulatory consulting
Combination Drugs Development
rx-to-otc switch
otc drug development

QuickBooks Payroll Support said...

QuickBooks Payroll is a feature of QuickBooks, which can help you in calculation of salaries, wages, commissions, and incentives, etc. Every version of QuickBooks Payroll has its essence. But while working with aforementioned versions any issue can come, so to resolve that, we have a dedicated team. We provide QuickBooks Payroll Support to terminate your any errors related to the software. Our QuickBooks Payroll Support Phone Number is 1-888-986-7735.

QuickBooks POS Support said...

QuickBooks Error 3180 occurs either due to when you send a record to QuickBooks or when you send time section from BillQuick to QuickBooks. Resolve this error as soon as possible because it can harm your company’s file.
Connect to our QuickBooks Pro Support team to resolve any QuickBooks error on 1-888-986-7735.

QuickBooks Database Server Manager
QuickBooks error 6000
QuickBooks POS Financial exchange error

QuickBooks Support | +1 (888)2530666 | Quickbooks Customer Support said...

In this competitive world keep the business growing with every passing day is very tough. In order to chase the company’s success, everything has to be done in a perfect manner. All resources of the QuickBooks Customer Support should work efficiently. It could be anything from a simple pantry service to intangible software. QuickBooks is also one of the parts from those important resources. QuickBooks is accounting software which is used by small and medium-sized companies. Visit : Quickbooks support

Stephanica said...

Thank you for the nice article here. Really nice and keep update to explore more gaming tips and ideas.

Game QA Company

Game Functionality Testing

Game Compatibility Testing

Game Compliance Testing

QuickBooks Enterprise support said...

quickBooks server requirements are the basic needs of any system that you be inherent or inserted for the hardware or software to function smoothly and efficiently.
here is all the detail realted to system reuirement.

John Trump said...

Great Post
QuickBooks Remote Access | QuickBooks Has Stopped Working

Microblading Oakville said...

Thanks for sharing the article. If you are looking for permanent solution for pigmentation and want to have permanent tattoo? Microblading Oakville is the permanent solution.

Furniture Restoration Brooklyn said...

Thanks for sharing the article. Book Furniture Restoration Brooklyn best case scenario rates. Simons of Manhattan prepared and ensured experts offer first rate Furniture Repair administrations.

Quickbooks Payroll said...

Nice Blog It’s a really informative for all. Quickbooks accounting software is helpful for maintain client payroll taxes accounting problems. We are providing technical support in Quickbooks Payroll Support Phone Number . Our technical support team is 24*7 available so if you need any help. Please call us our Toll-free Number + 1-800-986-4607.

Quickbooks Payroll said...

Nice Blog It’s a really informative for all. Quickbooks accounting software helps you to solve accounting problems. We are providing technical support in Quickbooks Support Phone Number 1800 . We also provide guidance & all types of information about Quickbooks. So if you need any issue Please call us our Toll-free Number + 1-800-986-4607

Quickbooks Payroll said...

Nice Blog It’s a really informative for all. Quickbooks accounting software helps you to solve accounting problems. We are providing technical support in Quickbooks Support Phone Number 1800 . We also provide guidance & all types of information about Quickbooks. So if you need any issue Please call us our Toll-free Number + 1-800-986-4607

otel rezervasyon said...

ek wielrennen 2019
ek wielrennen live
ek wielrennen 2019 alkmaar
ek wielrennen 2019
ek wielrennen live
ek wielrennen 2019 alkmaar
ek wielrennen 2019
ek wielrennen live
ek wielrennen 2019 alkmaar
ek wielrennen 2019
ek wielrennen live
ek wielrennen 2019 alkmaar
ek wielrennen 2019
ek wielrennen live
ek wielrennen 2019 alkmaar

Unknown said...

Hello friends, nice post and nice urging commented at this place, I am in fact enjoying by these.Regards Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.

software testing services
software quality assurance services
software testing services company
Software Qa Services
quality assurance service providers
Performance testing services
Security testing services
software testing Companies

Stephanica said...

Thank you so much for this nice information. Hope so many people will get aware of this and useful as well. And please keep update like this.

Various Stages of Game Testing Techniques you need to know

7 Essential Tips for Successful QA Implementation

Types of Game Testing Processes that need to be followed

How Game Testing differs from Software Testing

6 Challenges that every Game Tester Faces

9 Critical Bugs to be Identified in Game Testing process

Is the age of AAA gaming dying?

Major Mobile Game Testing Concerns for Testers

Game Testing Trends to watch out for in 2020

Mark james said...

Your HP printer may come to an error state when you get some new updates in your system. Sometimes the error may come due to driver problem. Overcome HP printer is in error state error with the help of certified experts.

Site Meter