Is your company’s code covered by a comprehensive test suite?

Transcript

Hello and welcome to the 3rd and final part of our webinar series on automated testing. I hope you’ve enjoyed the first two webinars of this series, which were about why everyone loves automated testing and why everyone is bad at it. If you haven’t seen them, please go have a look on our website or YouTube channel. Today, I’m going to talk about test coverage: specifically, is your company’s coverage good enough? I’m going to look at code coverage and I’m also going to talk about some other metrics you can use and how you can have a comprehensive enough unit test suite for your code base. 

So let’s talk about code coverage. Why is code coverage so important when we talk about comprehensive test suites? Well, first of all, many organizations and developers deprioritize unit testing. From an organizational point of view they see it as a cost that is not necessary, and from a developer’s perspective they don’t always want to spend their time writing tests. If we think about code coverage as a metric, it will tell you whether or not your developers are writing unit tests, so what we will do later on is talk about how we can measure whether your test suite is comprehensive using code coverage and other tools. Before we get there, let’s have a look at why we want a good test suite. 

Martin Fowler talks about a comprehensive test suite giving engineering teams the ability to modify and refactor code. So what does this mean? What we’re saying here is that if you have a comprehensive test suite, you can be confident that you’re not breaking the code as you move forward. So as you are adding that new feature or refactoring the code, you can be confident that you haven’t broken everything. We have spoken before about why regression bugs breaking something that works before is really bad. 

So how do you know if your company’s code is well covered? The important question here is: what does good coverage look like? Given that we have just spoken about bugs being really bad, what we’re looking for is a test suite that ensures that you know when you introduce any regression bugs. Some of the questions you can look at are: how often do you find bugs in production? How often does your QA team find bugs through manual testing? Are you picking up all your bugs during build time? Do you feel like there’s an issue in code quality? Quite often your developers will know the answer to this question. 

There is one small problem with all of this, and it’s really down to the lag time. If you’re waiting until your code goes into production to find out if your test suite is good enough, it’s too late. What you need is a metric that’s going to give you the confidence that you’ve got good coverage as soon as you possibly can. This is where code coverage comes in itself. It’s going to tell you which parts of your code base are covered and which are uncovered. It’s obviously not going to give you a view of the quality of the test cases and it’s not going to give you a view as to which use cases are covered. At least you’re not going to have a good idea of which code that you’ve written has been exercised by your unit test and which code is going to be exercised first time by QA (or even worse, by your customers). 

Once we’ve decided you’re going to use code coverage to measure how comprehensive the test suite is, we can improve quality. Coming back to the fact that by catching bugs earlier they are less likely to end up in production and increase productivity if you’re picking up bugs during build time, because then the developer can fix them straight away as soon as they have introduced it, and not days or weeks later. This in turn leads to faster delivery, and in any code base that’s easy to work with where developers feel productive, you’re going to get increased engagement from developers. 

So ensuring that you’ve got high-coverage, comprehensive test suites is going to be good for your business and for your engineering team. So I’m sure you’re all wondering what great secrets we have for increasing coverage. Well I recently wrote the blog on the topic and looking back over other parts of my career there have been two options: either manually writing test cases in-house or manually writing test cases using an outsourced team. There are obvious downsides here: both of them are really slow and very expensive. We propose option three: don’t let humans do it, make sure you are using technology to improve your coverage. You can see on our website the case study from Goldman Sachs where they have used Diffblue Cover for increasing coverage in legacy code and avoided the cost of using humans either internally or outsourced to deliver the same benefit. 

So over to your questions, let’s see what we have coming in. So the first question here: I’ve spoken a lot about coverage. Isn’t quality more important than quantity? That’s a great question and I think the answer is both are important. It’s no good having a massive unit suite with thousands of tests that doesn’t really test anything the customer is going to do. Equally, having one perfect test case is no good either. There needs to be a balance between the amount of code that’s being covered and the amount of functionality being tested and the quality of the test cases, both from the point of view making sure that they are testing what your customers are going to do with the code. Also that they are easy to maintain and debug, to make the developer’s life easier. 

We have another question here. What’s a good code coverage target? The suggestion here is 80%. This is one of my favorite questions. When you start looking at code coverage, is 80% good enough? Well, the answer is probably, maybe, it’s hard to say. Is 80% better than 79%? I really think it comes down to what is the coverage you’re missing. If in that 20% you’ve not covering the most critical code for your application (for example: if you’re doing an ecommerce website and you’ve covered 80% of your code but you haven’t covered the check out process) I pretty much say that it’s not good enough. 

Code coverage targets are great because they are quantifiable. You can easily set them and everyone understands what they mean, but they’re not the most important thing for you to track. You need to be looking at the code that’s uncovered. If you’re looking at a specific target, I’m a big fan of beat yesterday. In so many different areas of life, just saying I will be better than I was yesterday is a huge start, and it’s more about the trend and the change occurring than the absolute value. 

How do you balance test writing with new development? Well, we all know that if you don’t do the test writing, if you allow your test suite to decay over time; you’re basically building up a technical debt. You’re going to hit a point when new development suffers because of it. The problem is when your stakeholders are coming to you and saying “I need this new feature, I need it as quickly as possible, just give it to me now, I don’t care about everything else” and you think “just this once, I won’t write tests and I’ll come back to it later and fill it in,” but you never do. I think the only way to balance test writing with new development is when you quote for the development time, be it in story points in agile, in time, you include it in the time of writing tests. If we all know the downsides to not including the test writing at that point and the fact that we don’t come back to it, we should say it’s not negotiable, and we should include it in the estimates. 

Finally, someone’s come in talking about quality of tests and why using our tool is better than developers writing tests? Diffblue Cover is a great tool for creating test cases based on existing code. It’s not a great tool for understanding your business logic. So, if you’ve got the time, if you’ve got man-years of time to go and get your coverage from where it is today up to where you want it to be then go ahead and use the developers to do it, that’s probably going to be the best quality tests. 

However, I’m yet to find a company that’s got that amount of resources lying around spare. So what we’re looking at doing here is producing good quality tests for a fraction of the cost of using internal or external resources. So, not only we can give you your test suite overnight or in a day or two rather than years’ worth of effort, we can also do it from a fraction of the cost. I would suggest that having used the test cases from Diffblue Cover as you get through refactoring or editing the code you take a look at the test cases. You’ll probably find times where you can improve the quality of the tests with some simple maintenance and this should be factored in alongside your development costs. 

I think that is all we have time for today. Thank you very much for listening, do send us any further comments and questions. You can reach us through our website or Twitter. Thank you.