Does AI for Code give a false sense of security?

When talking to developers who are new to Diffblue Cover we often get asked, can Diffblue Cover give you a false sense of security? Some common questions are:

  1. Will Diffblue Cover generate tests (and therefore coverage) that will say current bugs are ok?

  2. Will Diffblue Cover generate lots of code coverage in low risk and low-value areas, therefore inflating the code coverage without adding real value?

To consider these questions we need to think about why we write test cases for the code that exists today. With AI for Code (you can find out more about what AI for Code is here) we can be highly accurate about how the code will behave. We cannot be anywhere near as accurate for business logic or requirements.

‘Clippy’ from Microsoft Word

‘Clippy’ from Microsoft Word

I’m sure we all remember Clippy - the wonderful paperclip assistant that I don’t think ever actually helped me. Why was it so ineffective? Because it was trying to second guess my every move. Contrast this with Grammarly. Grammarly is not trying to guess whether you are writing a letter, a report, or a thesis; it simply tells you that you just made a stupid mistake.

Diffblue Cover using AI for Code will help the developer write meaningful unit tests more quickly, cheaply, and easily than is possible by hand. It won’t tell you that it looks like you’re writing an e-commerce website and you must do it the way the tool says. Essentially one person's bug can be another person’s product.

In many initial scans of a repository, a large number of tests are generated. If you are scanning a product that is currently in production then you can assume the current behavior is ok. Sure, some of the behavior may not be desirable, but regression bugs are really painful (a topic for another blog post). In this position, it is ideal for Diffblue Cover to produce test cases that demonstrate the behavior of the code being analyzed.

Given that generating test cases against the current code is good, here are my top tips for using Diffblue Cover:

1. Focus on the high-value code

I bet that any developer who has worked on a project for a while has a list of code that scares them. This might be because it has broken multiple times in the past, or the impact of it breaking is huge, or no one on the project understands the code, or they are just uncomfortable about its quality. This is a great place to start using Diffblue Cover, get the tests and check them into your code base.

Now you can change this code with increased confidence. Why? Well, your regression suite just improved significantly with minimal effort.

2. Review test cases for new code

You’ve now generated tests for the new class you just wrote. Diffblue Cover has generated enough tests to hit your coverage targets. You’re done, right? Wrong! Review the tests, they are telling you how your code behaves, are you happy that every test is correct? If one of the tests shows a behavior you don’t want it’s showing you a bug. We often see examples of edge cases that developers haven’t considered being picked up by Diffblue Cover.

Here is the value add, don’t wait for Quality Assurance engineer to tell you there is a problem with the code, review the code’s behavior through the unit tests and fix bugs earlier.

3. Failing test cases are good

What do failing test cases mean? Simply that you have changed the behavior. No unit tests (even those written by humans) can guess the behavior you want in the product tomorrow. So every time a test fails, you need to think, is this desirable, or not? If it is desirable, feel free to use Diffblue Cover to create a new test case.

So does AI for Code and Diffblue Cover give you a false sense of security? No. Diffblue Cover is a tool; used correctly it will give you more confidence in the code changes that you are making at a lower cost than manual methods. Developers, we are empowering you by making you wildly more productive.

Want to find out what our tools can do for you? Take a look at Diffblue Playground, explore our website, request a demo of Diffblue Cover or get in touch with me directly @jgwilson42.

Image source - https://www.artsy.net/article/artsy-editorial-life-death-microsoft-clippy-paper-clip-loved-hate via Carnigie Mellon University.