Why improving testing capabilities with AI is so critical for DevOps

Transcript

Hello, and welcome to this AI and DevOps webinar with Andy Piper, where we’ll be talking through the question of why improving testing capabilities with AI is so critical for DevOps. As enterprises scale their DevOps software engineering practices, there can be many challenges, roadblocks and hurdles that need to be navigated to deliver the best result and improve testing capabilities. 

Over the course of this webinar, we will be taking a look at the key components of DevOps in testing, exploring the current landscape in DevOps and finally examining what the future holds for DevOps and the critical inclusion of Artificial Intelligence for continued scale. Andy, it’s great to have you here. Before we dive into the crux of today’s webinar, it would be great if we could get a summary on your background, what brought you to Diffblue and what are your thoughts around DevOps..  

So, I’m Andy Piper. I’m VP of engineering at Diffblue. My background is very much software product development. I’ve led development teams and I’ve been involved in the development of software products, primarily middleware based in Java, for 20+ years. 

So, DevOps, as everyone knows, isn’t a new thing. People have been trying to do software better for a long time, the challenge is businesses want it quicker but they don’t want bugs in it. Development wants to deliver quickly but recognizes the challenge with delivery of software. 

DevOps was developed around some of the agile practices that came around in terms of development leading to incremental delivery, and the automation around that allows us to do that successfully without network and retesting. So, we used to develop in a very linear fashion: we’d come up with function specs and design and do some coding and some more coding and some testing and some integration testing and some user testing, all in this very linear waterfall. Agile recognizes that doing things in that way meant that you stood at risk of ending up with a product nobody wanted or that was too late by the time you understood the problem. This took some development time, and by then the problem had changed or you were too far into implementation so we ended up with lots of failed projects because by the time they wanted to ship it, it was all too late. 

So, agile kind of recognized that you needed to counter the years of problems on a continuous basis as you would do in development so you could navigate your way through that problem and solution space. The way that agile did that was focusing on the important things, prioritizing things, allowing cross functional teams to come together to deliver solutions to user problems rather than implementation of specification. 

Apart from that iterative more collaborative style, you’re always trying to deliver value on an incremental basis. It then became the question of, how can I deliver that value right to the end user? How can I do that incrementally? I think that is one of the reasons DevOps came about; that desire to incrementally deliver value all the way to the end user. Now obviously an end user is not going to be happy that an alpha or beta version of the product is shipped every two weeks; they want the finished thing that they know works. So how do you solve that problem? You solve that problem by building in a lot of automation in testing and all the rest of it. 

Another difference is that the way software is delivered has changed massively. We no longer ship stuff on CDs and the rest of it. Stuff is online, software is a service, you can get incremental updates to software instantaneously; as soon as someone wants to release it you can have it in your hands because you are subscribing, because the whole model of software delivery has changed and the way we consume that software too, so the model and how we develop that software has changed. 

Getting back to why DevOps, that sort of concept of putting things in the hands of users very frequently exposes one of the issues that you can’t have humans involved. It has to be automated. You have to have this very fast cycle time and if you have people pushing buttons or reviewing things, that’s where errors creep in. It’s just not physically possible. A big drive around DevOps is automating the hell out of everything, automating every place where it is possible. So testing deployment, running test deployment, user experience, those sorts of things and that quest for automation is ongoing, so people want to automate more and take the possibility of user error out of the equation and focus on the automation side of things. 

Perhaps it’s interesting to explain a little bit about my personal exposure to DevOps because I wouldn’t be involved in this space if I didn’t believe in it. I guess I have a couple of interesting examples. In one, I was involved with a company where we were delivering a software product and when I joined the company we had a product that was composed of a number of different pieces in a number of different technologies. We had tremendous problems with simply getting that product in the hands of customers in a reliable repeatable way. 

So we had the classic symptoms of: someone’s done a build on their laptop, the customer picked it up and it was the wrong build or it had the wrong dependency, the repeatability was a big issue. On top of that, the time taken to create a release—having said a lot of software was online, this particular product was not online, it was a piece of software you downloaded and you put in the hands of customers but the same sorts of problems apply—the time taken to actually create the pieces to give to customers was too long and lots of manual steps in it and of course errors crept in, various issues, customers were upset because the bug they wanted fixed wasn’t fixed—you can imagine the scenario. 

As part of the leadership team, I changed that process so it was almost all automated, the release process of the product. That involved a number of interesting challenges. You have to be able to build all the different technology parts of the product separately and bring them together. You can’t build a Windows product on an Apple Mac and you can’t build Apple iOS targeted on Windows. So some interesting challenges there. We solved all of those but the net effect was that we were able to do releases much quicker and much more repeatedly so we could always guarantee the thing we were giving to the customer was the thing that we actually wanted them to have. 

That made a big impact on development, so we could focus more on development rather than the handle turning on how do we get this release out. Which meant we could do more with the resources we had available, and that was a key part in my DevOps experience. 

Another interesting part was that although we were putting software in the hands of customers, the infrastructure we needed to build that software wasn’t e.g. someone’s laptop—we’re talking servers and all the rest of it. The key part of DevOps is this whole idea of infrastructure as code, so the systems you use to assemble and run things are managed in the same way you manage your code. It’s in source code control and all the rest of it. That again means they are repeatable, so if a truck drives through your office and takes down your server room, you can rebuild it all because it is all in GitHub somewhere. So the idea that you treat the process like code was important to our success there and very formative to my understanding of DevOps. 

I think going back a bit further when I was at BA, we were doing DevOps types of things but no one was really talking about DevOps like that. One of the key points there was having a way of automating your running tests so that every developer checking could be verified in some way. While this wasn’t a process I drove, I was part of the experience while I was there and we went from an end-to-end linear process (where you would only really find out if you had broken things days later) to a process where we could run testing pipelines against individual developer changes. We wouldn’t merge those changes unless the testing pipelines ran, and that was great from developer productivity point of view, but also great from a delivery point of view. 

The worst thing in the world is when you commit a change, some tests breaks several days down the road and you don’t really know what’s broken. If you know what’s broken, it’s job done. That’s not normally the case; you do git blame or something to try and figure out where the problem occurred and that is tremendously expensive. You start having to do things like stopping developers from checking in stuff and it stops the pipeline. It brings the whole thing grinding to a halt. Once the system is no longer running smoothly it’s basically not running at all. 

This idea was that every time a developer checks something in, we knew that tests were not going to break, because we’d done that verification at front. That was tremendously powerful because it meant we could make guarantees around shipping time and we knew we wouldn’t have a massive ramp down into the ship because we knew there wouldn’t be any gremlins that would come out of the woodwork later on in the process. To dig in a little bit more, in the old world when we were very linear about development, the problem was that we found issues late and finding issues late is very expensive. Because it’s A. figuring out what the problem is and B. fixing and testing it. 

In software testing, there’s this classic QA pyramid where generally you want a lot of very simple tests run as close to the developer process as possible to find the obvious problems. Then on top of that you want a smaller number of complicated tests, then a small number of integration tests. Right at the top you’ll have QA folk doing manual regression testing (or even users, if you want to consider users as testing). The whole point of this is to try and catch problems as early as possible because that’s the cheapest and easiest time to find issues. 

Once you’ve started to stitch things together (or if you’re developing firmware for embedded systems and you’ve rolled it out to million devices) that’s not a great time to try and find bugs. It becomes tremendously expensive to fix. We can all think of examples: Intel microcode flaws make great examples of very expensive bugs to fix because they were found very late in the process. You want to find bugs as early as possible and because development is ongoing, development is continuous, you want to do testing in a similar manner; you want testing to be continuous and ongoing in a similar way. 

That said, if you move all your tests as early in the process as possible, the developer might be happy but they won’t be happy if moving all the tests early means that they have to wait forever for their changes to get validated. You still need this pyramid and you still need later stages to catch more complex scenarios. You need to do that in a way that doesn’t impact developer productivity but also makes sure the quality of the product is still high. As well as testing going on automatically and continuously as part of the development process, you want testing happening automatically and continuously as part of the end-to-end pipeline of development through to release to try and strike that balance of quality and speed. 

That’s great Andy. It’s really good to hear your personal experience and a really thorough overview and impact over the last couple years of where DevOps has come from since its inception. I’ve got some thoughts: if it really is seemingly the ideal path to follow DevOps, why do you think agile is still dominating conversation over DevOps? 

I don’t know for certain, but I have some observations. I think one would be that clearly a lot of people are doing what used to refer to waglie: agile and waterfall methodology. So not truly doing agile, just some of that is going on. I think also there’s been this shift earlier on in my career there was a very distinct delineation between development and QA. Development tends to do development and then stuff would get handed to QA for testing. That line has been blurred more and more. Modern development practices advocate things like TDD, coverage based development, where you’re trying to write tests as part of your development process being guided by the coverage you see in your ID. 

All of those things require a mental shift for developers. It requires discipline, it requires training, it requires education. Often, with developers, the adrenaline rush is when something works, and the most natural thing in the world is to ignore testing in the pursuit of getting something working. I suppose the educational change is that you can get there quicker if testing becomes part of that process. A lot of people haven’t adopted that, so I think they are beginning to get au fait with agile but not reacting to this need for testing to be built into the process. 

Cool, thanks Andy. I think it’s interesting your point around how testing might be seen as taking a backseat against developers when they are doing their software development. I’m wondering what you think about what AI is taking steps towards, with writing intelligent tests, or if there’s already talk of using software to automate testing and support DevOps and developers. I wonder what your thoughts are around AI being able to go that next level and actually write tests intelligently. 

It’s important to separate out the mechanics of running tests and detecting failures versus the process of writing tests, and tests can take a number of different forms. There are tests that you write to verify what you believe is correct behavior and correct operations. There are also tests that you write because of TDD, you’re writing tests when there is no operation and writing against implementation that doesn’t exist. You come up with failing tests and then you drive your development off the back of it. I think there are a number of different routes through that problem. 

I see test automation as being much more about the mechanics of running tests rather than the process of writing tests and mechanizing test running is a relatively easy problem to solve. Automating the writing of tests is a much harder problem and that’s why not many have done it. That is why Diffblue is focused on the problem and is doing good stuff there. I think everyone would love to automate the writing of tests if they could, but it’s a hard problem.

Thanks Andy. That’s some really interesting insight to understand the impact on day-to-day development processes. Now taking a wider view on DevOps, it’d be interesting to hear what you think around challenges, hurdles and potential roadblocks that could exist in an organization if they wanted to adopt the DevOps mindset. 

DevOps requires some discipline, so there is an upfront investment cost to DevOps. To do infrastructure as code and test automation you have to invest in technology, you have to invest in processes. For instance, if you’re going to automate anything, you have to first figure out what the thing is. You can’t automate a process you don’t understand or don’t know. Some companies have challenges with simply understanding what they’re doing at the moment, so automating those aspects is going to be hard, unless they can get their hands around that. 

I think some of the challenges are, customers recognize they are doing things wrong so they try to make changes to the process while they’re also trying to automate stuff and that never tends to end well. You’re better off automating what you have. Once you have automation and some feedback then you have the ability to figure out if you’re getting better or not. Then you can start to make changes. 

I think there’s some organizational challenges there. I also think there’s some investment timing challenges as well because DevOps requires some level of upfront investment. For instance, let’s take test automation. There is manual testing that at some level is always going to be quicker than writing an automation framework if you don’t have that framework. There’s always this challenge that businesses are under the gun to ship products, code, applications and sometimes they never get the opportunity to take the time and spend money. They need to pause to get the automation done then move on. I think that one of the challenges is delivery is always, “we’ll do it tomorrow because we need to ship today.” Whereas actually, if you look at the long view, it would be a much better place to actually make the investment and then reap the benefits. Yes there’s going to be some delay, some pause initially, but medium-term that’s going to pay off massively. 

That’s not a technology problem it’s an organization problem that people have to recognize that these things have costs. You’ve got to make strategic decisions about how you’re going to spend time and money to get benefits. I think the onus there is on business leaders to recognize that DevOps requires change and they need to be supporting in empowering that change. 

You probably wouldn’t need to go back very far and business leaders wouldn’t have a clue what DevOps was. Whereas these days, there’s a recognition that it’s changing. They talk about IT having a seat in the boardroom these days, and that’s changing more and more as the process of development and delivery becomes an important part of that IT conversation. 

So, for the future DevOps I think one of the key pieces is people don’t necessarily know what good looks like. They’ve read on Twitter about Skyscanner doing 200 developments a day, but they have no clue how to get there. The expertise required to get there is in high demand and short supply. This is true of IT all over. The really good people are thin on the ground. 

What is the solution there? One solution is to hope you can train up many more people to be really good at these things, that generally hasn’t worked out too well. What tends to happen is that we use computers to solve the same sorts of problems. We raise the problems to high levels of abstraction so that more normal people can use those tools or systems or processes. 

Compile is a great example of writing machine code. This was actually a hard thing to do, then compilers came along written by some really smart people, but they came along and raised a higher level of extraction to widen the user base. 

You don’t need to be a complete expert to write computer code. DevOps is the same, and I think tools and systems are the key here. Once we know what good processes and practice technology looks like, how can we automate that effectively? We’re seeing that shift happen as we speak; there have been some DevOps companies like Puppet and Chef traditionally dealing with configuration management but getting the process of, how do you put together DevOps pipelines easily in a way that you don’t have to be a complete expert to do it? 

Similarly, with Amazon code build, Amazon code pipeline, all those sorts of things, it’s putting easy-to-use tools that embody best practices in the hands of users that don’t have the time to do a PhD in DevOps. So, I think we’re going to see this more and I think that’s great because for businesses it takes a lot of the guesswork out of it. 

Also you don’t need some expert to come along and say “you need this widget and that widget, and no I can splice these together, and by the way it’s going to cost £7 million.” In the end, you can sign up for a service on Amazon and get the same thing for a much lower price point. I think that’s one trend, the next trend is obviously where’s next? That’s making the mechanical piece easier to consume, but ultimately with DevOps we’re trying to take people out of the equation where they can be taken out. 

Obviously we want developers, coders to do what they are best at, which is creatively develop solutions for user problems and not get them doing mechanical work. AI has a big part to play there because it’s not that AI can think like a human but AI can be used more intelligently to take some of the grunt work out of these sorts of tasks. 

Testing is a big part of that and I think everybody would recognize that there’s quite a mechanical aspect of testing. People know what tests or a certain class of test should look like and what they should do. Ideally, they would want a computer to do that for them. It’s one of those problems where it’s easy for a human to say, “well this is what it should do, this is how it should work,” and much less easy for a computer to do, but not impossible. 

That’s one of the key shifts and not just AI and machine learning. I know of companies using machine learning to optimize the operational side of things so its embodying human behavior in a machine where it’s appropriate, makes sense and where benefit can be realized. That’s freed up developers to do what they are good at, and the great thing about it is that it means that generally developers are happier because they are focusing on tasks they love. This generally means the business is happier because you’re removing the mechanistic time consuming parts of the process that can slow things down. 

Thanks Andy. It’s great to have a view inside around the current landscape of DevOps and the relationship it has with the wider organization. Let’s say if tomorrow an organization decided to focus on DevOps, what’s the best approach you think they should take and where can AI sit with this adoption? 

Key to the how is partly technology changes that can be made to increase in DevOps. It’s about adoption but also about the business’ ability to adopt the process changes that require the mind shift that is required to recognize why changes need to be made. Those pieces are critically important before you start looking at AI type solutions. 

That said, AI can help bridge some of the gap between developments and operations. Maybe it can do some things that otherwise you’d never get in terms of testing automation. As long as AI is low touch, what’s not to like about it? If we can add value to developer and developer workflows without increasing their burden, then that can only be a good thing. 

Over time, AI is going to be the critical thing in DevOps because that’s the only way we’re going to be able to scale quick developments to organizations. Organizations are always trying to get more value out of their resources, faster, at ROI and so on. AI is going to help us improve our coverage and also does so in a predictable automated way that doesn’t need maintaining. 

Ultimately, reducing cost can improve the security as well. I mentioned earlier that AI is about taking the mechanistic process that humans are currently doing and replacing that with a machine. The question is, where are you going to get the biggest bang for your buck? Testing is a big part of that. Everyone recognizes the value of testing. Everybody probably recognizes they don’t do enough testing. So how can you get to more testing without increasing overhead or introducing errors? 

This is where Diffblue is great. We have a product that allows you to write unit tests automatically based on your software and that can take out a whole sway of mechanistic testing that you might have to do as a developer. Martin Fowler has this famous quote ‘Imperfect tests run frequently, are much better than perfect tests that are never written at all’ and there’s this sense that developers will push tests out if they can and that never get the baby washed. So it doesn’t matter how simplistic it is, at least it is some testing. You’ve got much more knowledge about your exposure. 

Whereas if you don’t have enough testing, you have no idea where the bugs are, you have no idea where problems might be. We have a product called Cover that allows you to generate tests quickly, and you can do that both as the development process within IDE such as IntelliJ but also in the context of DevOps. We can plug in to your development pipeline to generate tests in a batch mode and do that on an ongoing basis as code changes overtime. 

The key here is to try and augment the developer. We’re never going to replace the development task, and obviously developers know how their code should work. There needs to be some intelligence there from the developer. There’s also a huge amount of stuff you need to get coverage of your code base, you’ve got to verify that, and not even verify stuff is working, just verify what you have. So if things change and what you have is broken in some way, maybe you’ve improved things and made things work. 

The worst scenario is where developers are making code changes but you have no idea of how to measure if that’s making things better or worse. Or even customers finding issues that are associated with those things. AI has a massive part to play here and I think everyone recognizes that they’ve got to get better at the testing part of development but also the testing part of maintenance of existing products. 

Andy, it’s been absolutely superb to hear about your expertise and inside around DevOps adoption and the inclusion of AI and why it’s so critical around that. For the listeners and viewers at home if you have any further questions you can find further information on our website www.diffblue.com. Again, thank you very much Andy.