tl;dr: Gartner recently published Hype Cycle for AI in Software Engineering 2024 which evaluates 31(!) AI technologies. AI-Augmented Testing is one of the three most mature innovations, and Diffblue is recognized as a Sample Vendor in that category. Want your copy of the report? Get it here.
A core (but not novel) part of my thesis about joining Diffblue centred on the AI-driven metamorphosis that the entire software development lifecycle (SDLC) is undergoing. With market disruptions come fresh opportunities for new entrants to focus, innovate, and execute with speed and rigor to become new leaders in their chosen segments.
We believe the new Gartner Hype Cycle for AI in Software Engineering 2024 offers a thorough overview of many AI technologies that are changing (or will or could change) almost every facet of the SDLC – 31 of them to be exact. Each technology is assessed on three dimensions…
- Benefit (low, moderate, high, transformational)
- Market penetration (% of target market using the technology), and
- Maturity (embryonic, emerging, adolescent, mature)
… and is also plotted on the renowned Gartner Hype Cycle graph.
AI-augmented testing is one of the 31 technologies covered in the report, and Diffblue is noted as one of the companies with products in this category. The report goes on to describe the business impact, drivers, and obstacles for the development of the market before making a handful of targeted recommendations, my favorite of which is:
Aside from something so obviously self-serving, I found three things about Gartner observations on AI-augmented testing particularly striking:
First, it’s the third most mature of the 31 AI technologies covered in the report. Additionally, it’s forecast to reach the fabled “Plateau of Productivity” as quickly as any of the technologies covered (2 – 5 years), except for one (retrieval augmented generation). And the overall “Benefit Rating” is deemed by Gartner to be high.
Second, despite AI-augmented testing’s relative maturity, its market penetration is still low – only 5 – 20% of the target audience according to the report. We feel this suggests that there’s massive growth potential for effective products serving this market need.
Third, the report cites limitations with LLMs as an obstacle to broader market acceptance of AI-augmented testing. To quote:
“Currently available tools are still relatively new, have a narrow scope and still need to prove their value. Large language model (LLM)-based generative AI, in particular, is the latest and most disruptive example. Hallucinations (content that is nonsensical or untruthful in relation to certain sources), subpar training data, potential copyright violations and security issues are the main risks associated with LLM-based AI technologies.”
We believe this perspective aligns well with Diffblue’s perspective on LLMs when it comes to test creation. While it is an incredibly powerful technology with broad applications, LLMs also have some significant limitations which mean that they are unlikely to be the ideal solution for many companies, particularly ones with stringent security and auditability needs.
That’s why Diffblue’s different form of generative AI – reinforcement learning – has proven popular with heavily regulated, security conscious customers like major banks, pharmas, and insurers. This is further exemplified in a recent independent comparison of Diffblue vs. an LLM-based approach. Eventually, a hybrid approach that combines the best of different AI technologies will likely be a good choice for many customers’ testing needs, but such solutions are not yet on the market.
To learn more, access your Gartner Hype Cycle for AI in Software Engineering 2024 with our compliments. And if you’re ready to follow Gartner recommendation about starting to evaluate AI-augmented testing tools, Diffblue’s free two-week trial of our platform would be my suggestion. Get it here.
