AI-driven software testing gains more champions but worries persist

da-kuk/Getty Images

Comprehensive quality engineering and testing are a must for today's software-driven organizations. Perhaps not surprisingly, generative artificial intelligence (Gen AI) is emerging as a cutting-edge component of the quality and testing phase of the software development lifecycle. 

However, long-term success in software-testing automation is about establishing the necessary organizational will and resources. In short, to paraphrase management guru Peter Drucker's oft-cited phrase: Culture eats software-quality strategies for breakfast.

Also: The data suggests gen AI boosts software productivity - for these developers

"The debate on which quality engineering and testing activities will benefit most from Gen AI remains unresolved," said the co-authors of an OpenText study involving 1,755 tech executives state. The survey, released by Capgemini and Sogeti (part of the Capgemini Group), pointed to a growing focus on leveraging Gen AI "for test reporting and data generation over test-case creation."

AI creates an answer, or at least a partial answer, to many nagging software quality issues. Software quality has been a challenge since the first computers were built eight decades ago, and in a world awash in technology networks and solutions, the problem has only grown more acute. Gen AI is emerging as an important step in managing quality.

Also: 6 ways to write better ChatGPT prompts - and get the results you want faster

The survey confirmed about seven in ten organizations (68%) employ Gen AI to assist with their software quality efforts. At least 29% of organizations have fully integrated Gen AI into their test automation processes, while 42% are exploring its potential.

The study also suggested that "cloud-native technologies and robotic process automation, with Gen AI and predictive AI both playing significant roles" are prevalent in this new area of test automation.

"Cloud-native technologies are appealing because they open the door to cost-effective solutions that eliminate the need for tooling licenses, which lowers overall operational expenses. It is no longer a question of 'if' AI and other emerging technologies will become a part of the DevOps fabric. We are in the early stages of a dynamic shift in the way we do business."  

Also: Google's new AI course will teach you to write more effective prompts - in 5 steps

The conclusion is that AI represents the next stage of automation for relatively complex quality assurance and testing processes. 

"There is a clear need to align quality engineering metrics with business outcomes and showcase the strategic value of quality initiatives to drive meaningful change," the survey's team of authors, led by Jeff Spevacek of OpenText, stated. 

"On the technology front, the adoption of newer, smarter test automation tools has driven the average level of test automation to 44%. However, the most transformative trend this year is the rapid adoption of AI, particularly Gen AI, which is set to make a huge impact."

Spevacek and his co-authors continued: "The evolution of large language models and AI tools, particularly Copilot, have enabled their seamless integration into existing software development lifecycles, ushering in a new wave of efficiency and innovation in quality engineering automation." 

In the previous year's software quality survey, "we saw an uptick in the investments made by organizations in AI solutions to drive the quality-transformation agenda," they wrote. "However, a significant number were skeptical about the value of AI in quality engineering."

Also: Think AI can solve all your business problems? Apple's new study shows otherwise

Attitudes toward AI have shifted significantly over the past 12 months: "A large number of organizations are now moving [away] from experimenting to real-scale implementation of Gen AI to support quality engineering activities. We truly believe we will see further advancements in this area."

However, employing AI as a software quality assurance tool is challenging. At least 61% of survey respondents said they worry about data breaches associated with leveraging generative AI solutions. A lack of comprehensive test automation strategies and a reliance on legacy systems were identified by 57% and 64% of respondents, respectively, as key barriers to advancing automation efforts.

The picture is also mixed for embedding quality engineers with Agile software delivery teams. Only one-third of respondents said most of their quality engineers participate in Agile teams. However, the authors suggested this lack of participation might not be a bad thing. 

"This suggests a growing recognition of the need for quality engineers who can operate independently of Agile teams, while still contributing to overall quality objectives. In fact, the number of standalone quality engineers is expected to increase from 27% to 38%." 

Also: Google survey says more than 75% of developers rely on AI. But there's a catch

The survey suggested this increase in high-quality engineers may also reflect a trend of cross-skilling of Agile teams to address software quality and testing: "The focus on cross-skilling to align quality engineers more closely with Agile teams appears to have paid off. This year's survey results show that organizations have made considerable progress in upskilling their teams -- only 16% of respondents now view a lack of skills as a major bottleneck, a significant improvement from last year's 37%."

However, despite this progress, most tech executives said there isn't enough emphasis on quality engineering. More than half (56%) said the challenge is that "quality engineering is not seen as a strategic activity in our organization." A similar proportion of respondents agreed that the "quality engineering process is not automated enough," and that "quality engineers lack the skillset to support Agile projects."

Also: Agile development can unlock the power of generative AI - here's how

The rise of Gen AI and predictive AI may offer a cost-effective and streamlined approach to aligning quality and testing efforts with overall software development and deployment. Some of the recommendations offered by the OpenText/Sogeti team for moving forward with automation and AI in software quality efforts included the following:

Take an enterprise-wide view: Clearly outline "the objectives and desired outcomes of quality engineering automation and pre-selecting the areas where to apply, increase or enhance test automation."Start now and keep experimenting: "If you are not yet exploring or actively using Gen AI solutions, it's crucial to begin now to stay competitive. Don't rush to commit to a single platform or use case. Instead, experiment with multiple approaches to identify the ones that provide the most significant benefits."  Leverage Gen AI's full range of capabilities: "Gen AI goes far beyond the generation of automated test scripts and helps with the realization of self-adaptive test automation systems."Tie in business key performance indicators: "Identify and leverage key business performance indicators influenced by quality engineering automation, with a clear focus on business outcomes, such as increased customer satisfaction, reduced cost of business operations, and others which are relevant to the business."Rationalize quality engineering automation tools: "Ensure that your quality engineering automation tools are streamlined and capable of integrating with emerging technologies, such as Gen AI, to maintain compatibility and future readiness."Enhance quality engineering talent and roles: "Incorporate more full-stack quality and software development engineers in test to strengthen your team's capabilities."Enhance, don't replace: "Understand that Gen AI will not replace your quality engineers but will significantly enhance their productivity. However, these improvements will not be immediate; allow sufficient time for the benefits to become apparent."

While AI offers great promise as a quality and testing tool, the study said there are "significant challenges in validating protocols, AI models, and the complexity of validation of all integrations. Currently, many organizations are struggling to implement comprehensive test strategies that ensure optimized coverage of critical areas. However, looking ahead, there is a strong expectation that AI will play a pivotal role in addressing these challenges and enhancing the effectiveness of testing activities in this domain."

The key takeaway point from the research is that software quality engineering is rapidly evolving: "Once defined as testing human-written software, it has now evolved with AI-generated code." 

Also: Could AI make data science obsolete?

As a result of this evolution, quality engineering is seeing an increased volume of code and test scripts that need to be generated, and there are new requirements for testing software chains from end to end.