Keynote: Mutation Testing in the Trenches

This is a keynote at the Mutation 2025 Workshop, Co-Located with the 2025 International Conference on Software Testing (ICST).

Speaker: Gregory Gay

Based on a Software Center-funded collaboration between Chalmers/GU and Zenseact, reported in these publications:

• https://greg4cr.github.io/pdf/24mutation.pdf
• https://greg4cr.github.io/pdf/23mutationci.pdf

The concept of mutation testing was first proposed over 40 years ago. There exists theoretical evidence that the technique is superior to structural coverage criteria as a means of assessing and improving the quality of an existing test suite. However, due to factors such as computational cost and the lack of availability of mature tools for many programming languages, mutation technique has not caught on as a standard practice in industry. In recent years, there has been significant progress with regard to cost and tool availability, potentially making the adoption of mutation testing more realistic in practice. Rather, we now hypothesize that such adoption is hindered by a lack of guidance on how to effectively and efficiently utilize mutation testing in a development and testing workflow.

In this talk, I will discuss the results of a two year collaboration with Zenseact – a developer of autonomous driving and advanced driver assistance systems – to implement mutation testing in their continuous integration pipeline and to explore how mutation testing can be integrated into the testing process. This collaboration has illustrated the potential of mutation testing in industry, and yielded a number of findings related to the technical integration of mutation testing, how and when mutation testing should be applied, how the results of mutation testing should be presented, and how the results should be applied. However, many research challenges remain to be solved before mutation testing will be used in “standard” practice.

More information is available at https://conf.researchr.org/home/icst-2025/mutation-2025#Keynotes