An Analysis and Comparison of Mutation Testing Tools for Python
Board Location: #186
Discipline: Computer Sciences and Information Management
Session: 2
Kadiatou Diallo - Lawson State Community College
Co-Author(s): Dr. Eric Wong, University of Texas at Dallas, Richardson, Texas, United States Zhizhao Chang, University of Texas at Dallas, Richardson, Texas, United States
Software testing is a crucial phase in the software development lifecycle, yet it often becomes a challenging task for engineers who must ensure comprehensive test coverage. While python unit testing frameworks like pytest and unittest are widely used to validate code functionality, passing these tests does not necessarily imply meaningful or effective testing. Mutation Testing addresses this gap by introducing deliberate faults, known as mutants, into the program under test to assess the quality of its test suite. This study evaluates five Python-based mutation testing tools: MutPy, Mutmut, Mutatest, Poodle, and Cosmic Ray, by applying them to two different open-source programs. The tools performances were compared using mutation-specific and tool-specific criteria to determine their strengths, limitations, and analyze which tool created the most competent mutants. In the end, Poodle created the most competent mutants with a 50.9% competency score, Cosmic Ray came next with 25.7% competency score, and Mutmut was inconclusive due to incomplete mutation results. Mutpy and Mutatest unfortunately do not function therefore was not able to be included in the mutation trials. In the future, we hope that developers consider our findings and address the limitations identified in this paper. We also plan to extend our research by testing larger suites, exploring a wider range of mutation operators, and using diverse test case scenarios to gain deeper insights into the reliability of mutation testing tools in Python. I would like to thank the STQA Lab at UT Dallas, the Lawson State STEM Program, Lawson State Community College, and Professor Scott Kelley for supporting me through my research!
Faculty Advisor: Dr. Eric Wong, ewong@utdallas.edu
Role: I developed the topic, implemented testing methods, and produced the results/findings.