Clever Geek Handbook

Learning mutmut - a Python mutation testing tool

Mutation testing allows you to identify bugs that are not covered by conventional tests.



Do you have tests for all occasions? Or maybe your project repository even contains the "About 100% Test Coverage" help? But is everything so simple and achievable in real life?







With unit tests, everything is more or less clear: they must be written. Sometimes they do not work as expected: there are false positives or tests with errors that will return "yes" and "no" without any code changes. The small bugs you can find in unit tests are valuable, but often the developer will fix them before they commit. But we are really worried about those mistakes that often slip out of sight. And worst of all, they often decide to make a name for themselves just when the product falls into the hands of the user.



It is mutation testingallows you to deal with such insidious bugs. It modifies the source code in a predetermined way (introducing special bugs - the so-called "mutants") and checks if these mutants survive in other tests. Any mutant that survived the unit test leads to the conclusion that the standard tests did not find the corresponding modified piece of code that contains the error.



In Python, mutmut is the primary tool for mutation testing .



Imagine we need to write some code that calculates the angle between the hour and minute hands in an analog clock: 



def hours_hand(hour, minutes):
    base = (hour % 12 ) * (360 // 12)
    correction = int((minutes / 60) * (360 // 12))
    return base + correction

def minutes_hand(hour, minutes):
    return minutes * (360 // 60)

def between(hour, minutes):
    return abs(hours_hand(hour, minutes) - minutes_hand(hour, minutes))


Let's write a basic unit test: 



import angle

def test_twelve():
    assert angle.between(12, 00) == 0


There are no ifs in the code . Let's check to what extent such a unit test covers all possible situations: 



$ coverage run `which pytest`
============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
rootdir: /home/moshez/src/mut-mut-test
collected 1 item                                                              

tests/test_angle.py .                                                    [100%]

============================== 1 passed in 0.01s ===============================


Excellent! Like 100% coverage. But what happens when we do mutation testing?







Oh no! Of the 21, as many as 16 mutants survived. How so?



For each mutation test, a portion of the source code needs to be modified that simulates a potential error. An example of such a modification is changing the comparison operator ">" to "> =". If there is no unit test for this boundary condition, this mutant bug will survive: this is a potential bug that none of the usual tests will detect.



Okay. All clear. We need to write better unit tests. Then, using the results command, let's see what specific changes were made: 



$ mutmut results
<snip>
Survived :( (16)

---- angle.py (16) ----

4-7, 9-14, 16-21
$ mutmut apply 4
$ git diff
diff --git a/angle.py b/angle.py
index b5dca41..3939353 100644
--- a/angle.py
+++ b/angle.py
@@ -1,6 +1,6 @@
 def hours_hand(hour, minutes):
     hour = hour % 12
-    base = hour * (360 // 12)
+    base = hour / (360 // 12)
     correction = int((minutes / 60) * (360 // 12))
     return base + correction


This is a typical example of how mumut works: it analyzes the source code and replaces some operators with others: for example, addition by subtraction or, as in this case, multiplication by division. Unit tests, generally speaking, should catch errors when changing statements; otherwise, they do not effectively test the program's behavior. This is the logic mutmut adheres to when making certain changes.



We can use the mutmut apply command on the surviving mutant. Wow, it turns out we didn't check if the hour parameter was used correctly. Let's fix this: 



$ git diff
diff --git a/tests/test_angle.py b/tests/test_angle.py
index f51d43a..1a2e4df 100644
--- a/tests/test_angle.py
+++ b/tests/test_angle.py
@@ -2,3 +2,6 @@ import angle
 
 def test_twelve():
     assert angle.between(12, 00) == 0
+
+def test_three():
+    assert angle.between(3, 00) == 90


Previously, we only checked for 12. Adding a check for the value 3 would save the day?







This new test has managed to kill two mutants: it's better than before, but more work needs to be done. I won't write a solution for each of the 14 remaining cases right now, because the idea is already clear (can you kill all mutants yourself?)



In addition to measuring coverage, mutation testing also allows you to evaluate how comprehensive your tests are. This way you can improve your tests: any of the surviving mutants is a mistake that the developer may have made, as well as a potential defect in your product. So, I wish you to kill more mutants!





Advertising



VDSina offers virtual servers for Linux and Windows - choose one of the pre-installed OS, or install from your image.





More articles:


All Articles