Mutation testing: are your tests any good?

Who it's for

Engineers whose agents write the tests as well as the code.

You'll be able to

Find the tests that run your code but would not notice if it were wrong.

Coverage tells you a line ran during a test. It does not tell you a test would notice if that line were wrong.

Tests can pass without checking much

When an agent writes both the code and the tests, the tests can look fine and still check very little. A test that calls a function and only checks that something came back covers every line of it. Break the function, and that test still passes.

Mutation testing asks a sharper question. If the code were wrong, would any test fail?

How it works

A mutation tool changes your code in small ways, one change at a time. A + becomes a -, for example. After each change it runs your tests.

If a test fails, the change was caught. Good.
If every test still passes, the change "survived". The tool lists it.

Each survivor is a place where your code could be wrong and your tests would not say so.

A survivor is a change to your code that no test noticed.

A small example

def total(price, tax_rate):
    return price + price * tax_rate

# a mutant the tool might try:
#   return price - price * tax_rate

A test that only checks that total returns a number lets this mutant survive. A test that checks total(100, 0.2) == 120 catches it. The usual answer to a survivor: write the test that would fail on it.

The function

def total(price, tax_rate):
    return price + price * tax_rate

The test

Calls total(100, 0.2) and checks only that a number comes back.

Coverage: this test runs every line, so coverage calls each line run, for every mutant below.

Mutants, one at a time

return price - price * tax_rate+ becomes -returns 80, still a number Survived
return price + price / tax_rate* becomes /returns 600, still a number Survived
return price + price + tax_rate* becomes +returns 200.2, still a number Survived
return Nonethe result becomes Nonereturns None, not a number Caught

Each survivor needs a test that would fail on it. This one catches all three: total(100, 0.2) == 120

A test that only checks for a number runs every line, and still lets three of four mutants survive.

Where the factory points it

In the factory, mutation testing runs on the logic: the service code that does the real work. By the factory's own rule, routes hold no logic. The Python tool used is mutmut, started with one command:

make mutation

The tool changes only code inside functions. If a function exists but no test runs it, the tool stops and says so. The answer there is also to write the test.

It reports, and it never blocks. It is part of no check in the gate. It runs once a week and whenever someone starts it by hand.

Try it

In your own repository, without any tool: