Killing mutations with Kotlin

by coding flower

What are these mutations? Why do we need them? Do we really need to kill them?

Mutation - application modified by PIT test.

We are creating tests to check that our implementations are working as we want. We are keeping high test coverage, but how can we check that our test really works? Traditional test coverage measures only which code was executed in your tests. It doesn't mean that this test is able to detect errors, it means that this code was used during tests.

How can we deal with this problem? We can run our tests against automatically modified versions of our application code. When the application code changes, it should produce different results and cause the test to fail. If after modification the test doesn't fail it could be broken.

Mutation testing

Running modified versions of applications looks reasonable, but we aren't going to change our code by hand, we are going to use the PIT test.

PIT test is mutation testing system which is compatible with JVM.

Let`s consider example, we are testing method which returns true if value is positive and false if is not.

class Mutations {
    fun isPositive(value: Int): Boolean {
        return value > 0
    }
}

Here is simple Kotest test:

class MutationsTest : ShouldSpec({
    should("return true if value is bigger than zero") {
        val mutation = Mutations()

        mutation.isPositive(20) shouldBe true
        mutation.isPositive(-20) shouldBe false
    }
})

If we run this test in Intellij Idea with code coverage we will see 100% of code coverage and test of course passed.

The mutation report will show us that our test is not perfect.

Our test had three mutations, two of them are dead, but one is still alive, but why? We already had 100% code coverage. The edge case is a problem, the zero number. Mutation modified condition from > to >= . The fix for this situation is very simple, we have to put another test case for zero number, then all mutation will be dead. By default only a few mutations are active, but you can enable more.

Summary

Mutation testing is a quite good way to verify that your tests really detect faults in code instead of just improving code coverage. It's really easy to integrate with kotest framework. The sad thing is this project doesn't have a supportive community, there are almost 300 GitHub issues already. If we have a bigger amount of tests, running mutation testing takes much time, so I don't recommend attaching it to the pipeline. I would say, this tool should be more used when your application failing but your test are green and have high code coverage.

Related Posts

Leave a Comment

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More