Adding infection to an existing project and publishing the results to Gitlab merge requests

In this blog post I want to describe how we recently added infection to our development workflow in an existing project. The project has a code coverage of ~40% and we have ~27k LLOC (measured with phploc src | grep LLOC). If you're not familiar with infection or mutation testing in general yet, I can recommend this blog post.

Adding infection to a greenfield project or some small module is not that hard: You can just add it to your CI pipeline together with a hard-limit via the --min-msi & --min-covered-msi flags and raise the bar from time to time, until you reach 100%.

When adding infection to an existing project it's a bit different and it was in fact what was hindering us from using that: How could we really make it part of our development process without (a) having a too strict check and (b) without risking that the reported problems will just be ignored, because there are obviuosly so many of them in the beginning?

We decided to try out a pragramtic solution: We run infection only for changed files in a merge request and publish the results directly in the merge request as a note, so that the author and the reviewers see all mutations.

Adding infection to the project

We added infection to the docker image we use in CI like so:

RUN wget https://github.com/infection/infection/releases/download/0.25.0/infection.phar \
    && wget https://github.com/infection/infection/releases/download/0.25.0/infection.phar.asc \
    && chmod +x infection.phar \
    && mv infection.phar /usr/local/bin/infection

Have a look at different ways to install infection here

Our infection.json is pretty basic, but it's important to have the different outputs in the logs section configured:

{
  "$schema": "https://raw.githubusercontent.com/infection/infection/0.25.0/resources/schema.json",
  "source": {
    "directories": [
      "src/Pyz"
    ],
    "excludes": [
      "DependencyProvider.php",
      "BusinessFactory.php"
    ]
  },
  "logs": {
    "text": "infection.log",
    "summary": "summary.log",
    "perMutator": "per-mutator.md"
  },
  "mutators": {
    "@default": true
  }
}

Your source directory is probably just src/ and you will need to adapt the excludes as well.

Infection allows you to limit which files it should mutate with the --git-diff-filter option. We use that like so in our CI:

- git fetch --depth=1 origin master
- infection --git-diff-filter=AM || true

This will execute infection in the pipeline and write the output also to the configured log files (infection.log, summary.log & per-mutator.md):

Publishing the results as a comment in the merge request

To make it a bit easier for the author and the reviewers, we added a php script that publishes these results as a comment directly in the merge request.

You need a Project Access Token in order to use the Gitlab API for stuff like this. You can create one at https://<your-gitlab-domain>/<group>/<project>/-/settings/access_tokens. Copy the token to a safe place and make sure that it's available as a environment variable during the CI job.

We've choosen this GitLab PHP API Client for our project:

composer require --dev "m4tthumphrey/php-gitlab-api:^11.4" "guzzlehttp/guzzle:^7.2" "http-interop/http-factory-guzzle:^1.0"

Now here's the script that reads the log files and sends them to the related merge request:

<?php

declare(strict_types = 1);

require_once 'vendor/autoload.php';

$projectId = (int)getenv('CI_PROJECT_ID');
if ($projectId === 0) {
    die('CI_PROJECT_ID is missing!');
}
$mergeRequestId = (int)getenv('CI_MERGE_REQUEST_IID');
if ($mergeRequestId === 0) {
    die('CI_MERGE_REQUEST_IID missing!');
}
$authToken = (string)getenv('GITLAB_AUTH_TOKEN');
if ($authToken === '') {
    die('GITLAB_AUTH_TOKEN missing!');
}

$client = new Gitlab\Client();
$client->setUrl('<your-gitlab-domain>');
$client->authenticate($authToken, Gitlab\Client::AUTH_HTTP_TOKEN);
if (!function_exists('str_starts_with')) {
    function str_starts_with(string $haystack, string $needle): bool
    {
        return strpos($haystack, $needle) === 0;
    }
}

/* ---------------------------------------------- */

$data = json_decode(file_get_contents('infection-log.json'), true);
$identifierText = '<details><summary>infection results 📋</summary>';
$noteBody = $identifierText . PHP_EOL . PHP_EOL;

$noteBody .= '| metric | value |' . PHP_EOL;
$noteBody .= '| ------ | ----- |' . PHP_EOL;
$noteBody .= '| Mutation Score Indicator (MSI) | ' . $data['stats']['msi'] . '% |' . PHP_EOL;
$noteBody .= '| Mutation Code Coverage | ' . $data['stats']['mutationCodeCoverage'] . '% |' . PHP_EOL;
$noteBody .= '| Covered Code MSI | ' . $data['stats']['coveredCodeMsi'] . '% |' . PHP_EOL;

$noteBody .= '```

' . file_get_contents('summary.log') . '

```' . PHP_EOL;
$noteBody .= '```

' . file_get_contents('infection.log') . '

```' . PHP_EOL;
$noteBody .= file_get_contents('per-mutator.md') . PHP_EOL;

$noteBody .= PHP_EOL . '</details>';

/* ---------------------------------------------- */

$notes = $client->mergeRequests()->showNotes($projectId, $mergeRequestId);
foreach ($notes as $note) {
    if (str_starts_with($note['body'], $identifierText)) {
        $noteId = $note['id'];
    }
}
if (isset($noteId)) {
    $notes = $client->mergeRequests()->updateNote($projectId, $mergeRequestId, $noteId, $noteBody);
    echo 'updated';
} else {
    $notes = $client->mergeRequests()->addNote($projectId, $mergeRequestId, $noteBody);
    echo 'added';
}

echo PHP_EOL;

The variables CI_PROJECT_ID and CI_MERGE_REQUEST_IID are automatically available in every Gitlab CI job, you just need to make sure that the auth token you generated is available in GITLAB_AUTH_TOKEN. Do not forget to replace <your-gitlab-domain> with the real url.

The script adds only one note per merge request, which will be updated if new results are generated. The comment will look like this:

published results as a comment in the merge request

You can use this of course for all kinds of reports, I think we'll add davidrjonas/composer-lock-diff next.

This setup allows us to see escaped mutants in every merge request and improve our codebase step-by-step. I hope you enjoyed it, feel free to ask any questions here or on twitter.