How to set up monorepo build in GitLab CI

If you migrate your multirepo to a monorepo, or if your project is getting big enough to consider running only part of continuous integration (CI) - then it can make sense to run only those parts of CI that could have been affected by the change. This article will show how to achieve it on the GitLab platform, using a simple repository as an example.

The approach

The project is split into folders. For distinguishing if a given part of the project was modified, we use rules:changes so every part of the project that we want to be able to run in separation from the rest should be placed in one folder.

My assumptions are as follow:

you want to run only changed sections of CI in the merge requests (MR) - mainly to save CI resources. So we can avoid running whole 30 minutes of jobs for a change we know is not likely to affect them. This is especially important if we have code in the repo that is not depending on each other - for example, our main application in one place and some landing pages in other folders.

after changes are merged to master/main, we want to build everything no matter if it was changed or not. In this way, our main branch is indeed continuously integrated, and we keep on checking on even less commonly changed parts of the project.

Configuration

My project structure is simple:

$ git ls-files
.gitlab-ci.yml
README.md
backend/README.md
frontend/README.md

I have 2 folders, backend & frontend. Each would host files of a given part of our project. This approach scales for any number of sub-projects - we could have company-website, slack-bot, or whatnot inside.

.gitlab-ci.yml step by step

The configuration starts with defining the stages:

stages:
  - build
  - test
  - deploy

This one is copied from GitLab's starting CI template. We can customize it with adding or removing stages. As we define needs:, there is no speed penalty for adding more stages - each job is executed as soon as its requirements are defined in needs: are met. For example, in my project, I ended up adding pre-build to run some preparation scripts before building docker images in my project.

variables:
  RULES_CHANGES_PATH: "**/*"

The default value for our changes configuration - by default, the job that extends our base config will be executed for any changes.

.base-rules:
  rules:
    ...

Our base config. I define it in a way that requires us to add it with extends: .base-rule - probably we could define those rules on the top level, but it's something a headache to configure everything in a way that works as expected in every case. I found it easier to have control over if the .base-rules are set or not.

.base-rules:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always
  ...

The rules are checked in order. Our 1 rule - if it's master/main, CI should always run the job.

- if: '$CI_PIPELINE_SOURCE == "push"'
      when: never

Here, we avoid duplicated jobs for merge requests. Without, GitLab would create 1 pipeline for the branch and a "detached pipeline" for the MR. As the branch pipeline doesn't seem to support changes:, we disable branch one & delay starting CI until an MR is created.

- if: $CI_COMMIT_TAG
      when: never

Similarly, we don't need CI running for a tag.

- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - $RULES_CHANGES_PATH

In MRs, we start jobs for when there are changes in the path defined in the RULES_CHANGES_PATH variable.

- when: manual
      allow_failure: true

Otherwise, we all the job to be triggered manually.

.frontend & .backend

Now, we define 2 more jobs to be extended from:

.backend:
  extends: .base-rules
  variables:
    RULES_CHANGES_PATH: "backend/**/*"

.frontend:
  extends: .base-rules
  variables:
    RULES_CHANGES_PATH: "frontend/**/*"

In this way, we avoid duplicating the same path definition in each job we define for one or the other part - a possible source of errors.

Example jobs

On top of that all, we can define our jobs as:

backend-build:
  stage: build
  extends: .backend
  needs: []
  script:
    - echo "Compiling the backend code..."

frontend-build:
  stage: build
  extends: .frontend
  needs: []
  script:
    - echo "Compiling the frontend code..."

Complete .gitlab-ci.yml

So, in the end, the complete config files are like this:

stages:
  - build
  - test
  - deploy

variables:
  RULES_CHANGES_PATH: "**/*"

.base-rules:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always
    - if: '$CI_PIPELINE_SOURCE == "push"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - $RULES_CHANGES_PATH
    - when: manual
      allow_failure: true

.backend:
  extends: .base-rules
  variables:
    RULES_CHANGES_PATH: "backend/**/*"

.frontend:
  extends: .base-rules
  variables:
    RULES_CHANGES_PATH: "frontend/**/*"

backend-build:
  stage: build
  extends: .backend
  needs: []
  script:
    - echo "Compiling the backend code..."

frontend-build:
  stage: build
  extends: .frontend
  needs: []
  script:
    - echo "Compiling the frontend code..."

backend-test:
  stage: test
  extends: .backend
  needs: ["backend-build"]
  script:
    - echo "Testing the backend code..."

frontend-test:
  stage: test
  extends: .frontend
  needs: ["frontend-build"]
  script:
    - echo "Testing the frontend code..."

backend-deploy:
  stage: deploy
  extends: .backend
  needs: ["backend-test"]
  script:
    - echo "Deploying the backend code..."

frontend-deploy:
  stage: deploy
  extends: .frontend
  needs: ["frontend-test"]
  script:
    - echo "Deploying the frontend code..."

Working CI

With a setup like this, you can have your backend CI run for backend MR:

frontend, for MR with frontend changes:

and each commit will trigger all CI jobs once it's merged to the main branch:

Refrences

You can find the repo I used to write this article here.

Summary

In this article, we have seen how to set up partially split CI for branches in GitLab. Please leave a comment if you find this article helpful or have some questions about this approach.

Tags:

Gitlab Monorepo

How to create commit automatically in a merge request in GitLab

How to build Docker images in GitLab CI

Gitlab CI/CD for npm packages

GitLab 14.0 is now available! 🌻

Setup CI pipeline for iOS projects on gitlab.com

How to set up Terraform on GitLab

Exploring the Monorepo #4: Adding Docker + CI

Installing gitlab on kubernetes with kustomize

How to set up monorepo build in GitLab CI

Tags:

Related Posts