17
loading...
An excellent way to deal with (lerna) monorepos in CircleCI: circletron
We started with a monorepo at my current company and have been using circle almost since the beginning. It was tough. It required a lot of boilerplate in our config and the necessity to get every developer to generate a circle API key and add it to git. It never felt that great but at least it worked. In April CircleCI released the dynamic configuration API and this allowed us to refactor our monorepo support into something we think is pretty great. We've gone from tolerating Circle to enjoying it, and now we've released the code as an open source project and provided the functionality as an orb so that any project can benefit.
The project is circletron, the code is hosted on github and the orb is here. It currently supports lerna monorepos but we plan to support other monorepos in the future.
Typically a circleci configuration exists in a single file within a repository at
.circleci/config.yml. For a monorepo a very minimal example may look something like this:version: 2.1
jobs:
validate-everything:
steps:
- checkout
- run: npm run general-validation
test-subpackage-a:
steps:
- checkout
- run: cd packages/package-a && npm run test
test-subpackage-b:
steps:
- checkout
- run: cd packages/package-b && npm run test
test-subpackage-c:
steps:
- checkout
- run: cd packages/package-c && npm run test
publish-subpackage-c:
steps:
- checkout
- run: cd packages/package-c && npm run publish
workflows:
validate-everything:
jobs:
- validate-everything
subpackage-a:
jobs:
- test-subpackage-a
subpackage-b:
jobs:
- test-subpackage-b
subpackage-c:
jobs:
- test-subpackage-c
- publish-subpackage-cObviously there are many problems with this, for a start all the CI is defined in one file at the root of the project. Worse, each of the jobs may take some time to complete and on every commit to the project, circleci will run every single job no matter whether there are any changes to the respective subpackages or not. It may not look so bad in this tiny example but the bigger the configuration gets, the worse it is to deal with.
With
circletron the .circle/config.yml is always the same:version: 2.1
setup: true
orbs:
circletron: circletron/[email protected]
workflows:
trigger-jobs:
jobs:
- circletron/trigger-jobsThe trigger job step will take many individual
circle.yml distributed within the project and combine them into a single configuration which will be issued via the continuation API. It will modify the configuration to ensure that jobs that are not necessary are no longer run, in a way that is friendly to CI branch protection rules.The single configuration file can now be split up across the monorepo, with one optional
circle.yml in the root of the project and where the CI for each subpackage lives in the directory for that subpackage.In this instance the
circle.yml in the root of the project will host configuration not specific to any subpackage:version: 2.1
jobs:
validate-everything:
steps:
- checkout
- run: npm run general-validation
workflows:
validate-everything:
jobs:
- validate-everythingThe jobs here are run on every commit and it's also a good place to set
version. It's also a good place to provide commands that can be used in subpackage specific circle.yml files.Now lets look at
packages/package-a/circle.yml:jobs:
test-subpackage-a:
steps:
- checkout
- run: cd packages/package-a && npm run test
workflows:
subpackage-a:
jobs:
- test-subpackage-aEverything related to this package all in one place. Even better, when the PR contains no changes within
packages/package-a the jobs for this subpackage will be skipped. You will probably want to use branch protection rules to ensure that when the test-subpackage-a job fails the PR will be blocked so omitting this job entirely would not be ideal. Omitted jobs remain in pending state permanently, blocking the PR. circletron replaces unneeded jobs with a simple job using the busybox:stable docker image that echoes "Job not required" and issues a success error code.What if the code in
packages/subpackage-c uses code from packages/subpackage-a? In this case the jobs within this package should run when the code in subpackage-a changes, even if the code in the subpackage itself doesn't change.There may also be some instances where a job should run on every push.
The configuration at
packages/subpackage-c/circle.yml shows how to add dependencies and create jobs which run unconditionally:dependencies:
- subpackage-a
jobs:
test-subpackage-c:
steps:
- checkout
- run: cd packages/package-b && npm run test
publish-subpackage-c:
conditional: false
steps:
- checkout
- run: cd packages/package-c && npm run publish
workflows:
subpackage-c:
jobs:
- test-subpackage-c
- publish-subpackage-aIn this instance
test-subpackage-c will only be run when changes to package-a or package-c are detected and publish-subpackage-c will run on every push.CircleCI does not pass the target branch to workflows or jobs so it becomes necessary to help circletron out. By default circletron considers the branches
main, master, develop and any branch starting with release/ as a target branch. The latest commit from the branch commit history which belongs to one of these branches is considered to be the branch-point. For pushes to one of the branches above, all of the jobs are run. This can be changed via the configuration file at .circleci/circletron.yml:targetBranches: ^(release/|main$|master$|develop$)Any branch which matches this regex is considered a target branch, the above configuration shows the default.
circletron is available to use now.
17