Build Monorepos, not Monoliths

This article was written as part of the C# Advent 2021 event. Find more articles and information here: https://csadvent.christmas/

Monolithic Software

C# has a long history with monoliths. They have their own MSDN page, which describes them as below.

A monolithic app has all or most of its functionality within a single process or container and it's componentized in internal layers or libraries

On MSDN, some downsides are immediately noted.

Scaling the individual components is either hard, or impossible in some cases. As an example, you may be able to scale up the database as a whole, but you would not be able to easily bump up the authentication layer of your application.

Changing a single layer of the application requires the whole app to be rebuilt and tested.

The whole app must be deployed at the same time.

Microservices

In response to these issues, many developers and architects have been building microservices. In a microservices architecture, each layer or vertical slice of the application is written and deployed as its own executable or container. Each service communicates through protocols like gRPC or HTTP, rather than directly calling functions.

This eliminates many of the scaling concerns, as each individual host for the services can be scaled independently. Testing can also be slimmed down. As long as the public API for a service is not changed, or the changes are backwards compatible, only that service must be tested.

However, some benefits from monoliths are lost. Sharing code between microservices is much harder. If you have a common "microservice-template" repository, updating it becomes a chore. Existing microservices may not get the benefits from the update, or you may have to manually copy and paste the functionality to each microservice. You may decide to publish a company internal package to nuget, or NPM to alleviate some of that pain, but now testing your changes across the services can result in going back and forth between solutions to check if your changes work.

Additionally, layers of complexity quickly stack up and progress can slow down to a halt if the architecture begins to slip.

Some big cons from microservices that are split across multiple repositories include:

Implementing one feature may have a developer submitting pull requests to several repositories. These PR's may require each other for integration tests to pass.

Deployment and CI/CD work is much more complicated, as integration tests should test the whole system but are harder to orchestrate

Modern Monorepos

The front-end community has began to adopt the concept of monorepos, and many top companies have been using them for years.

In its most basic form, a C# monorepo may be a git repository that contains all of the microservices inside a solution. Even at this basic level (which I refer to as colocation), you start getting some benefits.

A single pull request represents the entire change set for a feature.

System level integration tests and CI checks can be implemented much easier, since you are only dealing with one source repository.

Beyond Co-location

Introducing something like @nrwl_io's Nx build tooling allows for many more benefits to come through, with advanced task orchestration to maximize developer productivity and minimize CI times.

There are some similar tools available, but I'm most familiar with Nx so that is the example I will use here. For a comparison with other tools, see monorepo.tools.

After adopting Nx, and using its commands for your builds you get several things, but we will focus on a few of them.

Local and Remote caching for build / test targets

Project Graph Analysis

Distributed Task Execution

Local and Remote Cache

When using Nx, commands such as build and test have their outputs (both on disk and terminal) cached during execution. What this means, is that when running the same command with the same source files you don't waste computation or developer time.

With the help of Nx Cloud, this time saved gets extended on your CI/CD agents as well. Outputs get cached to the cloud servers, and if an agent runs a command which has already been ran it retrieves it from the cache instead of running the command.

This remote cache can also help developers, as when checking out a PR branch locally the builds will pull from the remote cache that was created during PR checks or on the other developers machine.

Project Graph Analysis

With the help of nx-dotnet, Nx is able to understand how the projects in your repository depend on each other. Nx is then able to look at what has changed on your branch and only build and retest what has been affected by your changes. When using microservices you may not have ProjectReferences in your .csproj files. Nx makes it easy to add an "implicit dependency" between projects to inform the tooling that there is a hidden link there.

Taking advantage of nx affected in CI / CD means that you are not wasting cloud agent time to retest things that haven't changed, but you are not missing tests that may fail in a dependent project.

Distributed Task Execution

Nx can intelligently run targets across a network of CI agents, such that dependent jobs are ran with the inputs they need as soon as they are ready. This can speed up CI/CD times by a large factor, as jobs are parallelized and computational time is not wasted redoing jobs that have already been completed.

There is an existing article describing how this process works from @nrwl_io.