AWS CodeArtifact - the long-awaited feature

The background

I have to admit, I am not a fan of AWS Codepipeline. Even now, after a few project where I heavily used CodePipeline, I am still not big fan. For me it is rather simple solution. Of course it can do the job, but in my private world, I used to use TravisCI before and now… Azure DevOps for AWS deployments.

And one of great advantages of Azure DevOps is Artifacts, where you can store and publish your packages. Something what we missed in AWS Codepipeline for long time. Until now.

AWS CodeArtifact

Not long time ago AWS announced this service. Now it is official. And it is a great news, indeed. Until today we had to use workarounds (let’s call it this way). S3 buckets, or build the artifact solution by our own.

This is not longer needed. AWS CodeArtifact is taking a great place in the CodePipeline services. First 2G of stored packages is free (monthly), also first 100K requests are free.

Let’s get started

What is important to understand, it is the domain concept. More explanation is in the Steve Roberts’ blog, here https://aws.amazon.com/blogs/aws/software-package-management-with-aws-codeartifact/. Important to remember is that Domain aggregates all repositories. So, we are consuming the package from the repository, or even multiple repositories, but in fact, this package is stored only once across the Domain. This saves space and possible costs.

Domain

So, let’s create the Domain! We can do it from AWS GUI, but this time, I will work with CLI. Of course, I assume you know how to configure CLI, if not, there is plenty of articles how to do it. Remember, you need the newest version of awscli, I recommend to use version 2.

Let’s check our domains now:

aws codeartifact list-domains
{
    "domains": []
}

You should receive empty JSON.

So, it is time to create the first Domain. The process is very easy

aws codeartifact create-domain --domain mytestdomain 
{
    "domain": {
        "name": "mytestdomain",
        "owner": "012345678",
        "arn": "arn:aws:codeartifact:eu-west-1:0123456789:domain/mytestdomain",
        "status": "Active",
        "createdTime": "2020–06–28T10:46:50.193000+02:00",
        "encryptionKey": "arn:aws:kms:eu-west-1:0123456789:key/aaaa1111-aa11–1234–1234-aa11aa11aa11",
        "repositoryCount": 0,
        "assetSizeBytes": 0
    }
}

And… That’s it! Let’s check it again

aws codeartifact list-domains 
{
    "domains": [
    {
        "name": "mytestdomain",
        "owner": "0123456789",
        "status": "Active",
        "encryptionKey": "arn:aws:kms:eu-west-1:0123456789:key/aaaa1111-aa11–1234–1234-aa11aa11aa11"
        }
    ]
}

aws codeartifact describe-domain --domain mytestdomain 
{
    "domain": {
    "name": "mytestdomain",
    "owner": "0123456789",
    "arn": "arn:aws:codeartifact:eu-west-1: 0123456789:domain/mytestdomain",
    "status": "Active",
    "createdTime": "2020–06–28T10:46:50.193000+02:00",
    "encryptionKey": "arn:aws:kms:eu-west-1:0123456789:key/aaaa1111-aa11–1234–1234-aa11aa11aa11",
    "repositoryCount": 0,
    "assetSizeBytes": 0
    }
}

Repository

Domain is already created, so we can create the repository.

But first, we need (again) to look on the construction here. We are in the Domain. Now, we can create Repository, and then Upstream Repository. By this approach you can separate your project repositories (by upstreams) and still be able to access all of them with one endpoint.

So, let’s create the repository first.

aws codeartifact create-repository --domain mytestdomain --repository "mymainrepo" --description "Main repository" 
{
    "repository": 
    {
        "name": "mymainrepo",
        "administratorAccount": "01234567890",
        "domainName": "mytestdomain",
        "domainOwner": "01234567890",
        "arn": "arn:aws:codeartifact:eu-west-1: 01234567890:repository/mytestdomain/mymainrepo",
        "description": "Main repository",
        "upstreams": [],
        "externalConnections": []
    }
}

Of course, we can look on it, using:

aws codeartifact list-repositories

aws codeartifact describe-repository --domain mytestdomain --repository mymainrepo

In order to login to the repository, you can follow the instruction available in the GUI console

The connection command is simple (but long)

aws codeartifact login --tool pip --repository mymainrepo --domain mytestdomain --domain-owner 01234567890

Successfully logged in to codeartifact for pip.

For configuring pip itself, we need to run two additional commands. First to get authorization token:

aws codeartifact get-authorization-token --domain mytestdomain --domain-owner 0123456789 --query authorizationToken --output text

second, to use it in our pip config

pip config set global.index-url [https://aws:HEREGOESTHETOKEN@mytestdomain-0123456789.d.codeartifact.eu-west-1.amazonaws.com/pypi/mymainrepo/simple/](https://aws:HEREGOESTHETOKEN@mytestdomain-0123456789.d.codeartifact.eu-west-1.amazonaws.com/pypi/mymainrepo/simple/)

Writing to C:\Users\pawel\AppData\Roaming\pip\pip.ini

As you can see, I’m using Powershell here.

Now we can check, if our pip is updated, by running pip config list. You should be able to see your token there.

Upstream repository

Let us follow the good practice, and now it is the time to create upstream repositories. We will add npm and pip in one shot,

aws codeartifact update-repository --repository mymainrepo --domain mytestdomain --domain-owner 0123456789 --upstreams repositoryName=pypi-store repositoryName=npm-store
{
    "repository": {
        "name": "mymainrepo",
        "administratorAccount": "0123456789",
        "domainName": "mytestdomain",
        "domainOwner": "0123456789",
        "arn": "arn:aws:codeartifact:eu-west-1:0123456789:repository/mytestdomain/mymainrepo",
        "description": "Main repository",
        "upstreams": [
            {
                "repositoryName": "pypi-store"
            },
            {
                "repositoryName": "npm-store"
            }
        ],
        "externalConnections": []
    }
}

Thanks to that, we are able to work with one endpoint for all of ours npm and pip packages. If it works for you — it is up to you.

Be aware of search order here. If you are requesting the package, your request is going through the upstreams in the create-repository order (or update — repository). If you are curious, what is the order, simply run describe-repository, and you will see upstreams in the search order.

External connections

When we finish our repos, we can create external connections to them. Let’s do it for both of our repos:

For pip:

aws codeartifact associate-external-connection --domain mytestdomain --domain-owner 0123456789 --repository pypi-store --external-connection "public:pypi"

And for npm:

aws codeartifact associate-external-connection --domain mytestdomain --domain-owner 0123456789 --repository npm-store --external-connection “public:npmjs”

Although, it is possible you will receive an error, as this connection already exists.

Let’s test our new shiny repository

So. We are ready to test our repo. We gonna install some pip package. I’ve created empty virtualenv, so we can install for example awscli. What we expect?

Let me go through this schema. We want to install package X. We configured our pip in the way, it is looking to our AWS CodeArtifact first. If the requested package is not there, it will connect to main public one and download it for us (with all dependencies). And then, the package will be installed from our AWS CodeArtifact repo.

Let’s install our package then:

pip install awscli
Looking in indexes: [https://aws:****@mytestdomain-0123456789.d.codeartifact.eu-west-1.amazonaws.com/pypi/mymainrepo/simple/](https://aws:****@mytestdomain-043144402336.d.codeartifact.eu-west-1.amazonaws.com/pypi/mymainrepo/simple/)
Collecting awscli

You easily can spot, pip is trying to collect the package from our repository. Now, the package is installed, let’s check our repository:

And voila! We successfully created, configured and used our CodeArtifact service!

Of course, our packages can be published into our repository. AWS CodeBuild and CodeDeploy can use our repository. In the essence, AWS CodePipeline finally has the proper solution for artifacts!

Summary

I really like the AWS CodeArtifact. It is easy to configure, easy to use. The idea with Domains and Upstreams is pretty nice.

The service is fast, reliable and managed by AWS. What else to expect? I strongly recommend CodeArtifact.

Important is, it can be used externally. not only cross-account inside AWS, but also as a proper external packages repository.

Currently it covers Python, Nodejs and Java repositories (edit: and C#), but I’m sure, we will receive more soon!

I wrote this article on Medium about one year ago. Time for migration :)

Tags:

Aws Cicd Devops

App Development Practices To Follow For Your Business In 2021

3 fundamental monitoring methods essential for every DevOps engineer 🚀💥

7 GitHub projects to make you a better Go Developer💥

How I passed the AWS Certified Cloud Practitioner Exam?

How to schedule ECS Services in AWS easily

The Koyeb Serverless Engine: from Kubernetes to Nomad, Firecracker, and Kuma

2021 Angular Interview Questions Beginners to Advanced

Docker Training 4 - Managing The Build Context

AWS CodeArtifact - the long-awaited feature

Tags:

Related Posts