Publishing a Python Package: What I Wish the Maze of Tutorials Covered

I've written a number of python packages over time for my own use really and I'm so snowed under just getting it done that every time I stopped to consider sharing one, so it could be pip installed, I searched on-line and immediately got lost among the tens, hundreds maybe, of guides, tutorials and options and what looked like enough material for a doctoral dissertation to wade through. And all of it looked scary talking about starting from scratch (not from an existing package), wanting me to add and write a dozen files and understand this and that, and worse there are old ways, new ways, alternate ways ... Aaargh.

But hey, we're in a snap 3 day lockdown here (thanks COVID!) and I was tidying up some projects and tried again ... this time, experience in hand, (and less hair) I figured I would put my blinkers on and use just the official tutorial:

And I was mildly pleased. I guess my expectations had dropped but hey, it proved manageable and I did succeed in publishing a few packages. But even this tutorial left me wasting far too much time trying to work out how stuff works and I wanted to write it down ASAP, for my own sake, and well, here we are ... share it.

So here's a better (IMHO) guide to packaging your Python Project and publishing it (better because it says what I wanted to know, and I'm making different mistakes and it has it's own shortcomings that I don't notice ;-).

1. Start with a package

Yep, I already have packages ... a good few of them. I want to publish them. Anyhow a package of course is just a folder with an __init__.py file in it, and if it's a small package (as many of mine are) that's all they are. Nothing more, nothing less, than a folder with an __init__.py in it that provided some classes and/or functions.

Sometimes there might be a few more .py files in the folder beside the __init__.py. Minor detail, just what happens when it's a little too big to fit conveniently into the one file.

Key thing here is we're not starting from scratch, but have a package.

2. Get the tools

You only need two tools as it happens and they are:

build - which creates from your package by default a .tar.gz file and a .whl file which is what pip wants/needs for it's installing career.

twine - which publishes to pypi.org

and so for prep:

pip install build twine

Not too bad. Step 2 was easy.

3. Prep the few extra files a package wants

Not many, don't fret.

README.md - a simple markdown file with a welcome message and whatever you want to add. What is this package, how do you use it? I use Typora but you can write it in any text editor and it can be as brief or in depth as you like. It is what's shown on the pypi.org page for your package so it's also your ad if you like for your package.

LICENSE.md - Not sure you need it but worth doing and easy as Py. I am beastly careless in this space and just love the Hippocratic License. Download the markdown version and save it as LICENSE.md

That's it! And to think it seemed so scary in past.

4. But no! There's more - just a little more, not much.

The tutorial recommends that you lay out your folder like this (yes, I've simplified it a bit):

the-folder-I-keep-it-in/
├── LICENSE
├── README.md
├── pyproject.toml
├── setup.cfg
└── src/
    └── my_package/
        └── __init__.py

The things to note are:

You don't need to put it in a src folder, but why not? If it ain't broke don't fix it. The src strategy means you can just drag and drop you package from where it was into src ... done. The stuff above it is the publishing kit ...

pyproject.toml and setup.cfg just tell build and twine what to do. We'll come back to these shortly. The first is just a standard file to tell build that we're going to use twine in a roundabout way ;-) and the second one describes your package so twine can publish it (purists may argue with this neat division, but let them).

The folder the-folder-I-keep-it-in can have any name you like. won't change a thing with the build or publish. I actually call it my-package (in this example). As to why, keep reading. It's just convenient that's all.

The folder my_package should use underscores between words, yes, do it. There's a bizarre confusion in the Python world between my_package and my-package.

my_package and my-package? What, why, when?

This is described nowhere, and I had to work this out with a lot of trial an error and hair pulling alas. But here's what I got for you.

my_package: just stick to this don't waver, never waver, use only this ;-). I kid ye not. Using my-package in either the folder under src or in setup.cfg will cause you grief during the build, publish, install and test.

Once it's published it will appear on pypi.org as my-package and people will install it with pip install my-package, but use it with import my_package. That's just the way it is, that's the convention, don't rock the boat, all you need to know is you don't have to lift a finger to make that happen, just stick with my_package in the src folder and in setup.cfg.

But of course, the-folder-I-keep-it-inis irrelevant here and I call it my-package just because, because that's what the package is called. The only other exception is the github repo if you're using one (and I do), that too can be my-package and is in my case in fact later you'll se I can exploit that for a nice two line install script.

5. pyproject.toml and setup.cfg

pyproject.toml is easy. Just copy the standard. Put this in it:

[build-system]
requires = [
    "setuptools>=42",
    "wheel"
]
build-backend = "setuptools.build_meta"

and be done with it. Ask no more. It's build internals and unless you're super keen in digging deeper, let it rest, this just means when you run python3 -m build in your package folder, it knows what to do (if you don't have this file it will ask for one). What it does, is created a dist folder and drops two files in it. These are what twine needs

setup.cfg is not hard either and here's my minimalist take and the clarifications that I felt were missing elsewhere:

[metadata]
name = my_package
version = 0.1
author = my name
author_email = my email address
description = My little package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/me/my-package
project_urls =
    Bug Tracker = https://github.com/me/my-package/issues
classifiers =
    Programming Language :: Python :: 3
    License :: Freely Distributable
    Operating System :: OS Independent
    Development Status :: 4 - Beta
    Framework :: Django :: 3.2
    Intended Audience :: System Administrators
    Topic :: Software Development :: Libraries :: Python Modules

[options]
install_requires =
    other_package1 >= 0.1.1
    other_package2 >= 2.0.1
package_dir =
    = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

And here's what I felt I should have known:

name should use my_package not my-package. Just believe me. Things go weird if it says my-package. Experiment if you like, I wish I didn't need to and the tutorial was clear here.

install_requires wants one indented line per requirement with relatively familiar syntax (similar to pip freeze - another one of those mysteriously named python commands that actually means pip show-me-whats-installed). This is completely missed in the tutorial.

package_dir is weird, yes, but forget it. Like install_requires it has a list of one liners beneath it, in this case just one. The one liners map package names to folders somehow in the internal complexities of setuptools - details most of don't care about or want to know about when publishing our simple one file package. The tutorial tells us that this line maps the "noname" package to the src folder, and that the "noname" package (that nothingness before the = sign) is a code name for the overarching root package, so the src folder becomes the mystical "root package". Do most of us actually care about this? What is a "root package"? anyhow. Nah, let's leave it for the boffins, and just accept this is the odd way of telling build and/or twine that our package is in the src folder.

there's nothing missing after find:. No. That's just the syntax, live with it. Refer back to the intro, re: my sentiments on the unnecessary befuddling cryptic nature of Python package publication ... Ditto the where = src, just accept it.

The classifiers are bit fiddly they have to come from the list of allowed classifiers. And they bothersomley lack a clear way of saying you're using the Hippocratic License (which I just happen to love).

6. The Importance and Catches with Testing

Publishing is as simple as:

python3 -m twine upload dist/*

BUT, it's committal. Once you've published there appears to be no way of undoing it and it consume the filenames you used (which means also the version you have in setup.cfg as these get built into the filenames in dist).

And so, testing first is critical. And pypi.org provide testpypi at https://test.pypi.org/ that you can publish to freely, as often as you need to get it right.

The main things that demand a retry are in my experience:

You look at it on pypi and README.md has issues. Either typos, or code lines that are too long and render badly etc. Either way, you get see how it's going to presented on pypi and can adjust your README to look nice.

Your test installing it with pip doesn't work. Which actually doens't happen now that I have a workflow, but happened a lot while Iw as trying to work all that setup.cfg syntax out that the tutorial deigns to gloss over.

To publish to the test site it's just small variant:

python3 -m twine upload --repository testpypi dist/*

The catches

So testing is great. A lifesaver. But it caused me some modest grief too (the flip side of the same coin).

Firstly you need to create an account on the site, and I did that but use Bitwarden always, and generate large random passwords for me - a habit (that we should all have).

twine when used as above prompts for username and password. Alas these long random passwords of mine are not easy to type, so I usually do a copy/paste but alas pasting the password does not work - I tried and tried.

Fortunately they can be provided on the command line as in:

python3 -m twine upload --repository testpypi -u $username -p $password dist/*

and I saved this in a file called test-publish that reads:

#!/bin/bash
source ~/.auth/pypi.auth
python3 -m twine upload --repository testpypi --verbose -u $username -p $password dist/*

Secondly, you can't republish. At all. You need to increment the version in setup.cfg and rebuild before you can republish. Slows things down some. Not least because of the time and energy spent searching online for ways and means to republish. Some on-line sources suggest --skip-existing does the trick, but it doesn't - not for me and it's not clear what it does or what it's for and maybe I just misread that. C'est la vie.

Thirdly, the dependencies listed under install_requires in setup.cfg don't work, presumably because, when testing the required packages aren't on https://test.pypi.org/. But it took a bit of head scratching and try and try again to convince myself of that, as I was trying believe it or not to validate the syntax for just that setting as it's not described in the tutorial and sent me looking at that warren of other sources quickly again. I do wish that testpyi would look at pypi for requirements as a fallback so this test cycle could be complete.

7. A Standard Workflow

OK, so having gone through that all now, like most folk eventually do, I have a standard template (the last package I published). I now routinely use five tiny little two line shell scripts to make life easy for myself.

Basically a build script, and two publish and install scripts.

In order:

A script to build:
build:

#!/bin/bash
rm dist/*
python3 -m build

A script to test publishing:
test-publish:

#!/bin/bash
source ~/.auth/pypi.auth
python3 -m twine upload --repository testpypi  --verbose -u $username -p $password dist/*

A script to install the test publish (test installing) - noting that errors here about requirements that cannot be met are expected:
test-install:

#!/bin/bash
package=$(basename $(dirname $(readlink -f "$0")))
python -m pip install --index-url https://test.pypi.org/simple/ $package

A script to publish properly:
publish:

#!/bin/bash
source ~/.auth/pypi.auth
python3 -m twine upload --verbose -u $username -p $password dist/*

A script to install the package properly:
install:

#!/bin/bash
package=$(basename $(dirname $(readlink -f "$0")))
python -m pip install $package

A basic example of that together you can visit at:

and see here:

I hope that helps someone save all the learning hassle, and publish something easily, by just adding 4 files to a folder (a README.md to write, a LICENSE.md to download, a pyproject.toml to copy, and setup.cfg to tune) and maybe 5 tiny little helper bash scripts and in no time a test and then publish cycle is underway.