Homebrew, pyenv, ctypes... oh my!

Background

Skipping to the End

The Problem

The Solution?

The Itch

Isolating the Useful Change

One Level Down
Another Level Down
Finally An Answer

In Other, Fewer Words...

Lingering Questions

Background

I usually have a number of Python versions, environments and tools in play. I generally use pyenv to manage Python versions, and Homebrew to manage pyenv.

Skipping to the End

If you've come across this post because you're having problems installing Python via Homebrew-managed pyenv, you have my sympathies. And in case it "just works" for you, feel free to give something like this a try:

CC="$(brew --prefix gcc)/bin/gcc-11" \
pyenv install --verbose 3.10.0

If you're interested in the circuitous road of troubleshooting and exploration that led to that command, you're welcome to keep reading and come along for the ride. Nice to have company :).

The Problem

After successfully installing Python 3.10 with pyenv, I found that I couldn't pip install packages with C extension modules. They complained about _ctypes being missing. And trying to import ctypes from the Python REPL confirmed the issue:

Python 3.10.0 (default, Dec 31 2021, 16:09:32) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/python-build.20211231145343.269699/Python-3.10.0/Lib/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'

Looking at the logs of my pyenv install command, I found this error:

*** WARNING: renaming "_ctypes" since importing it failed: libffi.so.8: cannot open shared object file: No such file or directory

Which took things a bit further and pointed at an underlying problem. Now, I've seen that issue (or one of its kin) before, and it usually means something like "go muck with LDFLAGS until it works". And after a bit of searching I found a number of relevant resources with plenty of suggestions:

https://stackoverflow.com/questions/27022373/python3-importerror-no-module-named-ctypes-when-using-value-from-module-mul

https://github.com/pyenv/pyenv/issues/2159

https://github.com/pyenv/pyenv/issues/2137

Along with the always-helpful suggested build environment information from the pyenv wiki.

That was plenty of information to fuel a few cycles of "try stuff and see what happens".

The Solution?

Exploring those linked issues was productive, but my Python environment was still broken. I also wasn't sure why my previous pyenv-installed versions of Python could import ctypes just fine.

But I noticed that many of the suggested fixes worked because they took homebrew out of the equation. What I really wanted was to use the build tooling Homebrew already had installed. And so I thought "Why am I passing Homebrew paths in for CFLAGS and LDFLAGS but still letting pyenv install use my system gcc?". So I tried:

CC="$(brew --prefix gcc)/bin/gcc-11" \
CFLAGS="$(pkg-config --cflags libffi)" \
LDFLAGS="$(pkg-config --libs libffi)" \
pyenv install --verbose 3.10.0

And it worked! I commented about it in this issue because I suspected other folks with a similar problem might benefit from the same solution. Then I went back to whatever the heck I was trying to do with Python in the first place :).

The Itch

The "solution" got my Python 3.10 environments working perfectly, but it also felt unsatisfying. I had some hunches about why it worked, but I wasn't really sure. And then I saw...

A few other folks hit the same issue, and note that my suggestion worked for them

A really great post from Julia Evans debugging a very similar issue

Which made me think, "This is a great problem to explore during the last week of December".

Isolating the Useful Change

I had been thinking that since adding a custom CC effectively fixed my Python build, the CFLAGS and LDFLAGS bits might be unnecessary. So I tried without them:

CC="$(brew --prefix gcc)/bin/gcc-11" \
pyenv install --verbose 3.10.0

And yes, that still worked. But why?

One Level Down

The next step was to drop down a level and use python-build to set up side-by-side build directories: one working, one broken. To get a broken one, I used:

python-build --keep --verbose 3.10.0 ./py310

And for a working one:

CC="$(brew --prefix gcc)/bin/gcc-11" \
python-build --keep --verbose 3.10.0 ./py310-2

From the broken directory, I could run make and see the ctypes warning. I could also see the object file renamed to ..._failed.so:

❯ fd "ctypes.*so"
build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu_failed.so
build/lib.linux-x86_64-3.10/_ctypes_test.cpython-310-x86_64-linux-gnu.so

Compared to the working directory which showed the expected file names:

❯ fd "ctypes.*so"
build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.10/_ctypes_test.cpython-310-x86_64-linux-gnu.so

The earlier error messages mentioned an error finding libffi, and ldd helped highlight and reproduce that. From my broken build, I see "not found":

> ldd build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu_failed.so
        linux-vdso.so.1 (0x00007ffd97fbf000)
        libffi.so.8 => not found
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f478258f000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f478256c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f478237a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f47825d8000)

While the working file references homebrew's libffi:

> ldd build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffcd35fb000)
        libffi.so.8 => /home/linuxbrew/.linuxbrew/lib/libffi.so.8 (0x00007f306da2d000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f306da08000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f306d9e5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f306d7f3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f306dc5f000)

I was providing custom CFLAGS and LDFLAGS options that I thought would avoid this issue, but obviously that didn't do it. What was that custom CC doing that the CFLAGS/LDFLAGS change didn't do?

Another Level Down

I looked in the python build log to get the gcc command that built _ctypes.cpython-310-x86_64-linux-gnu.so. It looked something like this (noise alert):

gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/home/linuxbrew/.linuxbrew/opt/zlib -I/home/linuxbrew/.linuxbrew/opt/zlib -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/home/linuxbrew/.linuxbrew/Cellar/libffi/3.4.2/include -I./Include -I. -I/home/linuxbrew/.linuxbrew/opt/readline/include -I/home/aj/code/scratch/pyenv-builds/./py310/include -I/home/linuxbrew/.linuxbrew/include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/tmp/python-build.20211231172024.425603/Python-3.10.0/Include -I/tmp/python-build.20211231172024.425603/Python-3.10.0 -c /tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.c -o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.o -DPy_BUILD_CORE_MODULE -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1
gcc -pthread -shared -L/home/linuxbrew/.linuxbrew/opt/readline/lib -L/home/aj/code/scratch/pyenv-builds/./py310/lib -L/home/linuxbrew/.linuxbrew/lib build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/callproc.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/cfield.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.o -L/home/linuxbrew/.linuxbrew/opt/readline/lib -L/home/aj/code/scratch/pyenv-builds/./py310/lib -L/home/linuxbrew/.linuxbrew/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -lffi -ldl -o build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.so

There's a lot going on there, but as a test I ran the same command with all the same command line options, but using Homebrew's gcc-11 instead of my system gcc. And... that still worked. That was another helpful input.

Finally An Answer

Once I found that I could fail or succeed to build _ctypes.cpython-310-x86_64-linux-gnu.so by varying only the gcc version, I ran both versions of gcc with the --verbose option to look for more clues. And I found my answer in the form of gcc specs. The verbose logs from my system gcc included this bit:

Using built-in specs.

While the Homebrew gcc log included:

Reading specs from /home/linuxbrew/.linuxbrew/Cellar/gcc/11.2.0_3/lib/gcc/11/gcc/x86_64-pc-linux-gnu/11/specs

And peeking into /home/linuxbrew/.linuxbrew/Cellar/gcc/11.2.0_3/lib/gcc/11/gcc/x86_64-pc-linux-gnu/11/specs revealed some custom options:

*cpp_unique_options:
+ -isysroot /home/linuxbrew/.linuxbrew/nonexistent -idirafter /home/linuxbrew/.linuxbrew/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include

*link_libgcc:
+ -L/home/linuxbrew/.linuxbrew/lib/gcc/11 -L/home/linuxbrew/.linuxbrew/lib

*link:
+ --dynamic-linker /home/linuxbrew/.linuxbrew/lib/ld.so -rpath /home/linuxbrew/.linuxbrew/lib/gcc/11

*homebrew_rpath:
-rpath /home/linuxbrew/.linuxbrew/lib

I've never messed around with gcc spec files to be honest. But it sure looks like that's what was causing my Homebrew gcc to work where my system gcc didn't.

In Other, Fewer Words...

My condensed takeaway after all of this exploration and troubleshooting is:

I'm trying to install Python using Homebrew-managed build dependencies. Those tools know where their pieces live and how they should work together. I should get out of their way and let them work. By pointing at Homebrew's gcc with pyenv install, I'm letting pyenv operate in the world it knows. The world where it was born.

Lingering Questions

There are some things I'm still not sure about. If you're reading this, you are a lovely human and a patient angel. And maybe you can help shed additional light on my busted system? Notably...

My Homebrew-managed pyenv on this system previously installed Python versions from 3.6 through 3.9 without any issues and without requiring any CC-fiddling. Did something change with Homebrew or pyenv, or is this possibly something specific to Python 3.10?

The options in the gcc spec look similar to the CFLAGS and LDFLAGS I was trying... is there some other magic combination of flags that would have allowed my system gcc to find everything it needed?

I may come back to these questions myself in a later troubleshooting phase. But I suspect I'm missing something obvious that someone else would catch at a glance!