Description
Binary extensions underlie much of the modern Python ecosystem, providing performance and access to a wealth of existing code. Packaging for these libraries is rapidly changing from thousands of lines of distutils and setuptools based hackery to build systems designed for binaries like scikit-build-core, meson-python, and maturin. NumPy, for example, went from around 13K to 2K lines of building related code by moving to Meson in NumPy 1.26.
These build systems provide a much more integrated experience than was previously possible for compiled extensions. For example, CMake or Ninja are only required if the system doesn’t already provide an appropriate copy, which allows building with an external ninja/cmake on systems without binary wheels on PyPI like Pyodide, BSD, and Android. Modern editable installs are supported. Support for advanced features like ABI3 wheels or wheels that don’t call CPython is usually just a single configuration option.
We will look at how easy it is now to set up a binary extension using CMake, Meson, or Maturin (Rust only). It can be done with only three files each containing only a handful of lines of code. Unlike the previous solutions, this covers cross-compilation,multithreaded builds, modern C++ standards, and other features that would each require custom code in a classic setup.py. Combined with cibuildwheel for building wheels and good support from modern binding tools like pybind11 and nanobind, the barrier for entry to reliable compiled extensions has dropped dramatically.
We will also look at the challenges and solutions from some larger conversions to modern build systems, like NumPy’s and RAPIDS.ai.
After this talk, you will know how to easily create compiled extensions to solve problems you encounter, and how to move existing projects to these modern build systems.