Description
Python is a wonderful language, capable of gluing just about any number of other languages together, allowing us to reap the strengths of each language. The cost of this is myriad opportunities for library and binary incompatibility. This talk will cover some of the common ways that incompatibility arises, as well as several approaches that Continuum uses or is developing to reduce headache.
Abstract
Each language and each package has some things that it does better than others. One of Python's major strengths is its ability to readily interface with many other languages and harness their strengths. When it comes to speed, compiled, statically-typed languages are hard to beat. Because of this, data science libraries often harness compiled shared libraries for their compute-intensive operations. In a small environment with few of these dependencies, this often works quite well. However, as the number of binary dependencies grows, the risk of incompatibility increases.
The Anaconda distribution and its package manager, Conda, have been very successful as platforms in part because the wide array of packages built for Anaconda by Continuum are built to be binary compatible. Unfortunately, as more people and more communities build their own Conda packages that are not binary compatible with Anaconda or each other, more users have expressed confusion and consternation when things don't just work in all cases.
To address this issue, we have needed to develop tooling at both package build time, to better track and establish the actual limits of binary compatibility, and also at package install time, to allow conda to make better use of this information to inform users of what they can expect to work. We’ll talk about new features in our build tool, conda-build, that helps us choose our desired binary compatibility. We’ll also talk about our new automated build system that helps us flesh out a much larger range of compatibility by building more combinations of packages with limited binary compatibility. Finally, we’ll also talk about the extra data that you’ll soon be able to provide to Conda, so that you will have more power to say exactly what you want.