Contribute Media
A thank you to everyone who makes this possible: Read More

Recent advances in python parallel computing


Modern hardware is multi-core. It is crucial for Python to provide
high-performance parallelism. This talk will expose to both data-scientists and
library developers the current state of affairs and the recent advances for
parallel computing with Python. The goal is to help practitioners and
developers to make better decisions on this matter.
I will first cover how Python can interface with parallelism, from leveraging
external parallelism of C-extensions –especially the BLAS family– to Python's
multiprocessing and multithreading API. I will touch upon use cases, e.g single
vs multi machine, as well as and pros and cons of the various solutions for
each use case. Most of these considerations will be backed by benchmarks from
the scikit-learn machine
learning library.
From these low-level interfaces emerged higher-level parallel processing
libraries, such as concurrent.futures,
joblib and
loky (used by
dask and scikit-learn) These
libraries make it easy for Python programmers to use safe and reliable
parallelism in their code. They can even work in more exotic situations, such
as interactive sessions, in which Python’s native multiprocessing support tends
to fail. I will describe their purpose as well as the canonical use-cases they
The last part of this talk will focus on the most recent advances in the Python
standard library, addressing one of the principal performance bottlenecks of
multi-core/multi-machine processing, which is data communication. We will
present a new API
for shared-memory management between different Python processes, and
performance improvements for the serialization of large Python objects (PEP 574, pickle extensions). These performance
improvements will be leveraged by distributed data science frameworks such as
dask, ray and
Modern hardware is multi-core. It is crucial for Python to provide
efficient parallelism. This talk exposes the current state and advances
in Python parallelism, in order to help practitioners and developers take
better decisions on this matter.


Improve this page