Description
This talk was presented at PyBay2019 - 4th annual Bay Area Regional Python conference. See pybay.com for more details about PyBay and click SHOW MORE for more information about this talk.
Description We will share our experiences on building Metaflow, a Python library that is used at Netflix to build and operate hundreds of machine learning applications. This talk is for you if you want to learn how to develop systems for big data and ML in Python.
Abstract Metaflow is a Python library that empowers data scientists to prototype, build, deploy, and operate end-to-end machine learning solutions. We started building Metaflow at Netflix to provide a solid foundation for hundreds of internal ML use cases, from classical statistical analysis to large-scale applications of deep learning. Metaflow is designed with a human-centric mindset: instead of reinventing the wheel for large-scale computing or machine learning, we integrate existing solutions into a delightfully consistent and easy-to-use package.
This talk focuses on the internals of Metaflow; it will highlight lessons that we have learned in building a Python library that needs to be robust, performant, and flexible enough to solve a large set of complex real-world business problems related to machine learning. We will share our learnings around optimizing the performance of Python, in particular, related to concurrency. We will also talk about the complexities of dependency management, the overall architecture of the library and the cloud-based compute and storage systems it relies on.
About the speaker Ravi is an engineer on the ML Infrastructure team at Netflix. He has been building large scale systems focusing on performance, simplified user journeys and intuitive APIs in MLI and previously in Search Indexing at Google (6 years).
Sponsor Acknowledgement This and other PyBay2019 videos are via the help of our media partner AlphaVoice (https://www.alphavoice.io/)!
#pybay #pybay2019 #python #python3 #gdb