Description
Carsten Pohl (@carstenpohl)
Before joining REWE digital I worked for Zalando, Fraunhofer IML. My first computer was a VIC-20. I develop professionally software since the 90ies. I am one of the main developers of the new Big data platform of REWE digital, and have deep knowledge about all aspects of the platform. And I have no favorite member of One Direction.
Abstract
shop.rewe.de is not only visited by human customers, but also by machines. We have built a deep learning platform using python with Keras, Tensorflow, on the Google infrastructure. In this talk we would like to show you how python is used in practice, supporting 2,5 million visitors each day.
Description
shop.rewe.de is visited over 2 million times each day. Every visitor is producing thousands of requests in our micro service architecture. We are trying to give the best shopping experience for our customers, and try to keep our platform safe from bad bots. This is partly done by rule sets, and furthermore done by a platform that uses machine learning to classify bad behaviour. In this talk I would like to present our architecture that is not only able to fulfill this use case, but enables our data scientists in general to use our big data platform. the main scope will be the presentation of the use case. I will cover:
- different microservices written in python using flask, google bigquery, tensorflow and keras
- scaling this microservices with kubernetes, that automatically starts more predictors in case of higher load
- implementation of quantifiers written in python, that generates data suitable for neural networks using numpy
- examples from the real world behaviour of this plattform with lessons learned
- examples how we feature engineered the data by analysing the data stored
Recorded at PyCon.DE 2017 Karlsruhe: https://de.pycon.org/
Video editing: Sebastian Neubauer & Andrei Dan
Tools: Blender, Avidemux & Sonic Pi