Description
Alex Vinyals - Ingesting 35 million hotel images with python in the cloud. [EuroPython 2016] [19 July 2016] [Bilbao, Euskadi, Spain] (https://ep2016.europython.eu//conference/talks/ingesting-35-million-hotel-images-with-python-in-the-cloud)
This talk covers the distributed architecture that Skyscanner built to solve the data challenges involved in the generation of images of all hotels in the world. Putting together a distributed system in Python, based on queues, surfing on the AWS Cloud.
Our goal? To build an incremental image processing pipeline that discards poor quality and duplicated images, scaling the final images to several sizes to optimise for mobile devices.
Among the challenges:
- Ingest all the input images that partners provide us.
- Detect and remove bad quality + duplicated images from reaching production.
- Resize all the generated images to optimise for mobile devices.
- Ensure the process scales and behaves in an incremental way.
- Ensure the whole process fits in a time constrained window.
Among the tools we used? Pillow, ImageHash, Kombu and Boto.