Contribute Media
A thank you to everyone who makes this possible: Read More

Create a serverless infrastructure for data collection with Python and AWS

Description

The talk illustrates a real-world example of how to collect data from your web, mobile, server and cloud apps and then send them to third party services and tools or load them into your data warehouse.

The data collection pipeline is integrated with multiple AWS services, such as Kinesis Firehose, Lambda functions and StepFunctions; Python is used to write each module. The data workflow is fully described pointing out how to store backup correctly, manage the conditional routing (in order to allow or discard data for specific services), implement a retry strategy on failure and finally compare performance and costs for each module.

Resources: clda.co/pycon8-data-collection-blogpost

Prerequisites: Basic knowledge of AWS services

Details

Improve this page