Description
The talk illustrates a real-world example of how to collect data from your web, mobile, server and cloud apps and then send them to third party services and tools or load them into your data warehouse.
The data collection pipeline is integrated with multiple AWS services, such as Kinesis Firehose, Lambda functions and StepFunctions; Python is used to write each module. The data workflow is fully described pointing out how to store backup correctly, manage the conditional routing (in order to allow or discard data for specific services), implement a retry strategy on failure and finally compare performance and costs for each module.
Resources: clda.co/pycon8-data-collection-blogpost
Prerequisites: Basic knowledge of AWS services