Description
In the past few years, there have been many introductions on the development of real-time data processing and analysis systems. In particular, data pipelines using Apache Kafka are being developed. I would like to share my experience developing an Apache Kafka consumer using Python through the system development process called real-time threat analysis. Through this presentation, I hope to be helpful to those who want to develop a service that collects real-time data to Apache Kafka and quickly develops a consumer using Python to process data. The main contents of this presentation are as follows. - Concepts required for data pipeline development - Introduction to Python Kafka client library - Experience in developing real-time data processing pipelines For questions and inquiries, please contact rlaxoghd91@gmail.com. Also, the code used in the presentation can be found at https://github.com/kimtaehong/pycon2022.