Description
Have you ever tried to define and process complex workflows for data processing? If the answer is yes, you might have struggled to find the right framework for that. You've probably came across Celery - popular task flow management for Python. Celery is great, but it does not provide enough flexibility and dynamic features needed badly in complex flows. As we discovered all the limitations, we decided to implement Selinon.
Have you ever tried to define and process complex workflows for data processing? If the answer is yes, you might have struggled to find the right framework for that. You've probably came across Celery - popular task flow management for Python. Celery is great, but it does not provide enough flexibility and dynamic features needed badly in complex flows. As we discovered all the limitations, we decided to implement Selinon.
Selinon enhances Celery task flow management and allows you to create and model task flows in your distributed environment that can dynamically change behavior based on computed results in your cluster, automatically resolve tasks that need to be executed in case of selective task runs, automatic tracing mechanism and many others.