Description
80 % of analyst's work is to search for meaning in data. Unfortunately 95 % of signals found are pure noise. This is tremendous waste of time and potential of one of the smartest people. Being in their shoes for such a long time I felt the same frustration and decided to solve it for once and for all. In 2015 I designed and developed python engine that leverages 20+ analytics libraries and encompass standard analyst’s processes. It can effectively find all important situations in data, faster and more precisely than any human analyst. But thats not all. It also understands context and can precisely add explanation to every situation. In my talk I will explain overall system architecture, its orchestration via Airflow and libraries we use including pandas, scikit-learn or networkx. I will also discuss the biggest issues I faced to make it fast, configurable and generally applicable to any data.