Anomaly Detection on Time Series Data Using Apache Flink

The emerging need to be cloud native caused a fast adoption of applications to be redesigned. However, many are managing the cloud native deployments without taking any precautions against the intrusions in network security. Therefore, the need to detect anomalies of the network activity time series data in real-time is crucial yet still nontrivial.

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. The ability of Apache Flink to process continuous data streams in a stateful manner makes it a perfect match for analysing time series data where the repeated observations of data points are continuously produced.

In this talk, we will walk through the steps to implement a real time anomaly detection system on time series data using Apache Flink. We will implement and compare several algorithms, Exponentially Weighted Moving Average (EWMA) and Probabilistic EWMA (PEWMA), from an academic paper - PROBABILISTIC REASONING FOR STREAMING ANOMALY DETECTION.