Découvrez de nombreuses nouveautés à l'occasion de notre lancement du 2e trimestre 2023 : moteur Kora Engine, règles de qualité des données, et bien plus encore | S'inscrire pour une démo
In our analytics use case we have a real-time ingestion pipeline where we ingest data from Kafka topics, map it to a particular type of object, perform the desired aggregations on the input records and finally write it to our database which is then available for querying. We calculate the ingestion end to end latency from the time when the record was published to Kafka until it is available for querying. The latency that is calculated is then populated to a dashboard for SLO and SLA calculation. This talk will dive into the details of calculating the end to end latency for our real time ingestion pipeline.