Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

Calculating the End-to-End Latency of Realtime Ingestion

In our analytics use case we have a real-time ingestion pipeline where we ingest data from Kafka topics, map it to a particular type of object, perform the desired aggregations on the input records and finally write it to our database which is then available for querying. We calculate the ingestion end to end latency from the time when the record was published to Kafka until it is available for querying. The latency that is calculated is then populated to a dashboard for SLO and SLA calculation. This talk will dive into the details of calculating the end to end latency for our real time ingestion pipeline.


Minakshi Korad

I am a Senior Software Engineer working in the Enterprise Insights team at Twilio. I am always excited about distributed systems, data driven technologies also interested in scalability and performance aspects of the product. I am always on the lookout for learning new technologies and optimizing the current usecases.