Real-time Geospatial Aircraft Monitoring Using Apache Kafka

Capturing geospatial data of aircraft is data intensive due to needing frequent updates for the quickly changing position and velocity data points. A well-tuned real-time system for high throughput and low latency is a must as data is useless to operators if it is just a few seconds behind as they cannot make meaningful real-time decisions otherwise. In addition, the collaboration between operators must also be in real-time. This system, called Raft’s Data Fabric, modernizes this process for the United States Air Force as the communication layer for the systems acting upon real-time geospatial data of all aircraft over the United States airspace.

Data Fabric is built with Apache Kafka at its core. It uses KeyCloak for authentication, Open-Policy Agent for authorization, Kafka Streams for stateful processing, database systems (relational, OLAP, and document) for search, WebSockets for visualization integration, and Grafana for observability.

From this real-world use case of using real-time technologies, you will walk away knowing how to build end-to-end real-time systems with Apache Kafka at its core. Specifically:

• Tuning Apache Kafka for high throughput and low latency for small and larger messages.
• Building a WebSocket Kafka consumer for web-based visualization.
• Securely fusing and moving real-time data from unsecured to secured environments to provide better decision-making abilities.
• Integration with database systems for real-time and historical use cases.
• Reasons to consider leveraging OAUTH authorization and Open Policy Agent for authorization for your Kafka Cluster.