General Coordinates Network: Harnessing Kafka for Real-Time Open Astronomy at NASA

Kafka has come to play an essential role in astronomy research. Particularly where black holes and neutron stars are involved, astronomers are increasingly seeking out the “time domain” and want to study explosive transients and variability. In response, observatories are increasingly adopting streaming technologies to send alerts to astronomers and to get their data to their science users in real time. In this talk, we will discuss architectural choices, challenges, and lessons learned in adapting Kafka for open science and open data. Our novel approach to OpenID Connect / OAuth2 in Kafka is designed to securely scale Kafka from access inside a single organization to access by the general public. We will present a case study of the General Coordinates Network (GCN), a public collaboration platform run by NASA for the astronomy research community to share alerts and rapid communications about high-energy, multi-messenger, and transient phenomena. Over the past 30 years, GCN has helped enable many seminal advances by disseminating observations, quantitative near-term predictions, requests for follow-up observations, and observing plans. GCN distributes alerts between space- and ground-based observatories, physics experiments, and thousands of working astronomers around the world.