Modifying Your SQL Streaming Queries on the Fly: The Impossible Trinity

Stream processing systems excel at efficiently handling SQL streaming queries. However, as business requirements continue to evolve, users often find themselves needing to make modifications to long-running streaming queries. The straightforward approach is to halt existing queries and initiate new ones. But this can result in service downtime and consistency issues. So, the question arises: How can we empower users to modify queries on the fly?

We introduce the concept of the ""impossible trinity"" for implementing online streaming query modifications. In this context, we contend that, when it comes to stream processing systems, achieving all three of the following properties - generality, efficiency, and consistency - simultaneously is an unattainable goal. We delve into the nuances of each property and provide a comprehensive explanation of why achieving all of them is impossible.

Furthermore, we draw upon our own experiences in implementing online streaming query modifications. We introduce a pragmatic approach that balances the trade-offs inherent in the impossible trinity. We share these insights and discuss the most practical way forward based on our real-world experiences.