Streaming is a Detail

We can all agree that streaming is super cool. And for a while now, the adoption conversation has been largely led with an all-in mentality. But that’s silly. The only concerns end users have are:

-The freshness of their data
-Latency they require to meet their SLAs from source to consumption
-All while maintaining data quality and governance.

Luckily, the industry has realized this and we have seen a shift of streaming capabilities surfacing as an in-database technology, via objects as trivial to analytics engineers as views - materialized that is. With this convergence of streaming capabilities and batch level accessibility, this is when ELT tools like dbt can join in and expand out the adoption story.

dbt is the T in ELT, Extract Load and Transform. In dbt, analytics engineers design models - SQL (and occasional python) statements that encapsulate business logic. At runtime, dbt will wrap that logic in a DDL statement and send it over to the data platform to execute.

In this session, we’ll discuss how we see streaming at dbt Labs. We will dive into how we are extending dbt to support low-latency scenarios and the recent additions we have made to make batch and streaming allies in a DAG rather than archenemies.