"Engaging Confluent gave us the peace of mind that comes with minimizing downtime and knowing that our Kafka systems are in the hands of one of the industry’s leading data-in-motion experts."
Josh LittleSun
Director of Technology, ETC
Founded in 1999, Electronic Transaction Consultants, LLC (ETC) is a leading U.S. intelligent transportation systems and smart mobility provider, developing and delivering best-in-class solutions for tolling, congestion management, urban mobility, and multimodal transportation needs. ETC’s passionate and innovative team has been driving the future of mobility since 1999, with many industry firsts, including all-electronic tolling (AET), dynamic pricing, agency interoperability, hosted mobility solutions, and machine learning.
For over two decades, ETC has delivered sophisticated solutions to many of the largest toll authorities in the U.S., including statewide programs, county networks, and tolling-specific authorities. ETC’s solutions process over two billion transactions annually, totaling over $3 billion in revenues for our customers, incorporating the latest in evergreen open source and SaaS technologies and big data architecture through our innovative riteSuite™ products.
Facing a rapidly increasing volume of data flowing in from the various sensors and devices associated with its customers’ ever-expanding toll and smart mobility projects, ETC scaled up to meet surging demand by turning to Kafka for messaging queue agility and to Confluent Cloud to maintain high uptime for critically important transaction processing systems and to reduce infrastructure-related and operational costs.
“When we first started implementing Kafka, we didn’t realize how important it was going to become for us frankly,” said Josh LittleSun, ETC’s Director of Technology. “We thought it was just going to be another tool in our toolbox. As it evolved though, we realized it had become the beating heart and backbone of our solution, so that’s really the main reason we switched over to Confluent.”
By bringing in Confluent Cloud, ETC was able to significantly reduce the amount of time its engineers spend on managing Kafka, while reducing downtime risk and increasing agility. Instead, ETC spends its valuable engineering resources on accelerating innovation and business-differentiating projects.
Technical Solution
As a leading systems and services provider for the rapidly evolving intelligent transportation systems and smart mobility industry, ETC needed to figure out three main challenges to support surging demand and digitization:
1. How to get more information upstream in a manageable way
2. How to make that information auditable
3. How to continue to scale horizontally as they add more sensors, without breaking the bank or over-consuming internal resources
ETC began to incorporate Kafka into its IT architecture in 2015 to address this need for increased scalability and flexibility. By the time ETC got its Kafka workloads into production four years later, Kafka had started to take on a significantly more important role than they had originally anticipated.
A year ago, ETC was leveraging Kafka to process close to ten million events per day for its dynamic pricing solutions— a number that will likely double or triple in the next couple of years. With such explosive growth, ETC realized its total cost of ownership (TCO) for self-supporting open-source Kafka—from infrastructure to operations and maintenance to downtime risk—was quickly piling up toward unacceptably high levels.
After conducting an internal TCO analysis, ETC decided to move from self-supporting Kafka in-house to Confluent Cloud in early 2021. To support the growing number of expected new customers on board, ETC will save considerably on both infrastructure costs and full-time employees previously tasked with managing and maintaining Kafka.
Minimizing downtime risk was another key reason ETC wanted to move its workloads to Confluent Cloud, as Kafka gradually became “the central nervous system” of its data infrastructure and applications. ETC’s engineers found that while it’s relatively easy to run Kafka smoothly when things are going well, debugging or preventing Kafka-associated downtime required more specialized in-house resources than could be provided cost-effectively. Essentially, Kafka had become “too big to fail.” In addition, the internal technical team had to dedicate significant resources to security and governance, because open source Apache Kafka does not provide sufficient tools at the enterprise level.
Therefore, migrating to Confluent Cloud became the clear way forward for ETC to be able to seamlessly and securely scale up for new customers and support rapidly increasing data volumes.
Business Results
-50% Reduction in the In-House Resources Dedicated to Managing Kafka
Between compute, storage, and networking, ETC had begun to feel the impact of increasing infrastructure costs as new customers came on board. The company was having to manually add more Amazon Elastic Compute instances to handle the increased throughput, in addition to managing the high networking costs of complying with government-enforced, 30-day multi-zonal persistence requirements. Thanks to Confluent Cloud, ETC has already begun to realize significant cost savings around infrastructure as it elastically scales to meet demand while adding more customers.
Additionally, Josh LittleSun estimated that, within a few months, the introduction of Confluent has already enabled ETC to reallocate ~50% of in-house FTE resources that had been previously dedicated to supporting open-source Kafka to more strategic projects accelerating innovation. These efficiencies come from the fact that, as a fully managed service, Confluent Cloud takes care of a wide variety of tasks that would otherwise consume a significant amount of valuable engineering resources, including capacity planning, scaling, load balancing, Kafka patching and performance, and government-required disaster recovery testing. As a result, ETC’s engineers have been able to focus on more strategic, high-priority projects that differentiate its business rather than managing low-level infrastructure. Notably, the ~50% in-house resource optimization is just for its current workloads. As ETC leverages Confluent Cloud for additional workloads, the realized cost savings are projected to grow over time.
Reduced Downtime Risk
In one notable example, ETC’s dynamically priced roadside solutions involve pushing the pricing generated based on traffic data to the rate signs, a process that largely relies on dependable performance from Kafka. “If it goes down, there are no rates published to the sign which would result in foregone revenue,” said Josh LittleSun. “In such a scenario, losses can potentially be substantial per each minute of downtime.” With Confluent Cloud’s 99.95% uptime SLA, this alone accounts for significant risk reduction for ETC from using Confluent Cloud, not to mention the additional cost of pulling valuable engineering resources from other projects to deal with Kafka-related fire drills.
Other Benefits
Increased processing speeds
Efficient and accurate auditing
Improved debugging and troubleshooting
“Being able to take and copy topics and replay topics has lowered our error rate in production and enabled us to solve issues for our customers even faster than before,” Josh LittleSun said. “Another benefit is the ability to decouple all our processing from our reporting, which allows us to process events faster and keep our database running seamlessly, while providing near-real-time analytics and quick reports to our customers.”
The Future
With Confluent Cloud, ETC can operate significantly more efficiently, with reduced infrastructure costs, maintenance overhead, and downtime risk to cost-effectively meet surging demand.
