Build Predictive Machine Learning with Flink | Workshop on Dec 18 | Register Now
Quality Gate Through Centralized Computational Policy Enforcement
Discover how Confluent Cloud Schema Registry and Stream Governance Advanced can enhance data product quality with centralized computational policy enforcement by enabling proactive quality gates for data contract management.
Ensure Consistency, Reliability, and Compliance Across Your Data Product’s Lifecycle
A quality gate represents a built-in checkpoint within your pipeline that software must pass before advancing to the subsequent phase. This checkpoint imposes specific rules and best practices that the code must follow to prevent the introduction of poor quality into the codebase. This approach aims to ensure consistency, reliability, and compliance with standards across the entire process by using automated tools or systems to apply these rules uniformly.
What does Quantyca’s solution solve for?
- Exploit a wider syntax than the one currently supported by Schema Registry’s domain rules
- Reference a central repository for policy definitions
- Apply policy enforcement as close as possible to data producers
- Define custom behavior to handle bad data
Benefits of Quantyca’s solution:
Avoid redundancy and facilitate integration.
Data is verified before its publication, enabling effective quality management.
Build with Confluent
This use case leverages the following building blocks in Confluent Cloud:
Stream Governance Advanced →
To enable usage of domain rules.
Schema Registry →
To bind policies defined externally to schemas.
Reference Architecture
Each time a data producer wants to write a message a Rule executor is used to talk to Open Data Mesh Platform’s Policy Service through a REST API call. This service is responsible for verifying compliance with policies defined centrally in a Policy Server.
The rule executor uses the message and the associated schema to communicate with the Policy Service: the schema specifies as domain rules the names of the policies to check, as they are part of the data contract of a data product.
Policy definitions are managed centrally on the Policy Server, this makes it possible to organize them in suites and manage their enforcement across different technologies by keeping a single repository to manage.
After collecting the verification answer a rule action is performed, for example to send an email and forward the message to a dead letter queue if the requirements aren’t satisfied.
Resources
Learn More at Quantyca
Email Quantyca
Contact Us
Contact Quantyca to learn more about this use case and get started.