Register for Demo | Confluent Terraform Provider, Independent Network Lifecycle Management and more within our Q3’22 launch!

Enabling product personalisation using Apache Kafka, Apache Pinot and Trino

Our core banking platform has been built using domain driven design and microservices and whilst this provides many well-known advantages, it also presents some challenges. Data encapsulation results in each application having its own data store and it becomes impossible to query the state of a customer’s relationship in totality to provide the right products. This challenge becomes even harder if we want to personalise products based on aggregate values of a customer’s behaviour over potentially large periods of time.

In this session, we describe how we overcome this problem to enable dynamic charging and rewards based on customer behaviour in a banking scenario. We describe • How we guarantee consistency between our event stream and our OLTP databases using the Outbox pattern. • The design decisions faced when considering the schema designs in Pinot and how we balanced flexibility and latency using Trino • Two patterns for enriching the event stream using Kafka streams and how we dealt with late arriving events and transactions.

Presenter

Stuart Coleman

I have been a practitioner in the data engineering and data science field for the past 12 years, working in an array of fields from building and designing event driven core banking systems to being the cofounder and CTO of an edtech startup. Prior to that I was an academic in the field of Statistical Physics

I have a passion for using data to enhance products for customers and my main area of expertise lies within the fields of Big Data Analytics, Machine Learning, Integration, Microservices and Core Banking

Currently, I’m the Head of Product and Engineering for 10x Banking, a cloud-native core banking platform.