"Data that took us a whole day to get previously is now streaming via Confluent and available in seconds."
Allen Woo
Chief Innovation Officer, Affin Hwang Asset Managment
With more than RM 80 billion ($19 billion USD) under management and close to 200,000 clients, Affin Hwang Asset Management Berhad is among the leading asset management firms in Malaysia. The firm’s roots stretch back to 2001, and over the past two decades its IT infrastructure expanded organically as it grew to meet the needs of institutions, pension funds, government-linked companies, and other clients. As a result, the back-end landscape included multiple siloed legacy systems and batch processes that, although fully functional, were beginning to strain under the load placed on them and affect the customer experience.
“We potentially run the risk of different versions of the truth, with data sitting in various systems,” says Allen Woo, Chief Innovation Officer at Affin Hwang Asset Management. “We relied on batch processes to roll all this data up to a data warehouse, but those processes were taking longer and longer to complete. If they failed and had to be restarted, we were at risk of clients not seeing up-to-date data when they logged in and the entire process was at risk of breaking down due to the sheer volume of data it was handling.”
To address this challenge and modernize its operations, Affin Hwang Asset Management launched a digitization initiative that is being powered in part by real-time data streaming with Confluent. Confluent is enabling Affin Hwang Asset Management to integrate siloed systems, reduce batch processing, and explore new product offerings. “At Affin Hwang Asset Management, we are great believers in not wasting time, so once we started our digitization initiative, we wanted to start making progress right away without long, drawn-out setup times for new technology,” says Woo. “The team at Confluent has been exceptional in understanding the resources that we have and working closely with us to support our vision of adopting data streaming quickly. Already, transaction data updates that used to take up to a day are now available right away, and we’re excited about the many other doors that data streaming will open for us.”
Although Affin Hwang Asset Management is in the early stages of its event streaming journey, the company is on a fast track to maximizing value by putting more of its data in motion. “We’ll be streaming onboarding and payment data next with Confluent, and we’re looking at incorporating streaming data with AWS Step Functions and Lambdas,” says Woo. “In addition, we have more training sessions planned for our teams with Confluent engineers, which will help us continue our adoption in an agile manner.”
Technical Solution
When Affin Hwang Asset Management began its digitization initiative, a top priority was to use solutions that fit well in a multi-cloud architecture. “Because we’re running on AWS and Azure currently, when we adopt a technology it has to work well in a multi-cloud environment,” says Woo. “So Confluent was a good fit for us.”
As they got started, the team had little experience with data streaming prior to the first proof of concept (PoC). “We understood the concept and we had experience with MSMQ, but we were new to Kafka,” says Woo. “That’s why the support we got from Confluent on the PoC was such a tremendous help.”
“The walk-through sessions led by Confluent engineers provided a valuable overview on data streaming and how to apply it to our use cases,” adds Hariz Farid, Unit Head, Rapid Development at Affin Hwang Asset Management. “And, the regular check-up calls we have with Confluent help us to keep moving forward and ensure we never stay stuck on some technical issue.”
For the PoC, the team wanted to move transaction data for investments and redemptions from multiple digital touchpoints— including its customer portal and mobile app—to a Snowflake data warehouse. They used microservices to publish the transaction data to a Kafka topic, and used Confluent’s fully managed Snowflake Sink Connector to persist the data to Snowflake.
The team uses ksqlDB to transform the data as it is in transit. “The data from our touchpoints is all JSON-based,” explains Farid. “So we use ksqlDB transformations to flatten it before it goes into Snowflake.” By merging and transforming the data sources with ksqlDB, Affin can generate a unified 360-degree view of their customers and provide the relevant data to different parts of the organization. ksqlDB has also helped streamline their internal processes and reduce their overall cloud data warehouse spend.
With the initial use case largely wrapped up, the team is now focused on streamlining the onboarding process with Confluent as momentum builds behind data streaming throughout the organization. “Currently, onboarding is one big interlinked process that involves suitability assessments, risk profiling, antimoney-laundering checks, and other steps,” says Woo. “We will be breaking down that process and others like it with Confluent, which will enable us to be more efficient and to get status updates for improved monitoring.”
Business Results
Time to market cut by months “With the resources and experts from Confluent pointing us in the right direction, we cut deployment time from as much as six months to just two weeks,” says Farid. “That saved us time in getting going with Apache Kafka® and enabled us to get to market faster.”
Day-long data access delays eliminated “Data that took us a whole day to get previously is now streaming via Confluent and available in seconds,” says Woo. “This is central to our digitization initiative because when a new customer is onboarded or a transaction is executed, that data needs to be reflected as soon as possible, and with Confluent we can do that.”
New business opportunities opened “Now that we’ve started streaming data with Confluent, we can explore offering new products—including products for which prices change more frequently,” says Woo. “Confluent is also making it easier for us to work with third-party eWallet providers and other partners who need near-real-time data.”
