Connecting the Dots: Simplifying Multi-API Data Flows into Apache Kafka®

In today’s data-driven software-as-a-service (SaaS) environments, the need for complete customer insights often requires fetching and sharing data that lives across multiple API endpoints. That’s why many of our customers want to use Confluent’s data streaming and integration capabilities to implement real-time API chaining—a technique that allows them to automatically follow relationships between APIs.

For example, user data might come from one API, while its activity or interaction history is retrieved from another. Integrating this scattered but related data can be complex—especially once you add Apache Kafka® to the architecture—but it’s still essential for real-time analytics, personalization, or compliance. Chaining these APIs together would allow you to fetch a user and then immediately pull their related records, such as transactions or visited locations. 

With recent updates to Confluent’s HTTP Source V2 connector, this pattern can now be implemented declaratively through configuration alone—with no custom code or external orchestrators needed. The result? Real-time ingestion of enriched, connected data into Apache Kafka with minimal effort.

This blog post covers:

• How API chaining works and how it compares to similar patterns

• Why and when to implement API chaining with Apache Kafka

• How parent-child data relationships work with real-time API chaining

• How to implement real-time API chaining declaratively with Confluent

Get started on Confluent Cloud for free and try API chaining with the HTTP Source V2 connector.

What Is API Chaining, and How Does It Work?

API chaining is a technique in which the output of one API call is used as the input for another. Typically, data from the first response, such as an ID, is passed along as a path or query parameter in subsequent requests. This creates a sequence of dependent calls that traverse or assemble related data across multiple endpoints. It enables you to build relationships between endpoints and extract deeply nested or related data.

When (and Why) to Use API Chaining

As a general rule of thumb, you should aim to minimize the number of remote API calls required to retrieve a given data set. For example, a list API endpoint may sometimes be parameterized to include detailed information about each item, avoiding multiple follow-up requests.

However, when such options aren’t available or when a single endpoint would return an unwieldy amount of data, API chaining can be a practical alternative. It allows you to progressively fetch and enrich data across multiple endpoints, effectively composing a complete dataset in real time.

While this approach offers flexibility and can simplify integration logic, it’s important to keep an eye on potential downsides such as increased latency, rate limiting, and dependency failures across chained calls. Thoughtful batching and pagination strategies—which are configurable in API chaining—can help mitigate these issues and keep performance predictable.

Applying API Chaining to Parent-Child Data Relationships

Think of API chaining as a treasure map in which each API response gives you a value that helps you dig deeper into the next layer of data. To understand how API chaining works, we’ll look at a common pattern found in modern SaaS applications—parent-child data relationships exposed through REST APIs—and discuss how to implement them traditionally versus with Kafka.

Let’s say you have two APIs:

• api1 → /users – returns a list of all users in the system

• api2 → /users/${id}/places – returns the places visited by a specific user

GET /users

The parent API endpoint retrieves all the users registered in the system.

GET /users/${id}/places

Then the child API endpoint returns all the places visited by a specific user, identified by their unique ID.

This is a typical structure across industries: a primary entity (e.g., a user, customer, patient) and its associated sub-resources (e.g., locations visited, orders placed, appointments scheduled). It’s a simplified but realistic pattern that captures the essence of how hierarchical data is modeled in RESTful systems.

Streaming API Requests into Apache Kafka

The connector continuously pulls fresh data from the APIs, enriches it with user context, and delivers it into Kafka as a unified event stream. This approach decouples the data producers (the APIs) from the data consumers (analytics, monitoring, or storage systems), enabling scalability and flexibility. As new users or places are added, the data pipeline automatically adapts, ensuring that downstream systems always have access to the most up-to-date view of user activity.

The aim here is to build an integration that collects all places visited by all users in the system. Here are the steps involved:

1. Call the parent API.

   ‎ 

   Fetch the list of users:

   [  
     { "id": "user101", "name": "Alice" },
     { "id": "user102", "name": "Bob" }
   ]

   ‎ 

2. Call the child API for each user.

   ‎ 

   Use each user’s ID to fetch the places they’ve visited:

   ‎ 

   GET /users/user101/places

   ‎ 

   GET /users/user102/places

   ‎ 

3. Aggregate the results.

   ‎ 

   Stream the data for all users into Kafka. Each record—representing a place visited by a user in JSON format—can be published to a Kafka topic, making it available for downstream consumers like analytics engines, data pipelines, or real-time dashboards. The child API response implicitly has the overall data, which in this case means a combination of places and user ID data points.

Each record returned from the parent API and child API is published to a Kafka topic. This enables:

• Real-time processing and analytics

• Seamless integration with downstream systems

• Efficient ingestion of nested API data structures into your event streaming platform

Enriching and Querying Chained API Data in Real Time With Apache Flink® 

Taking the above example, once the user and places data is ingested into separate Kafka topics via API chaining, tools such as Apache Flink® can be used to join and enrich these streams in real time. For example, user metadata can be combined with places visited to produce a complete view of user activity.

Common Kafka and Flink Use Cases for Multi-API Data Flows

API data flows with Kafka and Flink unlock powerful real-time integration patterns across virtually every industry. The ability to automatically chain related API calls and stream the enriched results into Kafka creates opportunities for deeper insights and more responsive business operations. Whether you're building customer 360 views, operational dashboards, or compliance reporting systems, API chaining eliminates the traditional barriers between siloed data sources.

The following examples demonstrate how organizations can leverage API chaining to solve real-world data integration challenges. Each pattern shows how parent-child relationships in APIs can be transformed into unified, real-time data streams that power analytics, machine learning, and operational intelligence across an enterprise.

eCommerce Orders and Items: Enrich order data with line items for inventory, fulfillment, and customer insights.

Hierarchical Data Fetching: Traverse nested resources (e.g., organizations → teams → members) in systems that don't support joins.

Logistics, Shipments, and Tracking Events: Capture both shipment metadata and real-time tracking updates to monitor delivery progress and operational performance.

Finance Customer Activities and Transactions: Get customer financial records alongside transaction histories for reporting, billing, or customer service.

Internet of Things (IoT) and Device Management: When managing devices or sensors, you might need to pull additional metadata, location info, or historical stats through chained endpoints.

Example : GET /devices → GET /devices/{id}/status → GET /locations/{location_id}

How to Easily Implement Multi-API Data Flows With Confluent

In systems like the above scenario, in which APIs are structured hierarchically, API chaining can be an incredibly powerful pattern in theory but incredibly complex in execution. In most integration scenarios, pulling data from just one endpoint isn’t enough. You often need to combine data from multiple related APIs (with the same base URL) to gain a full picture—for analytics, enrichment, monitoring, or personalization. 

While it's technically possible to build this logic yourself, doing so often means managing multiple layers of complexity: looping, stateful pagination, rate limiting, retries, and error handling. API chaining abstracts all of that, enabling teams to focus on delivering insights rather than plumbing infrastructure.

Whether you're enriching data pipelines or feeding a data lake, the HTTP Source V2 connector offers a powerful, low-friction way to bring structured API data into Kafka. The HTTP connector facilitates communications between Kafka client applications and HTTP-based services such as Marketo, Mailchimp, and ServiceNow. With this connector serving as a bridge between Kafka and your cloud services, you can enable communication between applications and any HTTP-based services you need to stitch together.

To implement API chaining in a reliable and scalable manner, customers should onboard to the HTTP Source V2 connector, which supports a wide range of HTTP-based ingestion patterns, including hierarchical API relationships via configuration.

This connector simplifies the chaining process—with no custom scripting or additional orchestration needed. Instead, you can declare how one API call leads to another using a straightforward config property.

Defining API Relationships With the HTTP Source V2 Connector

Let’s go back to our original parent-child API example. To chain api1 (/users) and api2 (/users/${id}/places) together with the HTTP Source V2 connector, you can use the api.chaining.parent.child.relationship property in the connector configuration:

{
  "kafka.api.key": "********",
  "kafka.api.secret": "************",
  "kafka.auth.mode": "KAFKA_API_KEY",
  "name": "API Chaining - Http v2 Source",
  "connector.class": "HttpSourceV2",
  "output.data.format": "AVRO",
  "tasks.max" : "2",
  "apis.num": "2",
  "http.api.base.url": "https://example.com",
  "api.chaining.parent.child.relationship": "api1::api2",
  "api1.http.api.path" : "/users",
  "api1.topics": "users-topic",
  "api2.http.api.path" : "/users/${id}/places",
  "api2.topics": "places-topic"
}

How it works:

1. The connector first calls api1 (/users) to retrieve all users.

2. For each user object, it extracts the id field.

3. It then calls api2 (/users/${id}/places) for each extracted id, replacing the ${id} placeholder dynamically.

4. All responses from api2 are published to the configured Kafka topic.

This chaining behavior is driven entirely through configuration. There’s no need to manually handle iteration, substitution, or sequencing. And the connector allows you to automatically substitute IDs from the parent response into child API calls, removing the need for custom scripting or manual data stitching. More importantly, it handles offset management under the hood, ensuring that both parent and child datasets are fully traversed without dropping records. 

Real-World Use Case: Parent-Child API Chaining in Action

API chaining unlocks real-time, enriched data pipelines across a wide range of domains. Let’s put what you’ve learned about real-time API chaining with Confluent into action and transform raw API responses into meaningful, structured data streams in Kafka.

Enriching ServiceNow Incidents With Work Notes

In IT service management, visibility into how tickets are resolved is just as important as the ticket status itself. By chaining APIs in ServiceNow, we can capture and analyze work notes tied to each incident.

Operational Insights

• Analyze how long incidents take to resolve.

• Identify high-effort tickets by note count or type.

• Spot recurring technical issues based on patterns in work notes.

  ‎ 

Compliance and Service Level Agreement (SLA) Tracking

• Ensure that agents follow resolution steps and documentation standards.

• Audit incident handling for regulatory or internal compliance.

• Improve accountability and transparency across support teams.

  ‎ 

API Chain:

• api1 – returns a list of incident tickets, each with a sys_id that uniquely identifies the incident

  ‎ 

  GET /api/now/table/incident

  ‎ 

• api2 – returns all work notes associated with a given incident

  ‎ 

  GET /api/now/table/sys_journal_field?sysparm_query=element_id=${sys_id}^element=work_notes

Connector Configuration:

{
  "kafka.api.key": "********",
  "kafka.api.secret": "************",
  "kafka.auth.mode": "KAFKA_API_KEY",
  "name": "API Chaining - ServiceNow",
  "connector.class": "HttpSourceV2",
  "output.data.format": "AVRO",
  "tasks.max" : "2",
  "apis.num": "2",
  "http.api.base.url": "https://instance-id.service-now.com",
  "auth.type": "BASIC",
  "connection.user": "admin",
  "connection.password": "**********",
  "api.chaining.parent.child.relationship": "api1::api2",
  "api1.http.api.path" : "/api/now/table/incident",
  "api1.topics": "users-topic",
  "api2.http.api.path" : "/api/now/table/sys_journal_field?sysparm_query=element_id=${sys_id}^element=work_notes",
  "api2.topics": "places-topic"
}

• api1 fetches all incidents.

• The connector extracts the sys_id from each incident.

• api2 is called for each sys_id to pull the related work notes.

• These notes are then streamed into Kafka for visibility and analysis.

Further, Flink can be leveraged to find all work notes for a particular incident and to provide a complete picture of the incident life cycle, enabling support leads, site reliability engineers, or auditors to instantly trace resolution steps for any ticket without scanning logs or stitching data manually.

Once incident and work note data is flowing into Kafka via API chaining, you can apply artificial intelligence and machine learning to unlock deeper insights. For example, using built-in functions like ML_PREDICT() or other streaming machine learning frameworks, you can analyze incident resolution patterns in real time, detecting recurring issues, predicting SLA breaches, or recommending next steps based on past resolutions. This brings intelligent automation to IT service management, enabling proactive support and faster root cause identification.

By chaining incident and journal APIs in ServiceNow, teams gain a 360-degree view of support workflows, which is critical for improving SLAs, optimizing staffing, and demonstrating compliance.

Wrapping Up: Unlocking the Power of API Chaining

When combined with Kafka and Flink, API chaining can be a powerful way to construct complete, contextual, and connected datasets from APIs that were never designed to work together. 

It also simplifies the handling of real-world API behaviors by supporting batching and a variety of pagination mechanisms—whether offset-based or cursor-based—ensuring reliable and efficient data retrieval across large datasets.

Whether you're syncing customer profiles from HubSpot and appending purchase behavior from Stripe, enhancing IoT telemetry streams with weather API context, or combining GitHub commits with deployment logs for continuous integration/continuous deployment traceability, Confluent’s HTTP Source V2 connector enables you to:

• Automate multistep data collection

• Preserve relationships across nested APIs

• Stream real-time, enriched data into Kafka for deeper analysis

• Turn complex workflows into simple, declarative pipelines

Ready to Get Started?

If you're already using Kafka Connect, consider onboarding to the HTTP Source V2 connector to unlock API chaining and other advanced features, such as pagination, transformation, and schema evolution.

Explore the official documentation to configure your first chained API flow or reach out to your Kafka platform team to learn how this fits into your existing data pipelines.

Apache®, Apache Kafka®, Kafka®, Apache Flink®, and Flink® are registered trademarks of the Apache Software Foundation. No endorsement by the Apache Software Foundation is implied by the use of these marks.