Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming

What Is an API (Application Programming Interface)?

An API is a set of definitions or protocols that allows two or more applications to communicate with one another seamlessly. In this guide, learn what an API is, how it works, and the common types of APIs used today, with real-life examples.

API stands for “Application Programming Interface.” It allows two or more applications to communicate with one another through a predefined set of rules. In other words, APIs allow companies to serve their tools and services in a fast, simple way, by making it easy to share data, features, and functionality.

APIs are essential for modern software development. For example, when you send money through PayPal, call an Uber, or add someone on social media, you're using an API.

APIs do not expose the entire application's data. Instead, they share data in packets via specific requests, which ensures security and efficiency between different software components.

api_img

How APIs Work

The key idea behind an API is that an application doesn’t need to know the details about how another application works; it only needs to know how to use the API to access the right data, feature, or functionality. This is the concept of abstraction, which makes it possible for new programs to build on top of the hard work encoded into other programs.

APIs communicate as requests and responses between clients and servers. The application submitting the request is the client, and the server provides the response. The API acts as the bridge to establish the connection between them.

Here’s a step-by-step breakdown of how this process works:

  • Initiating a request: The client application sends a request to the API. This request comprises various elements, such as a request verb (GET, POST, PUT, DELETE), headers, and sometimes a request body, which contains the necessary data for the server to process.
  • Processing the request: The API receives the request and forwards it to the appropriate server or external program that can handle the request. This involves interpreting the request details and determining the appropriate action.
  • Generating a response: The server processes the request, retrieves or updates the necessary data, and then sends a response back to the API. This response includes the requested data or the result of the action taken.
  • Returning the data: The API then sends this response to the client application.

APIs allow different systems to interact without exposing any code or data structure.

API definition

A great example of an API is a video game. When the video game player presses buttons, the video game console interprets those inputs and turns them into actions in the game. The player only needs to know which buttons to press to play the game; they don’t need to know how the console understands those inputs, and how it uses them to render actions on the screen.

Another key characteristic of APIs is that they follow a modular approach, helping developers use existing services and data to speed up development processes.

There are many kinds of APIs and a mountain of technical definitions. Below, we discuss a few modern APIs with practical examples and some hands-on components.

Types of APIs

APIs can be categorized based on their use case: data APIs, operating system APIs, remote APIs, and web APIs.

  • Data (or database) APIs: Data APIs connect applications with database management systems and help in the retrieval, insertion, updating, and deletion of data.
  • Operating System (local) APIs: Operating system APIs define how applications interact with operating system services and resources. They allow applications to use functionalities such as file management, memory allocation, and process control.
  • Remote APIs: Remote APIs allow applications on different devices to communicate over a network and exchange between systems in different locations.
  • Web APIs: Web APIs enable data and functionality transfer over the internet using HTTP protocol. These APIs allow web services and applications to interact with each other.

Today, most APIs are web APIs. The four main types of web APIs are:

  • Open APIs: Open APIs are open-source application programming interfaces accessible via the HTTP protocol, with defined endpoints and request and response formats.
  • Partner APIs: Partner APIs connect business partners, typically via a public API developer portal.
  • Private APIs: Private APIs, also known as “internal APIs,” remain hidden from external users and are used to connect systems and data within an organization.
  • Composite APIs: Composite APIs combine multiple data or service APIs and allow developers to access several endpoints in a single call.

Now, let's explore some common types and examples.

api common types

REST API

The REST API is the most popular API architecture in modern web applications. REST stands for “Representational State Transfer.”

The underlying theory is nuanced, but in practice, a REST API means there is a client who makes certain kinds of requests to alter resources on a server. Each request represents the state the client wants the resource to have and attempts to transfer that wish to the server. The server responds on its own without having to remember anything about the client (this is known as a “stateless request”).

The most common place you will see REST APIs at play is in your browser. You click a button, and the browser sends a request to a server to retrieve a resource and display that resource on your screen.

REST APIs are sometimes described as ”RESTful.”

Example: cURL a web API

t’s difficult to conceptualize a REST API without a concrete example. Suppose there is a RESTful book API service hosted at https://api.fakebooksite.com. Users can create, read, update, and delete books using the API.

The cURL command is an application that can create HTTP requests and is commonly used to interact with REST APIs from the command line. Here is an example of a request one might submit to the books REST API server with the cURL application:

curl \
request GET \

https://api.fakebooksite.com/v1/authors?author=chuck&sort=title:desc

The curl command specifies that the request should use the “GET” HTTP method.

https://
  • This is the part of the URL that signals what protocol will be used. In this case, we use the secure HTTP protocol
    • api.fakebooksite.com
  • This part of the URL is called the domain. This is what humans use to find servers on the internet (as opposed to IP addresses).
    • /v1/authors/
  • This part of the URL is the REST API endpoint. It points to resources on the server. It is often prefixed by an API version so that the API can evolve over time without upsetting those who prefer to use older versions. This endpoint points to the “authors” resource.
    • ?author=chuck&sort=title:desc
  • This part of the URL is called the query string which starts with a “?” and encodes query parameters, which are key-value pairs defined with an “=” and separated with “&”. In this case, we specify the name of the author and ask for the response to be sorted by title name in descending (reverse) alphabetical order.

The REST API server application would be written in a way to respond to this kind of request and serve a response. In this case, the books API might retrieve information about books written by “chuck” from a database, perform a sort operation, and send a response to the cURL client in a JSON format.

JSON stands for JavaScript Object Notation. Despite its name, it is simply a standard file format that any language can use to send and receive data over the internet. The next example will send JSON data to a REST API endpoint.

TIP: APIs often respond with data encoded in JSON format. The jq command is a handy tool for parsing json responses. It is useful to pipe the results of a curl command into jq to see a pretty output, like this (-X is a shortcut for --request):

curl -X GET https://my-cool-site.com/v1/cool-quotes | jq

It’s great to receive data from a REST API endpoint, but we also want to send a “payload” of data to a REST API. Here is an example that creates/updates the book “Chuck’s Cool Book”:

curl \
--request PUT \
--header “Content-Type: application/json” \
--user chuck:chuck-password \
--data ‘{ “author”: “chuck”, “title”:”Chuck’s Cool Book”, “text”: “When       I was a boy, I ate 3 dozen eggs each morning…” }’ \    
  https://api.fakebooksite.com/v1/books 

There are a couple of new things here to notice: - --request PUT

  • We are using a different HTTP method. “PUT” usually means we want to provide data the server can use to create a resource or replace a resource if it already exists.
    • --header “Content-Type: application/json”
  • Headers are key-value pairs that give the server or client additional information about the request. There are many standard headers, like “Content-Type”. In this case, the client is telling the server that it is going to send a JSON data payload.
    • --user chuck:chuck-password
    • cURL’s --user option creates an “Authorization” header that includes username and password so the server can authenticate the client and decide if it is authorized to have its request fulfilled. In this case, it’s probably a good idea to require users to sign in before they are allowed to post books, otherwise anyone could stuff the database full of whatever books they want.
  • It’s also important that the protocol for this API is HTTPS rather than just HTTP. HTTPS means the connection between the client and server is encrypted, so no one can eavesdrop on the connection and steal the user’s password.
    • --data ‘{...}’

We use this option to compose our JSON payload to send to the server. In this case, we define author, title, and book text in JSON format to send to the server.

In summary, the above cURL command will send an HTTP PUT request to the server at the /v1/books endpoint with a JSON payload. When the server receives the PUT request, it authenticates the user, checks whether the user is authorized to create that resource, perhaps updates a database record for that book, and then returns a response to the client (e.g., status code 200 to report “success”).

NOTE: The server might behave very differently depending on which HTTP method is used. In this case, a GET request to https://api.fakebooksite.com/v1/books?author=chuck&title=Chuck%27s+Cool+Book might not require authentication at all, and give a response that includes the contents of “Chuck’s Cool Book” in JSON format.

Example: Confluent Server REST API

Confluent offers a REST API that allows users to interact with resources related to its data streaming platform. This provides a nice real-world example of what a REST API is and how it’s used.

Confluent Server includes a fully managed, cloud-native distribution of Apache Kafka, the popular open-source data streaming platform. Here is an example cURL command to Confluent Server’s REST API that creates a so-called “Kafka topic” (see https://developer.confluent.io/learn-kafka/apache-kafka to learn more about Apache Kafka, including what Kafka topics are):

curl --silent -X POST -H "Content-Type: application/json" \
--data '{"topic_name": "my-cool-topic"}' \
-u chuck:chuck-password \
http://localhost:8090/kafka/v3/clusters/7cteo6omRwKaUFXj3BHxdg/topics | jq

Here are a few things to notice about the cURL command:

  • We are using the HTTP “POST” method. This makes sense because we are creating a resource (a Kafka topic)
  • --silent
    • This just mutes cURL’s progress meter and error messages so you can focus on the response payload
  • -X is just a shortcut for --request
  • -H is just a shortcut for --header
  • -u is just a shortcut for --user
  • We pipe the output to th jq command to make it look prettier

The Confluent server could respond with a 401 status code to indicate the user’s authentication failed, a 403 status code to indicate the user is not authorized to create the topic, or a 200 to indicate success. Here is the response JSON payload upon success:

{
	"kind": "KafkaTopic",
	"metadata": {
		"self": "http://localhost:8090/kafka/v3/clusters/7cteo6omRwKaUFXj3BHxdg/topics/my-cool-topic",
		"resource_name": "crn:///kafka=7cteo6omRwKaUFXj3BHxdg/topic=my-cool-topic"
	},
	"cluster_id": "7cteo6omRwKaUFXj3BHxdg",
	"topic_name": "my-cool-topic",
	"is_internal": false,
	"replication_factor": 0,
	"partitions": {
		"related": "http://localhost:8090/kafka/v3/clusters/7cteo6omRwKaUFXj3BHxdg/topics/my-cool-topic/partitions"
	},
	"configs": {
		"related": "http://localhost:8090/kafka/v3/clusters/7cteo6omRwKaUFXj3BHxdg/topics/my-cool-topic/configs"
	},
	"partition_reassignments": {
		"related": "http://localhost:8090/kafka/v3/clusters/7cteo6omRwKaUFXj3BHxdg/topics/my-cool-topic/partitions/-/reassignment"
	}
}

This indicates the resource was successfully created and gives detailed information about it.

The entire API reference is here, including many cURL examples: https://docs.confluent.io/platform/current/kafka-rest/api.html

GraphQL API

GraphQL is an API query language that provides an alternative to REST. GraphQL gives clients a more flexible way to query an API for exactly what they need and provides developers with an easier way to evolve their APIs while maintaining backward compatibility.

GraphQL specifies three execution operations:

  • Query (retrieve data from the server)
  • Mutation (change data on the server)
  • Subscription (subscribe to a stream of data events from the server)

Hands-on: Querying Countries

Go to https://lucasconstantino.github.io/graphiql-online/ to play hands-on with a graphQL API.

Here is an example query that you can input into the GraphQL playground:

query{
countries(filter:{code:{regex:".*A.*"}}){
currency
code
name
continent{
  name
}
  }
  languages (filter:{code:{regex:".*r"}}) {
native
name
code
  }  
}

This example illustrates how GraphQL gives the client the power to decide what data they want the server to return. With a REST API, making a GET request to a /countries endpoint would return a lot of information the client doesn’t necessarily need, and it would be up to the client to parse through the response.

In addition, GraphQL empowers the client to choose data they want from a variety of resources in a single request. In this example, we also see that the client has requested information about languages. With a RESTful architecture, the client would have to make two requests, one to /countries and another to /languages.

Here are a few things to notice about the query:

  • The query is defined with the “query” keyword within curly braces {}
  • A filter can be applied so that only a subset of data is returned to the client E.g. for (filter:{code:{regex:".*A.*"}}), we select only the countries whose country code contains an “A” using regular expressions.

The GraphQL web editor, called GraphiQL, is sending an HTTP request to a real GraphQL endpoint. Here’s what an equivalent cURL command looks like (note the escape backslashes for double quotes and the lack of newlines in the query value):

curl -s -X POST -H "Content-Type: application/json" \
--data '
{"query": "query{countries(filter:{code:{regex:\".*A.*\"}}){currency
code name continent{name}}languages(filter:{code:{regex:\".*r\"}}
{native name code}}" }' [https://countries.trevorblades.com/
(https://countries.trevorblades.com/) | jq

Next, explore the API on your own. It’s especially helpful to use Ctrl+Space to take advantage of auto-complete. Also don’t forget to explore the docs panel on the upper right. These docs are automatically generated from GraphQL’s type system (yay schemas and auto generated documentation!).

Real-time GraphQL API with Apache Kafka and ksqlDB

GraphQL has a “subscription” operation that goes well conceptually with Apache Kafka’s data streaming nature. Here is a blog post that goes into detail about how to create a GraphQL service with Apache Kafka, including a hands-on proof of concept application that uses ksqlDB as the streaming database layer on top of Apache Kafka.

RPC API

Another popular alternative to REST API architecture is RPC, which stands for “Remote Procedure Call.” With an RPC API, the client sends a request to a server that asks the server to execute a “procedure” (i.e. “function,” or “method”).

RPC is different from REST because the endpoint in a REST API is a resource like /v1/books/, whereas the endpoint in an RPC API is an action like /v1/book.update. In other words, REST APIs are centered around nouns whereas RPC APIs are centered around verbs.

RPC APIs are popular for synchronous communication between microservices. The most popular framework for developing RPC APIs is gRPC.

Example: Fictional Store API

This example by Arnaud Lauret is useful to illustrate the difference between REST and RPC APIs:

Operation RPC (operation) REST (resource)
Signup POST /signup POST /persons
Resign POST /resign DELETE /persons/1234
Read person GET /readPerson?personid=1234 GET /persons/1234
Read person's items list GET /readUsersItemsList?userid=1234 GET /persons/1234/items
Add item to person's list POST /addItemToUsersItemsList POST /persons/1234/items
Update item POST /modifyItem PUT /items/456
Delete item POST /removeItem?itemId=456 DELETE /items/456

Source: https://apihandyman.io/do-you-really-know-why-you-prefer-rest-over-rpc/#examples

Notice RPC APIs only use POST and GET HTTP methods, choosing to let the endpoint describe what kind of operation is taking place. Compare that to REST, where DELETE, GET, POST, and PUT methods to the same resource endpoint will result in different behaviors.

Slack API

Slack is a popular enterprise collaboration app. Slack offers an RPC API to interact with its resources. Here is an example for setting a user’s profile data:

# Set user profile data
curl -X POST \
-H “Content-Type: application/json” -H “Authorization: Bearer my-super-secret-token” \
--data ‘{

"profile": {

   "status_text": "riding a train",
"status_emoji": ":mountain_railway:",
"status_expiration": 1532627506,
"first_name": "John",
"last_name": "Smith",
"email": "john@smith.com",
"fields": {

   "Xf06054BBB": {
"value": "Barista",
    "alt": "I make the coffee & the tea!"
}
}
}
}’ https://slack.com/api/users.profile.set?user=W1234567890 | jq

The JSON data payload is taken straight from the example in Slack’s documentation. Notice the endpoint /api/users.profile.set is an action rather than a resource. Here just a few other action-oriented RPC endpoints related to users:

  • users.profile.get
  • users.deletePhoto
  • users.setPhoto
  • users.setActive

Slack API documentation pages for their endpoints (called “methods”, which makes sense for RPC):

Native Library API

All the API examples so far have been APIs over the HTTP protocol (REST, GraphQL, RPC), but another common way to use APIs is directly in the code of your own application through libraries.

native library api

A library is a collection of objects and functions that you can import into your own application to use. A library is typically tailored to solve a specific problem. Instead of solving that problem yourself, you take advantage of the library’s capabilities by using its API. This makes more sense as you dig into examples.

Example: Python’s JSON Library

Here is a simple Python program that uses the API provided by the json library to serialize a Python dictionary into a json file.

import json

my_dictionary= {“my-key”:[1,2,3]}
with open(“my-file.json”, “w”) as f:
  json.dump(my_dictionary, f)

Here are a few things to notice:

  • import json
    • This is where we import the library. Every language provides ways to import libraries.
  • json.dump(my_dictionary, f)
    • We can research the library’s official API documentation to find the functions and objects we need for our program.
    • In this case, we find that the dump command does what we want — it takes a Python object and serializes it into JSON file format.
    • The API specifies that dump has two required positional arguments — first, the object to be serialized, and second, a file writer object to write the JSON data to a file.
  • The important point is that we can use the dump function in our program without knowing at all how it is implemented.

Apache Kafka Producer API

Apache Kafka provides APIs to read from, process, and write data to Kafka servers (called "brokers"). There are Kafka client libraries available in many programming languages. We’ll use Java to contrast with the previous Python example.

…
import import org.apache.kafka.clients.producer.*;
…

public class ProducerExample {

public static void main(final String[] args) throws IOException {
…
final Properties props = new Properties();
InputStream propsFile = new 
FileInputStream("src/main/resources/producer.properties");
props.load(propsFile);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
String t = “my_topic”
String k = "mykey";
String v = "myvalue";
ProducerRecord<String, String> record = new ProducerRecord<String, String>(t, k, v);
producer.send(record);
}

The pattern is the same as the previous example:

  • import the library
  • research the API documentation to learn what objects and functions are available and how to use them
  • Profit

In this case, we

  • create a KafkaProducer object called “producer” configured with a properties file
  • Define a topic, key, and value
  • Create a ProducerRecord object called “record” that holds the record key, record value, and the topic where we want to send the record

Best API Tools

Here are some of the most popular tools for developing and using APIs:

Search hubs for APIs others have made:

Tools to interact with APIs:

Popular frameworks for developing web APIs:

Why Confluent?

Built by the original creators of Apache Kafka, Confluent offers a scalable data streaming platform to build scalable, real-time, event-driven applications. You benefit in many ways by using events as the basis for connecting your applications and services, gaining loose coupling, service autonomy, elasticity, flexible evolvability, and resilience.

You can use the APIs of Kafka and its surrounding ecosystem, including ksqlDB, for both subscription-based consumption as well as key/value lookups against materialized views, without the need for additional data stores. The APIs are available as native clients and over REST.

To learn more about data streaming and event-driven software architecture, check out Confluent Developer. Or get started with Confluent on any cloud, on any scale, in minutes. New users get $400 free to spend.

More Resources

Developer Mozilla

Excellent reference for all things HTTP, like headers, methods, status codes, etc.

Confluent Developer

The one-stop shop to learn about event-driven software architecture, Apache Kafka, and Confluent