So, what is Apache Kafka? Kafka software is an open-source, distributed event streaming platform that lets applications share data with each other in real time.

Why do we need it?

Suppose we have many applications, and each one needs data from other applications. We usually send data through API to other applications to achieve that goal.

Based on the above image, Order sends a couple of applications for every transaction. What happens if, during the next update, more applications need similar data from Order? We will need some improvement to add more endpoints. Or, what if one of the endpoints is down? It will not reach one of the endpoints. To solve these problems, we can use Apache Kafka as a message broker. 

In the image above, the application that sends data to Kafka is the Publisher (Kafka calls it a producer), and it writes that data to a topic. Any application that needs the data reads it by subscribing to that topic. These are the Subscribers (Kafka calls them consumers). The following application that needs similar data will only have to subscribe to a particular topic. As long as the data has been sent to Kafka, there is no need to improve the respective producer. In other conditions, if the producer forwarded data to the message broker while the consumer was down, Kafka will save the last data offset/position at the point where the consumer last consumed the data. When the consumer is up again, Kafka will send the data immediately.

Here are the definitions of the terms in Kafka:

1. Topic 
   The topic is a category name to which records are stored and published. Producers are the writers for topics, and consumers read from them. 

2. Topic Partition 
   Topic Partitions contain producers’ messages where each consumer consumes the message from one topic partition. Within a consumer group, each partition is read by only one consumer. So if a group has five consumers, the topic needs at least five partitions for every consumer to receive messages.

3. Offset 
   Records exist in a partition assigned and identified by a unique offset. 

4. Replica 
   A replica is a redundant unit of a topic partition implemented at the partition level. Partitions have one or more replicas to prevent data loss. Every partition replica has one replica acting as a leader, and the rest are a follower. The leader replica handles read-write requests, and followers replicate the leader. 

5. Consumer Group 
   Consumer groups have names to identify them from other consumer groups. The primary purpose is to distribute incoming data among multiple consumers and prevent numerous consumers in consumer groups from consuming the same data. 

6. Retention 
   Time-Based Retention marks a message for deletion. The default time-based retention period is seven days. Size-based retention sets the maximum size a partition can reach before Kafka deletes the oldest messages.

Another use case that might show up is when multiple instances of the same application consume the same message and proceed. Running several instances makes processing faster, but it causes problems when a message should only be processed once, such as a payment.

In the above image, two payment instances are needed to process the Payment from Order, and payment applications will proceed with two payments simultaneously. Kafka provides consumer grouping to prevent the message consumed two times or more by the consumer.

In this case, the consumer will have a consumer group name based on the transaction category. For example, let’s name it “Payment Consumer Group”. Every message sent to Kafka will be consumed by only one instance in its group. 

What is the Publisher-Subscriber pattern? 

Publishers are loosely coupled to subscribers. Commonly, the client cannot hit the endpoint of a particular server while its server is down. Using the Publisher-Subscriber pattern, each client can continue to operate independently. Also, in Kafka, horizontal or even vertical scaling are available. We can add more partitions to a topic for a horizontal scale when we need more subscribers to consume the message from a particular topic. We can also do vertical scale by implementing multithreading in subscribers, and each thread consumes the message from a specific partition.

What is Apache Kafka used for?

Kafka software fits wherever several applications need the same data without depending on each other directly. Common uses include:

• Real-time data pipelines that move events between systems as they happen
• Asynchronous communication between microservices, where each service publishes and consumes events independently
• Log aggregation and activity tracking across multiple applications

If you are connecting several platforms that need to share data reliably, our system integration services can help you work out whether event streaming is the right fit.

Conclusion 

The distributed streaming platform, Apache Kafka, helps applications to share information with other applications using Publish-Subscribe. It’s important to remember that if a consumer group has more consumers than the topic has partitions, the extra consumers will sit idle. Now that we have covered what Kafka is and how it works, the next step is deciding where it fits in your own architecture. If you want to try it hands-on, our guide on how to set up a Kafka environment walks through the installation.

Author: Handoyo, Analyst Programmer