Kafka is a message broker platform that can be used in event-driven application architectures. It has gained popularity because of its simplicity and speed. However, failing to choose a suitable optimization approach will mean that the performance of Kafka may suffer. This paper will describes various techniques that can be used to optimize Kafka.

I. Introduction

Kafka is a distributed system that allows for the creation and consumption of messages (sometimes known as documents). It was developed by a Java programmer, Emanuel Voss, in the early 2000s. Kafka is an open-source project and, as such, can be used by anyone.

Kafka was originally designed to solve the problem of batch-processing large amounts of data in real-time. The first version of Kafka was called Kafka 0.9 and was released in September 2009. It had support for topics, partitions, offsets, and write skew.

The next version of Kafka was released in May 2010 and added support for replication slots within a topic or partition. This allowed users to create multiple copies of their data without having to worry about losing any data due to network failures or other problems with replication flows between nodes in a cluster.

In 2012, Kafka 2.0 was released with support for multiple brokers per node (or “brokers”), which allowed users to scale up their systems without decreasing performance due to increased load on each broker, as well as increased latency due to split-brain failures caused by failures at either end.

I.1 Kafka architecture

Kafka is based on the idea that each message has its own identity and can be delivered to different consumers. The system consists of multiple servers, each of which handles one or more topics. These topics are then partitioned into groups called topic groups. There are no direct relations between messages in different topics. Instead, each message is processed independently and then sent to its final destination. This means that there are no cross-topic dependencies between messages within an individual partition or group, which makes it easy for developers to write code that scales as needed by increasing or decreasing the number of servers involved in processing requests from clients seeking data from those partitions/groups.

Kafka architecture consists of three main components: producer (for creating), consumer (for consuming) and broker (for distributing). The producer creates a topic that can store messages and then sends them to one or more consumers over time. Consumers read from topics, process their contents, and send back new messages to producers via brokers.

Kafka uses a leader-follower model to handle messages between nodes in a cluster. This means the leader is responsible for processing all the messages that it receives, while its followers only receive messages from their own follower nodes. The leader decides which follower will process each message based on its metadata properties, such as who sent it and where it originated. Kafka also provides an option for replicating data across multiple servers so that if one server fails then another server can take over seamlessly without losing any messages or state information.

The Zookeeper manages the state of a Kafka cluster, including replicas, leader election and replication. It manages configuration settings for Kafka cluster nodes, including their IP addresses, ports, node names and types (leader or follower), etc. It also maintains state information about the number of active topics and partitions across all nodes in the cluster. This allows the Kafka Zookeeper to failover when one node goes down without losing any data or losing any messages in transit between clients or servers that are connected via TCP/IP sockets or other network interfaces (e.g., HTTP clients).

The Kafka Leader is responsible for starting up new nodes in a Kafka cluster and assigning them to different roles within the cluster: leader (master) or follower (slave). Leader elections occur when one node becomes available as an active member of an existing group; if no leader exists already, then this node automatically becomes a leader when it starts up again after being restarted because there was no prior leader for its group.

I.2 Kafka Core Concepts

Broker

A broker is a single Kafka instance that receives messages from the producer. The messages will be assigned an offset value that will determine the order of the messages then stored in the partition.

Cluster

A cluster is a group of brokers to provide scalability, availability and fail-over. A message sent to the cluster will always be received by the leader and then replicated to the follower. When the leader is failing, a new leader will be elected from one of the available brokers. This mechanism is managed by the Zookeeper.

Topic

The topic is the unique name of a data set managed by Kafka. The producer sends a message to this topic while the consumer reads it. A topic can have one or more partitions assigned to it.

Partitions

A partition is a logical separation of messages inside a topic. This is useful to organize huge numbers of messages. The number of partitions is not managed by Kafka – it should be manually configured. In each partition, messages will be assigned an offset value. This offset value will determine the order of the message inside the partition.

Producer

The producer is an application that sends messages to Kafka. The producer decides to which partition the message will be sent by using the following rules:

• If the message doesn’t include a key, the producer will load balance the message to all partitions.
• If the message includes a key, the producer will use consistent hashing to map the key to destination partitions.

Consumer

The consumer is an application that reads messages from Kafka. The consumer reads messages in the Kafka partition in order. The order of the messages is determined by the message offset managed by Kafka.

Consumer Group

The consumer group is a group of consumers that reads messages from Kafka. There are two types of mechanisms that can be implemented by consumer groups:

• Fan out exchange.

Each group will read messages from Kafka topic. This is useful to broadcast messages so that the message will be available to each group.

• Order Guarantee.

Each consumer in the consumer group will read messages into a specific partition from Kafka topic. By utilizing consistent hashing, messages with the same key will always be sent to the same partition. Therefore, the order of the messages is retained.

II Kafka Optimization Techniques

Most Kafka clusters are configured to have a replication-factor of 3 and a minimum in-sync-replica of 2. This aim is to balance the durability and availability for general purposes. However, different applications might need different optimizations. Some applications might prefer having high availability and some might prefer high durability for data consistency.

Optimizing Kafka always refers to four factors:

• Durability.

Kafka stores messages on the disk. The message stored will be replicated in other brokers. This will make sure that the messages are stored in multiple brokers to avoid data loss.

• Availability.

Because Kafka stores and replicates messages in multiple brokers, this will ensure that the messages will always be available.

• Latency.

The amount of time needed to process a message starts from the message being sent until read. The lower the better.

• Throughput.

The number of messages and their size that can be processed within a given period of time.

The optimization will try to select the options from the above factors. One thing to note here is that optimizing one of the factors will sacrifice the others. For example, aiming for high durability might lower the availability, and aiming for high throughput might increase latency.

II.1 High Durability

In this scenario, Kafka is configured to have a higher number of replicas with a large number of producers. The producer can be set to use a higher acknowledgments value. This is to ensure that each message will be replicated to all replicas. The side-effect of this optimization is that it will increase the latency, because Kafka will prioritise the replication process first.

A typical use case for this scenario is in a Log Aggregation system that collects logs from multiple applications, databases, or infrastructure components. The logs will be stored in a centralized place so that they can be analysed and filtered.

Recommended Kafka configuration values:

Producer:
• replication.factor=3
• acks=all (default: all – default prior to Kafka 3.0: 1)
• enable.idempotence=true (default: true – default prior to Kafka 3.0: false), to prevent duplicate messages and out-of-order messages
• max.in.flight.requests.per.connection=1 (default 5), to prevent out of order messages when not using an idempotent producer

Consumer:
• enable.auto.commit=false (default true)
• isolation.level=read_committed (when using EOS transactions)

II.2 High Availability

In this scenario, Kafka is configured to have a small number of replicas with a small number of producers. This is to ensure that the data replication process is fast enough so the messages will be available across all Kafka nodes. The side-effect of this optimization is that it will reduce the replication process time and increase the throughput.

A typical use case for this scenario is in Messaging Systems where one application needs to communicate with others by exchanging messages. This information exchange is useful to decouple multiple applications where changes can be isolated in one or more applications without impacting others. Therefore, the consumer needs to receive the messages as fast as possible.

Recommended Kafka configuration value:

Consumer:
• session.timeout.ms: increase (default 10000)

Conclusion

Kafka provides a highly scalable and fault-tolerant distributed messaging system. Different optimization approaches are required based on the nature of the application itself. The optimization factors need to be considered because each of the optimization factors has opposing impacts. Kafka can be optimized to have a high durability (with the side effect of increasing the latency) which is suitable for applications that maintain data consistency. On the other hand, optimizing for high availability will reduce the number of replicas of the data, which is suitable for exchanging messages between applications.

References:
Ibryam, Bilgin. Fine-tune Kafka performance with the Kafka optimization theorem. Redhat Developer. https://developers.redhat.com/articles/2022/05/03/fine-tune-kafka-performance-kafka-optimization-theorem
Gwen et al. 2021. Kafka: The Definitive Guide, 2nd Edition. O’Reilly
Kreps, Jay. The Log: What every software engineer should know about real-time data’s unifying abstraction. LinkedIn Engineering. https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Optimize and Tune Confluent Cloud Clients. Confluent Developer. https://docs.confluent.io/cloud/current/client-apps/architecture.html

“If you want to go fast, go alone. If you want to go far, go together.”
– African proverb –

Why climb the career ladder by yourself when you can have support from the company you work for?  This time, we are focusing on the journey of Ida Bagus Gita Pradnyana, one of our skilful and experienced Analysts, highlighting how Mitrais has helped him prosper in his career.

Fresh out of college, Gita had no academic background in Software Engineering, instead relying on his degree in Mathematics from one of Indonesia’s top universities to apply for his dream role in Mitrais. Starting as a Junior Programmer, he received in-depth training in Software Engineering from the company – the solid foundation he needed to take his next career step.

Gita’s first role with Mitrais was as part of a maintenance team managing our internal systems. It did not take long for him to advance from a junior team member to a Team Lead. He also performed as Team Lead for some on-the-job training projects undertaken by groups of Junior Programmer inductees. As Gita explained, “These responsibilities were golden opportunities because they helped me shape my soft skills, including leadership, team management, and problem-solving earlier in my career. Often these opportunities are only available once you reach more senior levels.”

In Mitrais, promotions are based on merit, so individuals can influence how far and how fast their progress to upper thresholds can be made. But it is never a lonely journey, because Mitrais supports our staff when it comes to enhancing skills to accomplish goals. “Mitrais has provided comprehensive training and OJT to enable us to step up through each of the grades. There are a lot of opportunities to expand your experiences like certifications and weekly meetups where I get to attend a discussion on interesting topics in Software Engineering” he said.

In addition to technical skills, Gita has sharpened his communication skills, especially in the English language through the Business Communication Course delivered by Mitrais’ in-house trainers. These enhanced skills boost his confidence in dealing with international clients. “Just like the stereotype, software engineers like me can tend to be slightly introverted, communicating best with the computer. Communicating with real people is another story – it was challenging for me to communicate ideas I had so that they made sense to the listener. Fortunately, I attended a series of Mitrais English training on communication skills, including cultural differences between people from different countries.” recalled Gita. He added, “I am also grateful for the support and help from other business units at Mitrais. They are prompt and helpful, so I know I can always count on them whenever I need a helping hand.”

Ten years have passed, and the Junior Programmer has now transformed into an experienced and reliable Analyst. Recently, Gita received a Mitrais Gold Award for his loyalty and contribution to the company. Gita pointed out that the opportunity to tackle problems that are more complex in scope is the best thing about his current role. “We are expected to see potential roadblocks in a more holistic way, rather than as a isolated issue as we used to do when we were at junior levels. It is quite a challenge, but it is always very exciting to do.”

But Gita’s life is not just about coding. Born and raised in beautiful and vibrant Denpasar, he continues to develop his long-time interest in music. He loves to listen to many genres and attend music concerts held in his area.  With the beautiful scenery that Bali has to offer, Gita also enjoys photography. He is an active person, flexing his muscles by running, badminton and travelling. Gita is passionate about his hobbies, and spends his downtime doing things he loves. It’s all about a great work-life balance in Mitrais!

After three long years of the pandemic shaking the global economy, 2023 has started with many businesses still struggling to get back on their feet.

Amid the uncertainty, however, Mitrais remains as strong as ever, wrapping up a successful 2022. The result, of course, was the fruit of everyone’s commitment and perseverance, supported by well-established systems that provide added value to our clients’ businesses.

Mitrais held a festive combo event to celebrate the successful business year: a fun fusion of the hybrid online and offline Open Forum and happy hour at all the offices last December.

The Open Forum is Mitrais’ platform where our Management showcases the company’s updates while listening to any concerns from employees. Indeed, open conversations and transparency are always expected here. During the last Open Forum, the management team revealed discretionary bonuses for all – always music to everyone’s ears.

And the good news did not end there! On top of the bonuses, Mitrais awarded 122 employees with #mitraisgold awards for their 10- and 25-year outstanding service to and trust in Mitrais. The culture of appreciation that boosts employee morale is a tradition that Mitrais takes great pride in.

Nugroho Danang Sasongko, one of our Analyst Programmers who brought a gold awards home, was delighted. “It’s an absolutely wonderful and joyful event! I feel honored and appreciated to receive the Mitrais Gold for being with Mitrais for 12 years.” Time flies, indeed. “For 2023, I hope Mitrais will land even bigger conquests in both Indonesian and international markets!” wished the Yogyakarta-based Analyst.

Excited by the news and the gathering, Ika Putri Melindawati, our Bali-based Scheduling & Facilities Services officer also shared her impression of the event: “It’s such an amazing celebration, and I’m so thrilled because finally I can catch up with my colleagues, celebrate our wins together, and enjoy the feast during the happy hour. Ika was optimistic that Mitrais will exceed its target in 2023, “I wish an even brighter future for Mitrais and all the best for the next year!”

As we ended 2022 on such a positive note, all of us at Mitrais are delighted to welcome 2023 with the optimism to break yet another glass ceiling. We are looking forward to more growth and more people to board this new chapter with us.