fbpx

Software Development Newsletter: Q1 2023

Director Message

kafka

Welcome to the first Mitrais Newsletter of 2023. There is no doubt that Global Economies still face significant challenges as we enter the new year. Geopolitical unrest, surging fuel prices, challenging recruitment environments and inflationary pressures make progressing businesses difficult, but rewards those who exhibit good business fundamentals, and are agile enough to adapt to the new world. It is heartening to see that Mitrais and our client’s number themselves amongst those who are riding out the heavy weather, and steadily growing despite the environment.

As the opening article, we present the case study of our client with the Olive Web project in CAC Corporation. Get to know the barriers and struggles we overcame together. With the help of our expert team and cutting-edge technology, the client was able to achieve remarkable success. This article will motivate you to take your own business to new heights.

What is a Mitrais Newsletter without an opportunity to learn more about an exciting current technology that Mitrais is deploying for clients right now? Dig little deeper into Kafka, initially developed to handle LinkedIn log files, but now an established streaming platform of its own, enabling unified, high-throughput, low-latency platform for handling real-time data feeds. We’re thrilled to take you on an exploration of Kafka and show you how to optimize it for maximum results.

In this edition’s Company Update, you can read about how we remain committed to achieving goals alongside our clients. As we continue to grow, Mitrais’ Long-Term High-Trust credo not only extends to our clients but also to our valuable staff. With more than 100 Mitrais staff recently celebrating more than 10 years (and, in some cases, up to 25 years) as part of the Mitrais family, it is clear that the phrase “Our staff are our most important asset” is more than just a marketing cliché at Mitrais – it is a philosophy we live by!

A perfect example of what makes Mitrais successful is the story of Ida Bagus Gita Pradnyana (Gita). While Gita came to Mitrais from college with no background in software engineering at all, he found a place that provided him with stimulating work that excited him, and the structured development that allowed him to steadily grow his career. His journey from fresh graduate to experienced Analyst is a positive and uplifting one that I’m sure you will find inspiring. Learn more about his story in this edition’s Feature Staff Member article. As always, I hope that you enjoy this newsletter, and we wish you all continued health and prosperity for 2023 and beyond.

The Challenges and Solutions of Collaborative Agile Development - CAC Olive Project

Mitrais did a good job for the CAC Olive web application, using Agile methodology. We worked together with their 20 professional and skilled software engineers for more than a year to finish this project. Despite the language and time zone differences, we appreciate their effort to get the project done. We enjoyed working with them and look forward to another project collaboration in the future.”

Royko Seino,

Manager CAC Corp – Olive Web, Japan

Since becoming a member of the CAC Holdings Group family a few years ago, Mitrais’ successful track record with complex software development was something everyone sought to leverage. Of particular interest was Mitrais’ specialist skills in Agile (specifically SCRUM-based) development methodologies. It was hoped that CAC teams could be exposed to SCRUM working in conjunction with experienced Mitrais team members. It was in this environment that the Olive Project was selected as an ideal candidate development.

 

The Background

CAC Olive was envisaged as a web application designed to facilitate executive schedule management supporting the administration office and making it possible to streamline, centralise and standardise their work. Encompassing a full-featured scheduling system, Olive would also manage customer information, event and gift management, visitor management and hospitality functions. Olive needed to support

major PC browsers, as well as iPhone/iPad and Android clients. CAC’s person in charge for Project Olive is Seino-san, General Industry Solution Dept. II and Junior Service Producer.

 

The Challenge

Undertaking a complex project like Olive was not without it’s challenges. As Seino- san said, “Due to the complexity and large number of functions, we thought that specification consistency and quality assurance would be a challenge. Also, since we were new to Agile development, we thought that the difference in methodology and awareness from waterfall would also be a challenge”. Another potential complexity was that of language and communications. Being a “hybrid” team comprising native Japanese speakers and Bahasa Indonesia/English speakers meant that clear and effective communications would be of utmost importance, especially so since the Agile methodology was not a familiar one to some team members. “Of course, technical capabilities are important. However, since the system would be linking multiple functions, we knew that collaboration would be important” Seino-san said.

All of this was to be achieved bridging 3 time-zones (Tokyo, Jakarta and Bali) and against the backdrop of the COVID-19 pandemic that was such an issue when the project started in February 2021.

 

The Solution

The technology stack selected for Project Olive reflected best practice for this type of complex web application, including NodeJS for the back-end, and ReactJS for the

front-end. SQL Server was identified as the database of choice, with Jenkins (DevOps) and Sonarqube/Jmeter as quality systems.

Given the scope of Project Olive, it was decided to establish 2 SCRUM teams (totalling 21 developers) comprising full-stack, front-end and back-end developers, and dedicated Quality Assurance staff from both organisations. The teams would schedule alternating Sprints to maximise project velocity, but this meant that Sprint planning and management was critically important. To support such an ambitious schedule, the teams were overseen by Product Owners from CAC, and a team of Senior Manager, SCRUM Master, Technical Evangelist and UI\UX Designer from Mitrais.

From the start, concerns about the effective specification of Project Olive were addressed. Seino-san was pleased with the team’s approach. “They solved the problem by gathering information to the representative members. They also pointed out inconsistencies and omissions in the specifications that we had missed, which helped us to prevent problems in advance” she said. “We are grateful that the team was brought together by an experienced Senior Manager.”

The CAC/Mitrais team also brought with it other key benefits. “The ability to respond to development in a flexible manner. Since we were not used to agile development, the project sometimes proceeded a little like waterfall. However, we were able to finish the development with the precise advice from Mitrais” Seino-san said.

So how does Seino-san view the team’s performance in retrospect? “The speed at which the sprint progresses within a short cycle is something we hope to utilise in the future” she says.

Given the success of the Olive Project, we would love to work together (with Mitrais) in the future if the opportunity arises”.

Kafka Optimization Techniques

Kafka is a message broker platform that can be used in event-driven application architectures. It has gained popularity because of its simplicity and speed. However, failing to choose a suitable optimization approach will mean that the performance of Kafka may suffer. This paper will describes various techniques that can be used to optimize Kafka.

I. Introduction

Kafka is a distributed system that allows for the creation and consumption of messages (sometimes known as documents). It was developed by a Java programmer, Emanuel Voss, in the early 2000s. Kafka is an open-source project and, as such, can be used by anyone.

Kafka was originally designed to solve the problem of batch-processing large amounts of data in real-time. The first version of Kafka was called Kafka 0.9 and was released in September 2009. It had support for topics, partitions, offsets, and write skew.

The next version of Kafka was released in May 2010 and added support for replication slots within a topic or partition. This allowed users to create multiple copies of their data without having to worry about losing any data due to network failures or other problems with replication flows between nodes in a cluster.

In 2012, Kafka 2.0 was released with support for multiple brokers per node (or “brokers”), which allowed users to scale up their systems without decreasing performance due to increased load on each broker, as well as increased latency due to split-brain failures caused by failures at either end.

I.1 Kafka architecture

Kafka is based on the idea that each message has its own identity and can be delivered to different consumers. The system consists of multiple servers, each of which handles one or more topics. These topics are then partitioned into groups called topic groups. There are no direct relations between messages in different topics. Instead, each message is processed independently and then sent to its final destination. This means that there are no cross-topic dependencies between messages within an individual partition or group, which makes it easy for developers to write code that scales as needed by increasing or decreasing the number of servers involved in processing requests from clients seeking data from those partitions/groups.

Kafka architecture consists of three main components: producer (for creating), consumer (for consuming) and broker (for distributing). The producer creates a topic that can store messages and then sends them to one or more consumers over time. Consumers read from topics, process their contents, and send back new messages to producers via brokers.

kafka

Kafka uses a leader-follower model to handle messages between nodes in a cluster. This means the leader is responsible for processing all the messages that it receives, while its followers only receive messages from their own follower nodes. The leader decides which follower will process each message based on its metadata properties, such as who sent it and where it originated. Kafka also provides an option for replicating data across multiple servers so that if one server fails then another server can take over seamlessly without losing any messages or state information.

The Zookeeper manages the state of a Kafka cluster, including replicas, leader election and replication. It manages configuration settings for Kafka cluster nodes, including their IP addresses, ports, node names and types (leader or follower), etc. It also maintains state information about the number of active topics and partitions across all nodes in the cluster. This allows the Kafka Zookeeper to failover when one node goes down without losing any data or losing any messages in transit between clients or servers that are connected via TCP/IP sockets or other network interfaces (e.g., HTTP clients).

The Kafka Leader is responsible for starting up new nodes in a Kafka cluster and assigning them to different roles within the cluster: leader (master) or follower (slave). Leader elections occur when one node becomes available as an active member of an existing group; if no leader exists already, then this node automatically becomes a leader when it starts up again after being restarted because there was no prior leader for its group.

kafka

I.2 Kafka Core Concepts

Broker

A broker is a single Kafka instance that receives messages from the producer. The messages will be assigned an offset value that will determine the order of the messages then stored in the partition.

Cluster

A cluster is a group of brokers to provide scalability, availability and fail-over. A message sent to the cluster will always be received by the leader and then replicated to the follower. When the leader is failing, a new leader will be elected from one of the available brokers. This mechanism is managed by the Zookeeper.

Topic

The topic is the unique name of a data set managed by Kafka. The producer sends a message to this topic while the consumer reads it. A topic can have one or more partitions assigned to it.

Partitions

A partition is a logical separation of messages inside a topic. This is useful to organize huge numbers of messages. The number of partitions is not managed by Kafka – it should be manually configured. In each partition, messages will be assigned an offset value. This offset value will determine the order of the message inside the partition.

kafka

Producer

The producer is an application that sends messages to Kafka. The producer decides to which partition the message will be sent by using the following rules:

  • If the message doesn’t include a key, the producer will load balance the message to all partitions.
  • If the message includes a key, the producer will use consistent hashing to map the key to destination partitions.

Consumer

The consumer is an application that reads messages from Kafka. The consumer reads messages in the Kafka partition in order. The order of the messages is determined by the message offset managed by Kafka.

Consumer Group

The consumer group is a group of consumers that reads messages from Kafka. There are two types of mechanisms that can be implemented by consumer groups:

  • Fan out exchange.

Each group will read messages from Kafka topic. This is useful to broadcast messages so that the message will be available to each group.
kafka

  • Order Guarantee.

Each consumer in the consumer group will read messages into a specific partition from Kafka topic. By utilizing consistent hashing, messages with the same key will always be sent to the same partition. Therefore, the order of the messages is retained.
kafka

II Kafka Optimization Techniques

Most Kafka clusters are configured to have a replication-factor of 3 and a minimum in-sync-replica of 2. This aim is to balance the durability and availability for general purposes. However, different applications might need different optimizations. Some applications might prefer having high availability and some might prefer high durability for data consistency.

Optimizing Kafka always refers to four factors:

  • Durability.

Kafka stores messages on the disk. The message stored will be replicated in other brokers. This will make sure that the messages are stored in multiple brokers to avoid data loss.

  • Availability.

Because Kafka stores and replicates messages in multiple brokers, this will ensure that the messages will always be available.

  • Latency.

The amount of time needed to process a message starts from the message being sent until read. The lower the better.

  • Throughput.

The number of messages and their size that can be processed within a given period of time.

The optimization will try to select the options from the above factors. One thing to note here is that optimizing one of the factors will sacrifice the others. For example, aiming for high durability might lower the availability, and aiming for high throughput might increase latency.

kafka

II.1 High Durability

In this scenario, Kafka is configured to have a higher number of replicas with a large number of producers. The producer can be set to use a higher acknowledgments value. This is to ensure that each message will be replicated to all replicas. The side-effect of this optimization is that it will increase the latency, because Kafka will prioritise the replication process first.

A typical use case for this scenario is in a Log Aggregation system that collects logs from multiple applications, databases, or infrastructure components. The logs will be stored in a centralized place so that they can be analysed and filtered.

Recommended Kafka configuration values:

Producer:
• replication.factor=3
• acks=all (default: all – default prior to Kafka 3.0: 1)
• enable.idempotence=true (default: true – default prior to Kafka 3.0: false), to prevent duplicate messages and out-of-order messages
• max.in.flight.requests.per.connection=1 (default 5), to prevent out of order messages when not using an idempotent producer

Consumer:
• enable.auto.commit=false (default true)
• isolation.level=read_committed (when using EOS transactions)

II.2 High Availability

In this scenario, Kafka is configured to have a small number of replicas with a small number of producers. This is to ensure that the data replication process is fast enough so the messages will be available across all Kafka nodes. The side-effect of this optimization is that it will reduce the replication process time and increase the throughput.

A typical use case for this scenario is in Messaging Systems where one application needs to communicate with others by exchanging messages. This information exchange is useful to decouple multiple applications where changes can be isolated in one or more applications without impacting others. Therefore, the consumer needs to receive the messages as fast as possible.

Recommended Kafka configuration value:

Consumer:
• session.timeout.ms: increase (default 10000)

Conclusion

Kafka provides a highly scalable and fault-tolerant distributed messaging system. Different optimization approaches are required based on the nature of the application itself. The optimization factors need to be considered because each of the optimization factors has opposing impacts. Kafka can be optimized to have a high durability (with the side effect of increasing the latency) which is suitable for applications that maintain data consistency. On the other hand, optimizing for high availability will reduce the number of replicas of the data, which is suitable for exchanging messages between applications.

References:
Ibryam, Bilgin. Fine-tune Kafka performance with the Kafka optimization theorem. Redhat Developer. https://developers.redhat.com/articles/2022/05/03/fine-tune-kafka-performance-kafka-optimization-theorem
Gwen et al. 2021. Kafka: The Definitive Guide, 2nd Edition. O’Reilly
Kreps, Jay. The Log: What every software engineer should know about real-time data’s unifying abstraction. LinkedIn Engineering. https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Optimize and Tune Confluent Cloud Clients. Confluent Developer. https://docs.confluent.io/cloud/current/client-apps/architecture.html

Ida Bagus Gita Pradnyana – From Scratch to Analyst

“If you want to go fast, go alone. If you want to go far, go together.”
– African proverb –

Why climb the career ladder by yourself when you can have support from the company you work for?  This time, we are focusing on the journey of Ida Bagus Gita Pradnyana, one of our skilful and experienced Analysts, highlighting how Mitrais has helped him prosper in his career.

Fresh out of college, Gita had no academic background in Software Engineering, instead relying on his degree in Mathematics from one of Indonesia’s top universities to apply for his dream role in Mitrais. Starting as a Junior Programmer, he received in-depth training in Software Engineering from the company – the solid foundation he needed to take his next career step.

Gita’s first role with Mitrais was as part of a maintenance team managing our internal systems. It did not take long for him to advance from a junior team member to a Team Lead. He also performed as Team Lead for some on-the-job training projects undertaken by groups of Junior Programmer inductees. As Gita explained, “These responsibilities were golden opportunities because they helped me shape my soft skills, including leadership, team management, and problem-solving earlier in my career. Often these opportunities are only available once you reach more senior levels.”

In Mitrais, promotions are based on merit, so individuals can influence how far and how fast their progress to upper thresholds can be made. But it is never a lonely journey, because Mitrais supports our staff when it comes to enhancing skills to accomplish goals. “Mitrais has provided comprehensive training and OJT to enable us to step up through each of the grades. There are a lot of opportunities to expand your experiences like certifications and weekly meetups where I get to attend a discussion on interesting topics in Software Engineering” he said.

In addition to technical skills, Gita has sharpened his communication skills, especially in the English language through the Business Communication Course delivered by Mitrais’ in-house trainers. These enhanced skills boost his confidence in dealing with international clients. “Just like the stereotype, software engineers like me can tend to be slightly introverted, communicating best with the computer. Communicating with real people is another story – it was challenging for me to communicate ideas I had so that they made sense to the listener. Fortunately, I attended a series of Mitrais English training on communication skills, including cultural differences between people from different countries.” recalled Gita. He added, “I am also grateful for the support and help from other business units at Mitrais. They are prompt and helpful, so I know I can always count on them whenever I need a helping hand.”

Ten years have passed, and the Junior Programmer has now transformed into an experienced and reliable Analyst. Recently, Gita received a Mitrais Gold Award for his loyalty and contribution to the company. Gita pointed out that the opportunity to tackle problems that are more complex in scope is the best thing about his current role. “We are expected to see potential roadblocks in a more holistic way, rather than as a isolated issue as we used to do when we were at junior levels. It is quite a challenge, but it is always very exciting to do.”

But Gita’s life is not just about coding. Born and raised in beautiful and vibrant Denpasar, he continues to develop his long-time interest in music. He loves to listen to many genres and attend music concerts held in his area.  With the beautiful scenery that Bali has to offer, Gita also enjoys photography. He is an active person, flexing his muscles by running, badminton and travelling. Gita is passionate about his hobbies, and spends his downtime doing things he loves. It’s all about a great work-life balance in Mitrais!

 

 

 

Concluding 2022 with Lots of Surprises in a Combo Event: Open Forum & Happy Hour

After three long years of the pandemic shaking the global economy, 2023 has started with many businesses still struggling to get back on their feet.

Amid the uncertainty, however, Mitrais remains as strong as ever, wrapping up a successful 2022. The result, of course, was the fruit of everyone’s commitment and perseverance, supported by well-established systems that provide added value to our clients’ businesses.

Mitrais held a festive combo event to celebrate the successful business year: a fun fusion of the hybrid online and offline Open Forum and happy hour at all the offices last December.

The Open Forum is Mitrais’ platform where our Management showcases the company’s updates while listening to any concerns from employees. Indeed, open conversations and transparency are always expected here. During the last Open Forum, the management team revealed discretionary bonuses for all – always music to everyone’s ears.

And the good news did not end there! On top of the bonuses, Mitrais awarded 122 employees with #mitraisgold awards for their 10- and 25-year outstanding service to and trust in Mitrais. The culture of appreciation that boosts employee morale is a tradition that Mitrais takes great pride in.

Nugroho Danang Sasongko, one of our Analyst Programmers who brought a gold awards home, was delighted. “It’s an absolutely wonderful and joyful event! I feel honored and appreciated to receive the Mitrais Gold for being with Mitrais for 12 years.” Time flies, indeed. “For 2023, I hope Mitrais will land even bigger conquests in both Indonesian and international markets!” wished the Yogyakarta-based Analyst.

Excited by the news and the gathering, Ika Putri Melindawati, our Bali-based Scheduling & Facilities Services officer also shared her impression of the event: “It’s such an amazing celebration, and I’m so thrilled because finally I can catch up with my colleagues, celebrate our wins together, and enjoy the feast during the happy hour. Ika was optimistic that Mitrais will exceed its target in 2023, “I wish an even brighter future for Mitrais and all the best for the next year!”

As we ended 2022 on such a positive note, all of us at Mitrais are delighted to welcome 2023 with the optimism to break yet another glass ceiling. We are looking forward to more growth and more people to board this new chapter with us.