Imagine a partnership built on trust, innovation, and a shared commitment to excellence. Mitrais has been more than just a software development partner to Integrated Research (ASX:IRI), a leader in critical system communication, payment, and infrastructure monitoring.

It wasn’t just about writing code. Integrated Research significantly increased their development velocity while maintaining top-notch quality by partnering with Mitrais, as we focus on measurable outcomes and constant improvement. Plus, providing the right talent, perfectly matched to their needs, has been a key part of our long-term partnership.

Our General Manager, Rob Mills, and Michael Tomkins, Chief Technology Officer of Integrated Research highlight that our partnership is about building a strong, collaborative relationship.

Want to see how Mitrais ticks all the boxes for Integrated Research’s software development needs? Click here to watch the video now!

Abstract

In this white paper, we explore the implementation of a Large Language Model (LLM)-powered chatbot within the Competency System of our organisation. The chatbot utilises Retriever-Augmented Generation (RAG) embeddings to answer employee enquiries related to company policies, competencies, and other system-related queries. Additionally, function calling is integrated to dynamically fetch personalised employee details. This document highlights the technical aspects of the system, its operational benefits, challenges encountered, and the future potential of AI-driven HR support systems.

Introduction

With the rapid advancement of artificial intelligence, organisations are increasingly leveraging AI to streamline internal processes and improve employee experiences. A notable innovation in this space is the deployment of LLM-powered chatbots, which provide on-demand support and information to employees. In our organisation, we have integrated an LLM chatbot into the Competency System, allowing employees to access relevant information about their competencies, policies, and more, through natural language conversations.

This white paper outlines the technical implementation of this system, including the use of RAG embeddings for answering policy-related queries and function calling to retrieve employee-specific data, as well as the resulting impact on operational efficiency and employee satisfaction.

The Competency System and Its Challenges

The Competency System in our organisation serves as a central hub for employee development, containing critical information regarding job roles, required competencies, performance evaluations, and more. Traditionally, accessing this information required navigating through various manuals, policy documents, or interacting with HR representatives.

However, as the workforce grew and the volume of questions regarding the system increased, there arose a need for an efficient, scalable way to provide employees with answers to their enquiries. The integration of an LLM chatbot was seen as a strategic move to address this challenge by offering real-time, accurate responses based on our competency policies.

Solution Overview: Implementing the LLM Chatbot

LLM Chatbot Architecture

The chatbot is built upon a state-of-the-art LLM, capable of processing natural language queries and delivering coherent, contextually accurate responses. The system leverages the following components:

• Retriever-Augmented Generation (RAG) Embeddings: The LLM is enhanced with RAG embeddings, which help the model retrieve relevant information from a large corpus of policy documents and knowledge bases. This allows the system to answer employee questions about competencies, policies, and organisational procedures with high accuracy.
• Function Calling: To personalise responses and provide specific employee information (e.g., performance evaluations, training completion), the system calls on internal databases to fetch relevant data dynamically. This integration ensures that responses are not only contextually relevant but also tailored to the individual employee’s profile.

Workflow and Integration

The chatbot operates in real-time, processing employee queries and retrieving information from both static (policy documents) and dynamic (employee data) sources.

The process flow is detailed below:

Figure 1 M-Compass chatbot workflow

• The employee submits a question via the chatbot interface.
• The LLM, enhanced with RAG embeddings, parses the query and retrieves relevant information from internal policies and knowledge bases.
• If the query requires personalised data, function calls are made to fetch specific employee details from the internal system.
• The chatbot generates a coherent and personalised response, which is displayed to the employee.

Technical Details and Considerations

Challenges in Policy Retrieval

One of the primary challenges we faced was ensuring that the LLM could accurately retrieve and present information from a vast array of policy documents. This was addressed by using RAG embeddings, which combine traditional retrieval-based methods with generative language models to enhance the quality and relevance of the responses. The process begin with preprocessing the policy documents: they were segmented into manageable chunks (e.g., paragraphs or sections), tokenized, and converted into dense vector representations using a transformer-based embedding model. These embeddings were then indexed in a vector database optimized for fast similarity searches.

During query processing, the system first encodes the user’s question into a vector and retrieves the top-k most relevant document chunks based on cosine similarity or another distance metric. The LLM then uses this retrieved context to generate a coherent and contextually appropriate response, rather than relying solely on its pre-trained knowledge.

Integration with Employee Data Systems

Another significant challenge was ensuring seamless integration with internal systems to pull personalised employee data. Function calling was implemented to ensure secure and efficient retrieval of sensitive information while maintaining privacy and compliance with data protection regulations. The process likely involved defining a set of API endpoints each tied to a specific function, such as getEmployeeDetails(employeeID) or fetchCompetency(employeeID) that the LLM could invoke dynamically based on the query’s intent.

To implement this, the team uses LLM tool features to recognise when a query requires external data (e.g., “What’s my detail information?”) and map it to the appropriate function call. Data privacy was maintained by limiting data retrieval to the minimum necessary fields.

Ensuring Accuracy and Relevance

Given the complexity and variety of queries employees could ask, we focused on fine-tuning the LLM to handle diverse topics and provide precise answers. Continuous monitoring and adjustments were made to improve the accuracy of the responses, especially for more complex or ambiguous questions. We ensure relevance by incorporating user context wherever possible leveraging metadata like the employee’s role, or department to tailor responses. We also converted the policy document into Q&A format which could increase the accuracy of the responses.

Benefits and Impact

Enhanced Efficiency

The implementation of the LLM chatbot has drastically reduced the time HR staff spends answering repetitive or routine questions. Employees now have 24/7 access to the information they need, improving overall efficiency.

Improved Employee Satisfaction

The ability to quickly retrieve answers to competency-related questions has contributed to improved employee satisfaction and engagement. With shorter wait times and easier access to important information, employees are better supported in managing their professional development.

Scalability

The LLM chatbot has proven to be highly scalable, handling hundreds of employee enquiries without additional strain on HR resources. As the company grows, the system can easily accommodate more users and incorporate additional functionality.

Future Directions

As AI technology continues to evolve, there are several potential avenues for further improving the system:

• Enhanced Personalisation: Leveraging deeper integrations with employee performance data and development goals could allow the system to offer more tailored career advice and insights.
• Continuous Learning: The system could be further enhanced with continuous learning capabilities, allowing it to adapt to changes in policies and employee needs over time.

Conclusion

The implementation of an LLM chatbot in our Competency System represents a significant step forward in leveraging Artificial Intelligence to improve employee engagement, streamline HR processes, and increase operational efficiency. By combining RAG embeddings and function calling, we have created a system that not only provides accurate policy-based answers but also personalises responses based on individual employee data. As the system evolves, we are confident it will continue to transform the way our employees interact with the Competency System, driving greater satisfaction and fostering a more agile workforce.

Abstract

In this white paper, we explore the implementation of a Large Language Model (LLM)-powered chatbot within the Competency System of our organisation. The chatbot utilises Retriever-Augmented Generation (RAG) embeddings to answer employee enquiries related to company policies, competencies, and other system-related queries. Additionally, function calling is integrated to dynamically fetch personalised employee details. This document highlights the technical aspects of the system, its operational benefits, challenges encountered, and the future potential of AI-driven HR support systems.

Introduction

With the rapid advancement of artificial intelligence, organisations are increasingly leveraging AI to streamline internal processes and improve employee experiences. A notable innovation in this space is the deployment of LLM-powered chatbots, which provide on-demand support and information to employees. In our organisation, we have integrated an LLM chatbot into the Competency System, allowing employees to access relevant information about their competencies, policies, and more, through natural language conversations.

This white paper outlines the technical implementation of this system, including the use of RAG embeddings for answering policy-related queries and function calling to retrieve employee-specific data, as well as the resulting impact on operational efficiency and employee satisfaction.

The Competency System and Its Challenges

The Competency System in our organisation serves as a central hub for employee development, containing critical information regarding job roles, required competencies, performance evaluations, and more. Traditionally, accessing this information required navigating through various manuals, policy documents, or interacting with HR representatives.

However, as the workforce grew and the volume of questions regarding the system increased, there arose a need for an efficient, scalable way to provide employees with answers to their enquiries. The integration of an LLM chatbot was seen as a strategic move to address this challenge by offering real-time, accurate responses based on our competency policies.

Solution Overview: Implementing the LLM Chatbot

LLM Chatbot Architecture

The chatbot is built upon a state-of-the-art LLM, capable of processing natural language queries and delivering coherent, contextually accurate responses. The system leverages the following components:

• Retriever-Augmented Generation (RAG) Embeddings: The LLM is enhanced with RAG embeddings, which help the model retrieve relevant information from a large corpus of policy documents and knowledge bases. This allows the system to answer employee questions about competencies, policies, and organisational procedures with high accuracy.
• Function Calling: To personalise responses and provide specific employee information (e.g., performance evaluations, training completion), the system calls on internal databases to fetch relevant data dynamically. This integration ensures that responses are not only contextually relevant but also tailored to the individual employee’s profile.

Workflow and Integration

The chatbot operates in real-time, processing employee queries and retrieving information from both static (policy documents) and dynamic (employee data) sources.

The process flow is detailed below:

Figure 1 M-Compass chatbot workflow

• The employee submits a question via the chatbot interface.
• The LLM, enhanced with RAG embeddings, parses the query and retrieves relevant information from internal policies and knowledge bases.
• If the query requires personalised data, function calls are made to fetch specific employee details from the internal system.
• The chatbot generates a coherent and personalised response, which is displayed to the employee.

Technical Details and Considerations

Challenges in Policy Retrieval

One of the primary challenges we faced was ensuring that the LLM could accurately retrieve and present information from a vast array of policy documents. This was addressed by using RAG embeddings, which combine traditional retrieval-based methods with generative language models to enhance the quality and relevance of the responses. The process begin with preprocessing the policy documents: they were segmented into manageable chunks (e.g., paragraphs or sections), tokenized, and converted into dense vector representations using a transformer-based embedding model. These embeddings were then indexed in a vector database optimized for fast similarity searches.

During query processing, the system first encodes the user’s question into a vector and retrieves the top-k most relevant document chunks based on cosine similarity or another distance metric. The LLM then uses this retrieved context to generate a coherent and contextually appropriate response, rather than relying solely on its pre-trained knowledge.

Integration with Employee Data Systems

Another significant challenge was ensuring seamless integration with internal systems to pull personalised employee data. Function calling was implemented to ensure secure and efficient retrieval of sensitive information while maintaining privacy and compliance with data protection regulations. The process likely involved defining a set of API endpoints each tied to a specific function, such as getEmployeeDetails(employeeID) or fetchCompetency(employeeID) that the LLM could invoke dynamically based on the query’s intent.

To implement this, the team uses LLM tool features to recognise when a query requires external data (e.g., “What’s my detail information?”) and map it to the appropriate function call. Data privacy was maintained by limiting data retrieval to the minimum necessary fields.

Ensuring Accuracy and Relevance

Given the complexity and variety of queries employees could ask, we focused on fine-tuning the LLM to handle diverse topics and provide precise answers. Continuous monitoring and adjustments were made to improve the accuracy of the responses, especially for more complex or ambiguous questions. We ensure relevance by incorporating user context wherever possible leveraging metadata like the employee’s role, or department to tailor responses. We also converted the policy document into Q&A format which could increase the accuracy of the responses.

Benefits and Impact

Enhanced Efficiency

The implementation of the LLM chatbot has drastically reduced the time HR staff spends answering repetitive or routine questions. Employees now have 24/7 access to the information they need, improving overall efficiency.

Improved Employee Satisfaction

The ability to quickly retrieve answers to competency-related questions has contributed to improved employee satisfaction and engagement. With shorter wait times and easier access to important information, employees are better supported in managing their professional development.

Scalability

The LLM chatbot has proven to be highly scalable, handling hundreds of employee enquiries without additional strain on HR resources. As the company grows, the system can easily accommodate more users and incorporate additional functionality.

Future Directions

As AI technology continues to evolve, there are several potential avenues for further improving the system:

• Enhanced Personalisation: Leveraging deeper integrations with employee performance data and development goals could allow the system to offer more tailored career advice and insights.
• Continuous Learning: The system could be further enhanced with continuous learning capabilities, allowing it to adapt to changes in policies and employee needs over time.

Conclusion

The implementation of an LLM chatbot in our Competency System represents a significant step forward in leveraging Artificial Intelligence to improve employee engagement, streamline HR processes, and increase operational efficiency. By combining RAG embeddings and function calling, we have created a system that not only provides accurate policy-based answers but also personalises responses based on individual employee data. As the system evolves, we are confident it will continue to transform the way our employees interact with the Competency System, driving greater satisfaction and fostering a more agile workforce.

This issue we take the time to introduce I Putu Arya Kusuma Wijaya, known fondly as Arya. Hailing from Denpasar, Arya’s journey into the world of technology and cybersecurity is an inspiring and impressive one.

Arya’s academic journey began with a bachelor’s degree in electrical engineering from Udayana University. Eager to carve out a niche in the IT world, he has pursued an array of added certifications, including Security+, Azure, AWS, CISA, and CISSP. These credentials not only broadened his expertise but also prepared him for the dynamic challenges of the cybersecurity realm.

Arya’s professional career landed him at Mitrais, the leading IT company in Bali. Initially joining as part of our Information and Communications Technology (Infocomm) team, Arya’s dedication and proven skills soon earned him promotions, and he now serves as the Compliance and Operations Lead. In this role, he lends his considerable abilities to IT operations, desktop support, network monitoring, and ISO 27001 compliance.

His passion for cybersecurity was ignited in 2017 when he embarked on his first training with CompTIA (one of the world’s most trusted certifying bodies) in Security+. This marked the beginning of a journey filled with learning and professional development. As an ISO 27001 implementer at Mitrais, Arya continues to refine his skills and stay abreast of industry advancements through added certifications.

Reflecting on his ambitions, Arya says that it is remarkable how his dreams steadily evolved from simply creating games to aspiring for leadership. His current goal is to become an exemplary leader, guiding his team towards success and continuous growth.

One of the aspects Arya appreciates most about Mitrais is the company’s culture. He finds the team’s humility and friendliness refreshing, and he enjoys the continuous learning environment that Mitrais fosters. This supportive atmosphere has been instrumental in his professional growth and achievements.

Implementing ISO certification for the first time posed some significant challenges for Arya’s team, but with the unwavering support of Mitrais management and the proactive approach of his colleagues, they successfully navigated the rigorous certification process. This collaborative effort ensured that the company met (and still meets) all necessary standards and achieved that valuable certification, with all the benefits that our clients gain from that.

Arya’s journey is a testament to the power of ambition, continuous learning, and teamwork. Looking forward, he is still committed to his goal of becoming a successful and productive leader within Mitrais. His story no doubt serves as an inspiration to many, highlighting how dedication and a supportive environment can pave the way for remarkable achievements in the IT and cybersecurity fields.

But Arya’s story is not just about individual success; it’s about the collective spirit of Mitrais that nurtures talent and drives growth. As Arya continues to excel and inspire, we can only anticipate more extraordinary milestones in his journey, and Mitrais and our clients benefit from that.