Investigating Open Source Transformer Techniques for Question Answering Systems on Cloud Domain

In recent times, there has been considerable interest in Question Answering (QA) systems, making them a prominent area of study in Natural Language Processing (NLP). These systems play a vital role in applications involving human interaction, such as customer service chatbots and virtual assistants like SIRI or Alexa. The main objective of a QA system is to generate natural language responses that satisfactorily address user queries. However, this goal often requires more than simply retrieving answers explicitly stated in a text.

As noted by Saeidi et al. and Mensio et al., many practical QA challenges demand a deeper comprehension of the text, enabling the derivation of responses through background knowledge and context. This necessitates reasoning capabilities and an understanding of nuanced meanings within a query. Mohnish et al. emphasized that raw responses from QA systems often fall short of user expectations, and Strzalkowski et al. observed that users value detailed, context-rich information over direct, concise answers.

Advances in Transformer Models for QA

Modern NLP approaches, such as transformer models, have been employed to address the complexities of QA tasks. Unlike traditional sequential models that process input step-by-step, transformers utilize self-attention mechanisms to compute the relationships between all elements of an input sequence, capturing meaningful connections between distant components. Research by Izacard et al. shows that incorporating supplementary knowledge through retrieval significantly enhances QA performance in transformer models.

The advent of Large Language Models (LLMs), which feature millions or billions of parameters, has further advanced QA systems. Proprietary LLMs like OpenAI’s ChatGPT 3 and its successors have achieved remarkable success in various NLP tasks, with ChatGPT3.5-Turbo gaining over 100 million active users by 2022. Despite their capabilities, concerns about privacy persist, including issues around training data, user data handling, and potential vulnerabilities. These challenges highlight the importance of exploring decentralized and open-source (OS) approaches to QA systems that prioritize privacy and transparency.

This research investigated the effectiveness of various OS transformer models—Extractive Pre-trained Transformer (EPT), Generative Pre-trained Transformer (GPT), and Text-to-Text Transfer Transformer (T5)—in the context of a real-world QA task. These models were compared against OpenAI’s proprietary LLM, ChatGPT3.5-Turbo, to evaluate their performance and competitiveness.

The domain selected for this research is cloud computing, with a focus on Kubernetes technology. The QA system uses the Kubernetes public documentation, augmented by real-time searches on Google, as its primary knowledge base. To assess the performance of each model, an innovative Machine-trained Evaluation Score (MTES) called Estimated Human Label (EHL) was developed. This score leverages machine learning techniques trained on N-gram-based metrics and human-labeled datasets that encompass diverse question types, including close-ended, open-ended, conceptual, and procedural queries.