Evaluation of open-source transformers for cloud question answering
In recent years, there has been a surge of interest in Question Answering (QA) systems, making this an important area of research within Natural Language Processing (NLP). These systems are widely used in applications involving human interaction, such as customer service chatbots and virtual assistants like Siri and Alexa. The primary goal of a QA system is to generate natural-language responses that accurately answer users’ questions. In practice, however, this requires more than simply extracting explicitly stated answers from a text.
As described by Saeidi et al. and Mensio et al., many practical QA challenges require a deeper understanding of text, with answers being derived based on background knowledge and context. This requires reasoning ability and insight into subtle meanings within a question. Mohnish et al. emphasized that raw answers from QA systems often do not meet user expectations, while Strzalkowski et al. observed that users value detailed, context-rich information more than short, direct answers.
Developments in Transformer Models for QA
Modern NLP approaches, such as transformer models, are used to address this complexity in QA tasks. Unlike traditional sequential models, which process input step by step, transformers use self-attention mechanisms to compute relationships between all elements within an input sequence. This allows them to capture meaningful connections between components that are far apart. Research by Izacard et al. shows that incorporating additional knowledge via retrieval significantly improves the QA performance of transformer models.
With the emergence of Large Language Models (LLMs), which have millions to billions of parameters, QA systems have continued to evolve. Proprietary LLMs such as OpenAI’s ChatGPT 3 and its successors have achieved great success across a wide range of NLP tasks, with GPT-3.5 Turbo reaching more than 100 million active users in 2022. Despite these achievements, concerns remain regarding privacy, including issues related to training data, the handling of user data, and potential vulnerabilities. This underscores the importance of exploring decentralized and open-source (OS) QA solutions that prioritize privacy and transparency.
Research focus and evaluation
This study focused on evaluating the effectiveness of various open-source transformer models—Extractive Pre-trained Transformer (EPT), Generative Pre-trained Transformer (GPT), and Text-to-Text Transfer Transformer (T5)—within a realistic QA use case. These models were compared with OpenAI’s proprietary LLM, GPT-3.5 Turbo, to assess their performance and competitiveness.
The domain chosen for this research is cloud computing, with a specific focus on Kubernetes technology. The QA system uses the public Kubernetes documentation, supplemented with real-time searches via Google, as its primary source of knowledge. An innovative Machine-trained Evaluation Score (MTES) called Estimated Human Label (EHL) has been developed to evaluate the different models. This score uses machine learning techniques trained on N-gram-based metrics and human-labeled datasets. It includes a variety of question types, including closed-ended, open-ended, conceptual, and procedural questions.
