| Bibliography | Anwender, Colin: Evaluating Retrieval-Augmented Generation to AI for Requirements Engineering. University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology, Bachelor Thesis No. 67 (2025). 89 pages, english.
|
| Abstract | Requirements Engineering (RE) plays a foundational role in software development, ensuring that systems meet stakeholder needs and comply with regulatory and business constraints. However, traditional RE methods are often manual, time-intensive and prone to inconsistencies. With the advent of Large Language Models (LLMs), new opportunities have emerged to automate and support RE workflows. Despite their strengths in general language understanding and generation, LLMs often struggle in highly specific or niche domains, particularly when relevant contextual information was not present in their pretraining data or is not publicly available. This challenge is particularly common in the domain of RE, where practitioners frequently need to account for domain-specific policies, regulatory constraints and organizational standards that are not commonly represented in publicly available datasets. To address these challenges, this thesis investigates the integration of Retrieval-Augmented Generation (RAG) into an LLM fine-tuned specifically for RE. RAG enhances generative models by combining them with an external knowledge retrieval mechanism, enabling the dynamic incorporation of relevant domain-specific information at inference time. A custom RAG pipeline was implemented and subsequently evaluated using both automated retrieval metrics, such as F1@K, NDCG@K and the Retrieval Augmented Generation Assessment (RAGAS) framework and a structured user study involving domain experts, where the original model was tested against the RAG-enhanced version. The evaluation results identified thenlper/gte-base as the most effective embedding model for the implemented RAG pipeline, achieving the highest average query score across tested alternatives. Furthermore, the RAGAS evaluation yielded high performance across all assessed metrics, although the faithfulness score indicated potential for improvement in aligning generated content more accurately with the retrieved context. The user study employed a mixed-methods approach, combining Likert-scale assessments of task adherence, context relevance and requirement quality with open-ended feedback. Statistical analysis using the Wilcoxon signed-rank test revealed no statistically significant improvement in task adherence or context relevance for the RAG-enhanced model compared to its baseline counterpart. Moreover, the RAG-based model demonstrated a decline in requirement quality, which can be attributed to its difficulty in consistently generating singular requirements. This suggests that while RAG introduces additional contextual information, it may also increase the complexity of the generation task, leading to diminished clarity and focus in the output. This thesis highlights the benefits and limitations of integrating retrieval into generative Artificial Intelligence (AI) workflows and provides actionable insights for future research and real-world applications in AI-assisted software engineering.
|