| Bibliography | Zacher, Daniel: Adaptive GUI Testing with LLMS: Reducing Maintenance Overhead. University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology, Master Thesis No. 116 (2025). 81 pages, english.
|
| Abstract | Context. Graphical user interface (GUI) testing remains one of the most challenging and labor intensive areas in software engineering, as even small interface changes can break existing tests and require substantial maintenance effort. Ensuring reliable, scalable, and adaptable GUI testing is therefore a key concern for modern software development teams. Problem. GUI testing remains a challenging area within software engineering, as test scripts are highly prone to breaking after even minor interface changes. Additionally, traditional GUI testing tools struggle to reliably interact with dynamic environments such as tables, tree structures, or context dependent UI elements. Objective. The objective of this thesis is to investigate whether a Computer Using Agent (CUA), powered by modern large language models, can reduce the maintenance effort typically associated with GUI testing. By allowing an agent to visually interpret the interface and autonomously perform interactions, we aim to evaluate whether such an approach can provide a more resilient and flexible alternative to traditional script based testing methods. Method. We implemented a CUA that leverages an LLM to visually interpret and interact with the GUIs of PREEvision and Indigo. We then conducted a experiment to identify the most effective configuration of the CUA, evaluate the agent’s performance, and assess its robustness under GUI changes. Result. The experiments show that the CUA requires substantial time for each action and frequently interacts with incorrect GUI elements, causing most test cases to fail. However, its performance remains largely unaffected by GUI changes, and it is capable of handling dynamic interface elements such as tables. Conclusion. The results show that while the CUA demonstrates clear potential for more flexible, perception driven GUI testing, it is not yet suitable for production use due to slow action execution and limited interaction accuracy. With future improvements, particularly in its precision, it could become a valuable tool for testing dynamic and rapidly evolving GUI components during development.
|