Masterarbeit MSTR-2025-67

Bibliograph.
Daten
Zhao, Jiahao: Situated interactive guidance and assistance in extended reality using multimodal large language models.
Universität Stuttgart, Fakultät Informatik, Elektrotechnik und Informationstechnik, Masterarbeit Nr. 67 (2025).
54 Seiten, englisch.
Kurzfassung

This paper addresses the need for "providing actionable guidance for process tasks in real-world environments� by proposing an interactive XR system that combines a multimodal large language model with Apple Vision Pro. The system employs a three-stage architecture with cloud-end decoupling: Vision Pro handles acquisition and spatial projection, a Python bridge server manages sessions and state machines, and the LLM backend uses an open-source vision model for object localization and generates voice/text instructions. We also compare local inference models with AWS cloud-based inference models: local deployment offers lower latency, while cloud-based deployment is more scalable but subject to greater EBS throughput and initialization overhead. Finally, a study with eight participants shows that the AR-based SUS had higher usability and lower overall NASA-TLX load. The difference in completion time was not significant but trended faster. Participants also perceived AR guidance as more accurate. The paper discusses trade-offs between local GPU and cloud-based deployment, current technical bottlenecks, and potential future research directions.

Volltext und
andere Links
Volltext
Abteilung(en)Universität Stuttgart, Institut für Visualisierung und Interaktive Systeme, Visualisierung und Interaktive Systeme
BetreuerSchmalstieg, Prof. Dieter; Pabst Michael
Eingabedatum19. Dezember 2025
Neuer Report   Neuer Artikel   Neues Sammelwerk   Institut   Informatik