Bibliograph. Daten | Zhao, Jiahao: Situated interactive guidance and assistance in extended reality using multimodal large language models. Universität Stuttgart, Fakultät Informatik, Elektrotechnik und Informationstechnik, Masterarbeit Nr. 67 (2025). 54 Seiten, englisch.
|
| Kurzfassung | This paper addresses the need for "providing actionable guidance for process tasks in real-world environments� by proposing an interactive XR system that combines a multimodal large language model with Apple Vision Pro. The system employs a three-stage architecture with cloud-end decoupling: Vision Pro handles acquisition and spatial projection, a Python bridge server manages sessions and state machines, and the LLM backend uses an open-source vision model for object localization and generates voice/text instructions. We also compare local inference models with AWS cloud-based inference models: local deployment offers lower latency, while cloud-based deployment is more scalable but subject to greater EBS throughput and initialization overhead. Finally, a study with eight participants shows that the AR-based SUS had higher usability and lower overall NASA-TLX load. The difference in completion time was not significant but trended faster. Participants also perceived AR guidance as more accurate. The paper discusses trade-offs between local GPU and cloud-based deployment, current technical bottlenecks, and potential future research directions.
|
Volltext und andere Links | Volltext
|
| Abteilung(en) | Universität Stuttgart, Institut für Visualisierung und Interaktive Systeme, Visualisierung und Interaktive Systeme
|
| Betreuer | Schmalstieg, Prof. Dieter; Pabst Michael |
| Eingabedatum | 19. Dezember 2025 |
|---|