Bachelor Thesis BCLR-2025-14

BibliographyHina, Amtal Mateen: Explaining Disagreement in Visual Question Answering Using Eye Tracking on Questions.
University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology, Bachelor Thesis No. 14 (2025).
51 pages, english.
Abstract

While observing the different answers of participants to the same question of Visual Question Answering (VQA) the question came up: What are the reasons behind the disagreement in the answers? Previous work has investigated if the gaze data of participants on the images can explain the disagreement. The novel approach was to use the gaze data on questions instead of the images. The mandatory goal is to find out whether there are differences in visual attention between participants who gave different answers. Secondly, we identify cases of disagreements caused by synonyms, granularity and spelling mistakes. To obtain these goals, we analysed the answers and gaze data and manually inspected the examples where differences in visual attention support the disagreement. The textual analysis of answers finds 1169 out of 3990 examples of disagreement caused by small mistakes, don’t know (lack of information), synonyms and granular. On the other hand, there is no significant difference in the gaze metrics between cases of agreement and disagreement. Moreover, manual inspection only reveals 47 out of 3990 examples. Overall, we find that the difference in visual attention to the question is not able to explain the answers’ disagreement. Keywords: Visual Question Answering (VQA) · VQA-MHUG · Question-image pair (QI pair) · Gaze data · Disagreement · Natural Language Processing (NLP

Department(s)University of Stuttgart, Institute of Visualisation and Interactive Systems, Visualisation and Interactive Systems
Superviser(s)Bulling, Prof. Andreas; Hindennach, Susanne
Entry dateJuly 10, 2025
New Report   New Article   New Monograph   Institute   Computer Science