Bachelorarbeit BCLR-2025-23

Bibliograph.
Daten
Zhang, Yichi: Do Transformers Attend Similar to Human Gaze - Revisited.
Universität Stuttgart, Fakultät Informatik, Elektrotechnik und Informationstechnik, Bachelorarbeit Nr. 23 (2025).
45 Seiten, englisch.
Kurzfassung

Attention is a key aspect in the research question ”Do Transformers attend similar to human gaze?”. There exist multiple methods to extract attention data from a neural model or from transformers. In prior works, researchers tried to answer this research question, however with limited success. Their research was limited to insufficient and flawed methods, such as attention weights and attention flow. Further limitations are, for instance, that humans have multiple perceptions whereas the transformer model is limited to monomodality in prior works. There are also counter-intuitive findings. For instance, fine-tuning a model improves accuracy on a task but does not improve correlation to the human gaze. We revisit this research question, overcoming the limitations of previous works in this thesis. To overcome the limitation of flawed methods, we extend the set of extraction methods with a newly proposed, computationally efficient and more faithful method, Attention-Aware Layer-wise Relevance Propagation (AttnLRP). Its is been proven to be optimized for the transformer architecture. With the additional extraction method, we show that the similarity of human gaze and transformer attention might have been overestimated due to unfaithful methods inflating the correlation values. Our hypothesis is that it results in a higher correlation between transformer and human gaze than evaluated in previous research. Counter-intuitive findings are resolved at the end of this research by reproducing the same experiment setup, extended with AttnLRP. We overcome the limitation of monomodality by utilising a transformer model capable of multimodal input and extracting attention using attention weights as well as AttnLRP. From the attention, we compute novel correlation data on textual and visual inputs and compare it to the human gaze.

Abteilung(en)Universität Stuttgart, Institut für Visualisierung und Interaktive Systeme, Visualisierung und Interaktive Systeme
BetreuerBulling, Prof. Andreas; Kögel, Fabian
Eingabedatum8. August 2025
   Publ. Institut   Publ. Informatik