In Visual Question Answering (VQA), strategies like majority voting lead to the loss of information, as annotations that diverge from the majority are excluded even though they may also be grounded on evidence within the image and present a valid alternative or complementary answer. We investigate question image pairs within the VQA-V2 Dataset that are prone to disagreement, either because of multiple references in the image or varying levels of concreteness in the answers. To this end, we conduct an eye-tracking study with participants performing annotations and find systematic differences in gaze distributions across AoIs in relation to them being targets or distractors. Furthermore, we show that relative dwell time on AoIs, fixation maps, and notably the final fixation of annotators can serve as an explanatory tool and show promise in distinguishing between human errors and valid alternative answers in one of the groups. While an expected relationship between answer concreteness and disagreement was observed only weakly and did not reach statistical significance, the results suggest that concreteness captures a complex phenomenon that cannot be reduced to a single metric. Overall, our findings demonstrate that gaze can serve as an informative signal in cases of disagreement but remains constrained by other factors such as the saliency of Aois.