After the arrival of AI in smartphones, assistance systems and many Internet applications, it slowly makes it way to embodied agents, for example self-driving cars. While physically manifested agents are uncommon and limited to special use cases, it is important to investigate how agents can be implemented to successfully cooperate in spaces with other participants. To test the social intelligence of agents, the Watch-and-Help challenge was presented, in which AI agents work together with humans in shared apartments. During the watch-phase of this challenge, the agent must infer another actor’s goal as they perform a common household task. Originally, the agent perceived its environment using a transformer to encode the state of the apartment. In this work, we tested if it was possible to replace the transformer with a Graph Neural Network, which are capable of encoding environments. We further investigate how different encoded relations in the environment graphs effect the capabilities of our new model, discuss its current problems, and propose approaches how to deal with these.