The ability to infer the believes, desires and preferences of other humans around us - referred to Theory of Mind - is crucial for effective human cooperation. In this work we investigate how this ability can facilitate cooperation among artificial agents, particularly in zero-shot cooperation scenarios where the partner might be human. While previous works had access to ground truth belief states of the other agents during training, we study Theory of Mind based collaboration in a multi-agent collaborative environment where no ground-truth belief states exists, namely Overcooked. We propose three auxiliary tasks that agents are trained with which in turn are inspired by Theory of Mind: Predicting (i) partner’s next action, (ii) partner’s next strategic goal and (iii) partner’s neural state. Our research demonstrates that self-play agents trained with these auxiliary tasks exhibit improved competence in playing the game but tend to underperform when interacting with others, suggesting a tendency towards overspecialisation on oneself. To address this challenge, we add our auxiliary tasks to population based methods for training against diverse populations and show increased performance on several benchmarks, especially on the layout Asymmetric Advantages. Overall, our work shows the importance of explicitly modelling Theory of Mind for multi-agent cooperation.