Autonomous driving requires to predict intents of surrounding traffic agents in order to navigate safely. This work presents a generic, learning-based approach to classify attributes of traffic agents by considering their context. A heterogeneous scene graph ontology is proposed that explicitly represents different types of dynamic and static entities from a traffic scenario. Relational types and features are used to model heterogeneous interaction between entities. Extensibility of the proposed graph structure is demonstrated by using in-house data with a large set of features. AGraph Neural Network architecture is designed based on heterogeneous convolutions on graphs to learn a generic encoding of nodes that covers structural features from the scene graph. EdgeSAGE is proposed as an effective operator to exploit relational features. Other aspects of the architecture consider the processing of temporal information, residual connections and task-specific decoders. An extensive evaluation is performed based on the classification of parking and ghost agents. The evaluation includes a discussion of the results, baseline comparisons, ablation studies and an extension to multi-task learning. Despite being generic, the proposed approach outperforms all existing task-specific baselines in both applications.