Master Thesis MSTR-2026-27

BibliographyYoussef, Mohamed: Ontology-Guided Diffusion for Zero-Shot Visual Sim2Real Transfer.
University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology, Master Thesis No. 27 (2026).
60 pages, english.
Abstract

Bridging the simulation-to-reality (sim2real) gap remains a central challenge in robotics and visual learning. Although synthetic data is scalable and inexpensive, models trained in simulation often fail in real-world conditions due to structured appearance mismatches in lighting, materials, shadows, optics, and sensor artifacts. Existing diffusion-based sim2real methods typically rely on unstructured prompts or implicit statistical alignment, which limits interpretability and controllability. This thesis introduces Ontology-Guided Diffusion (OGD), a neuro-symbolic framework for zero-shot sim2real image translation. OGD models realism as structured knowledge by decomposing appearance realism into interpretable traits and encoding their supportive and opposing dependencies in a signed knowledge graph. Given a synthetic image, trait activations are predicted from frozen CLIP features, refined through graph propagation, and used to produce realism-aware conditioning embeddings. In parallel, a symbolic planner (PDDL) computes a causally valid sequence of trait transformations from synthetic to target realism states; this sequence is converted into a structured editing prompt. Both graph embeddings and symbolic prompts are injected into an instruction-guided diffusion model via cross-attention, enabling semantically grounded and causally consistent image editing. The framework is evaluated on realism representation quality and sim2real translation quality. Quantitative results show that ontology-guided realism embeddings strongly improve real-vs-synthetic discrimination (up to 98.4% accuracy and 0.99 ROC-AUC) and that OGD outperforms strong diffusion baselines on sim2real translation metrics (TraitDist, LPIPS, and SSIM). The proposed method is data-efficient: realism representation learning is trained with a small unpaired set (140 images), and inference is performed in a zero-shot setting. Overall, the thesis demonstrates that explicitly modeling realism structure is an effective direction for interpretable, controllable, and practical sim2real transfer.

Department(s)University of Stuttgart, Institute of Visualisation and Interactive Systems, Visualisation and Interactive Systems
Superviser(s)Bulling, Prof. Andreas; Elfares, Mayar; Bortoletto, Matteo
Entry dateAugust 13, 2026
   Publ. Institute   Publ. Computer Science