| Bibliography | Youssef, Mohamed: Ontology-Guided Diffusion for Zero-Shot Visual Sim2Real Transfer. University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology, Master Thesis No. 27 (2026). 60 pages, english.
|
| Abstract | Bridging the simulation-to-reality (sim2real) gap remains a central challenge in robotics and visual learning. Although synthetic data is scalable and inexpensive, models trained in simulation often fail in real-world conditions due to structured appearance mismatches in lighting, materials, shadows, optics, and sensor artifacts. Existing diffusion-based sim2real methods typically rely on unstructured prompts or implicit statistical alignment, which limits interpretability and controllability. This thesis introduces Ontology-Guided Diffusion (OGD), a neuro-symbolic framework for zero-shot sim2real image translation. OGD models realism as structured knowledge by decomposing appearance realism into interpretable traits and encoding their supportive and opposing dependencies in a signed knowledge graph. Given a synthetic image, trait activations are predicted from frozen CLIP features, refined through graph propagation, and used to produce realism-aware conditioning embeddings. In parallel, a symbolic planner (PDDL) computes a causally valid sequence of trait transformations from synthetic to target realism states; this sequence is converted into a structured editing prompt. Both graph embeddings and symbolic prompts are injected into an instruction-guided diffusion model via cross-attention, enabling semantically grounded and causally consistent image editing. The framework is evaluated on realism representation quality and sim2real translation quality. Quantitative results show that ontology-guided realism embeddings strongly improve real-vs-synthetic discrimination (up to 98.4% accuracy and 0.99 ROC-AUC) and that OGD outperforms strong diffusion baselines on sim2real translation metrics (TraitDist, LPIPS, and SSIM). The proposed method is data-efficient: realism representation learning is trained with a small unpaired set (140 images), and inference is performed in a zero-shot setting. Overall, the thesis demonstrates that explicitly modeling realism structure is an effective direction for interpretable, controllable, and practical sim2real transfer.
|