Using software for large-scale simulations has become an important research method in many disciplines. With increasingly complex simulations, simulation software becomes a valuable assest. Yet, the quality of many simulation codes is worrying. In this paper, we want to collect and structure the challenges for a systematic simulation software engineering as a reference and the basis for further research. We describe our own experiences with developing simulation software and collaborating with non-computer-scientists. We complement our experienced challenges with a brief literature review. We structured the challenges for simulation software engineering into six areas: motivation and recognition; education and training; developer turnover; software length of life; verification, validation and debugging; and efficiency vs. maintainability. Overcoming these challenges needs efforts from research agencies, scientific computing researchers as well as software engineering researchers.