Long short-term memory (LSTM) networks are known to be extremely accurate at working with temporal features and time series data, for their ability to memorize sequences of data. This is practical for predicting the rainfall-runoff process, as the input is typically a sequence of weather features for a certain duration. Furthermore a modeled system of a river catchment has an inherent memory, namely all the accumulating weather circumstances that effect the generation of runoff over time, which can be reflected in an LSTM model by its hidden states. A model has been implemented and trained using weather data from 1961-2011 extracted from the upper Neckar catchment in southwest Germany that had very satisfactory results. To assess and understand the black box nature of the LSTM model in its application in regards to rainfall runoff predictions, as well as understanding the reason for its good performance in solving problems in regards to hydrological modeling, a visualization tool has been developed. Different aspects of the neural network have been explored through the visualization tool such as the hidden states throughout testing, input sequences dimensionality reduction, correlation between individual hidden layer cells and weather features and internal model features, etc. A suggested workflow has been explored for using the tool, that derives some insights into the model and its performance. Finally different observations have been thoroughly explored and interpreted. Keywords: Machine Learning, Deep Learning, Long Short-Term Memory (LSTM), Neural Network, Rainfall Runoff Modeling, Visualization, Interpretation, Explainable AI.