In the quest for scientific developments and advancements in the society, machine learning applications are becoming part of almost every process in the industries. The world is heading towards the utilization of experience gained by machines. What if the experience is gained from faulty dataset or the predictions of the machine learning algorithm are wrong due to some other reason? This would corrupt the entire system and lead to an enormous loss of time and money. So, it is important to take a note of the performance of the machine learning application before deploying it. In the current scenario, understanding of machine learning model is a continued field of research. The visual analytics approach which involves interactive visualization and explorative analysis of dataset can be exploited in the model development process, as it integrates human-knowledge with the power of machines. In this thesis, a contribution is made in this area for monitoring the machine learning application in real-time and analysis of the same using Visual Analytics to address the problem of concept drift. An approach is designed and a software implementation is done for its demonstration. Interactive visualizations have been provided for the actual dataset and the predictions obtained from the machine learning model. A simulation for continuous arrival of data streams has been developed. The idea is to recognize the right point of time at which a new model needs to be trained. Tools have been integrated for further interactive analysis of this dataset. As the data from a News Agency has been used, analysis of textual data and its visualization have formed a significant part of the visual explorative analysis.