Article in Journal ART-2019-07

BibliographyFritz, Manuel; Behringer, Michael; Schwarz, Holger: Quality-driven early stopping for explorative cluster analysis for big data.
In: Software-Intensive Cyber-Physical Systems.
University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology.
pp. 1-12, english.
Springer Berlin Heidelberg, February 6, 2019.
ISSN: 2524-8510; 2524-8529; DOI: 10.1007/s00450-019-00401-0.
Article in Journal.
CR-SchemaE.0 (Data General)
H.2.8 (Database Applications)
H.3.3 (Information Search and Retrieval)
KeywordsClustering; Big Data; Early Stop; Convergence; Regression
Abstract

Data analysis has become a critical success factor for companies in all areas. Hence, it is necessary to quickly gain knowledge from available datasets, which is becoming especially challenging in times of big data. Typical data mining tasks like cluster analysis are very time consuming even if they run in highly parallel environments like Spark clusters. To support data scientists in explorative data analysis processes, we need techniques to make data mining tasks even more efficient. To this end, we introduce a novel approach to stop clustering algorithms as early as possible while still achieving an adequate quality of the detected clusters. Our approach exploits the iterative nature of many cluster algorithms and uses a metric to decide after which iteration the mining task should stop. We present experimental results based on a Spark cluster using multiple huge datasets. The experiments unveil that our approach is able to accelerate the clustering up to a factor of more than 800 by obliterating many iterations which provide only little gain in quality. This way, we are able to find a good balance between the time required for data analysis and quality of the analysis results.

Full text and
other links
Springer Link
CopyrightSpringer Berlin Heidelberg
Contactmanuel.fritz@ipvs.uni-stuttgart.de
Department(s)University of Stuttgart, Institute of Parallel and Distributed Systems, Applications of Parallel and Distributed Systems
Entry dateMarch 7, 2019
   Publ. Computer Science