Master Thesis MSTR-2021-72

BibliographyRau, Simeon: Visualization for human-AI collaborative music composition.
University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology, Master Thesis No. 72 (2021).
77 pages, english.

We propose an AI-assisted approach based on interactive visualizations to support users in composing music and getting insights into the AI through hyperparameter analysis. Our user-centered approach allows the user to better control the composition by steering the AI’s suggestions. We use symbolic music data and piano rolls as visual music notation for easier understanding for amateur users and interaction with the notes of a melody. As the user requests multiple possible continuation for a given seed melody, and also multiple continuations for each of the previous continuations, a tree or graph structure of melodies occurs. We visualize this structure with an icicle plot, where the nodes are represented by a piano roll, to show the hierarchical structure of the melody samples. To add sorting options for easier sample selection, while still displaying the structure, we added links between the nodes. Both visualizations enable listening to selected melodies. For larger numbers of generated suggestions, we added a similarity-preserving scatterplot to visualize all samples at the same time with different glyphs representing melody samples. The scatterplot improves the efficiency of sample selection, as similar samples are close together and the user can disregard entire neighborhoods if one sample does not fit at all. We support brushing the scatterplot to select neighborhoods for which we then show visual aggregations to allow for insights into groups. To evaluate our design, we conducted a pair analytics study with two participants with limited musical knowledge. Both participants were able to quickly create compositions they liked and found our approach helpful. They also learned new things about the AI, like the influence of the hyperparameter temperature on the resulting melody.

Full text and
other links
Department(s)University of Stuttgart, Institute of Visualisation and Interactive Systems, Visualisation and Interactive Systems
Superviser(s)Sedlmair, Prof. Michael; Heyen, Frank
Entry dateFebruary 15, 2022
   Publ. Computer Science