Augmenting Sheet Music with Rhythmic Fingerprints
Daniel Fürst, Matthias Miller, Daniel Keim, Alexandra Bonnici, Hanna Schäfer, Mennatallah El-Assady
AAugmenting Sheet Music with Rhythmic Fingerprints
Daniel Fürst * University of Konstanz
Matthias Miller ∗ University of Konstanz
Daniel A. Keim ∗ University of Konstanz
Alexandra Bonnici † University of Malta
Hanna Schäfer ∗ University of Konstanz
Mennatallah El-Assady ∗ University of KonstanzFigure 1: Augmenting sheet music with rhythmic fingerprints allows for viewing only the rhythmic aspects of a composition. Thisexcerpt shows the beginning of the
Aria from Bach’s
Goldberg Variations conveying a similar rhythm in the first and fifth measure. A BSTRACT
In this paper, we bridge the gap between visualization and musicol-ogy by focusing on rhythm analysis tasks, which are tedious dueto the complex visual encoding of the well-established CommonMusic Notation (CMN). Instead of replacing the CMN, we augmentsheet music with rhythmic fingerprints to mitigate the complexityoriginating from the simultaneous encoding of musical features. Theproposed visual design exploits music theory concepts such as therhythm tree to facilitate the understanding of rhythmic information.Juxtaposing sheet music and the rhythmic fingerprints maintainsthe connection to the familiar representation. To investigate theusefulness of the rhythmic fingerprint design for identifying andcomparing rhythmic patterns, we conducted a controlled user studywith four experts and four novices. The results show that the rhyth-mic fingerprints enable novice users to recognize rhythmic patternsthat only experts can identify using non-augmented sheet music.
NTRODUCTION
Common Music Notation (CMN) resulted from a century-long de-velopment [42] to visually encode musical information. While alter-native notations have been proposed [32], none has been adopted bythe community to replace the CMN. Hence, musicians, composers,and music analysts must be proficient at reading CMN [17]. Inex-perienced music readers face a steep learning curve to read sheetmusic. Moreover, music analysis tasks require knowledge beyondthe reading of individual notes. Such tasks involve finding harmonicprogressions [30], melodic motifs, and rhythmic patterns [47] whichhelps understanding musical structure and interpreting a compo-sition. While melody, harmony, and rhythm are simultaneouslypresent in music, analysts often need to focus on the different fea-tures separately [32]. In this paper, we concentrate on rhythm , acompound feature building on the primitive attributes meter, onset,and duration [41]. Rhythmic characteristics build the foundation ofcompositions and have a significant influence on their organizationalstructure [47]. Besides, rhythm plays a crucial role in different musicanalysis tasks, including comparative and structure analysis. * e-mail: fi[email protected] † e-mail: [email protected] Information Visualization has proven effective for music analysistasks, such as identifying similar patterns or highlighting relevant as-pects [22]. In previous work, we demonstrate how augmenting sheetmusic with harmonic fingerprints helps to identify patterns [31]. Theresults of the accompanying study confirm that with the harmonicfingerprints, even novice users could uncover harmonic patternsthat only music experts were able to see before. As both harmonyand rhythm are essential for music analysis, we argue that rhythmdeserves separate consideration. Hence, as a direct follow-up, weintroduce a rhythmic fingerprint to extend our work on augmentingsheet music.In this paper, we contribute a rhythmic fingerprint design to aug-ment sheet music (see Fig. 1) with entities that visually representrhythmic characteristics. We exploit the visual metaphor of a clockusing a radial tree layout to reflect the hierarchical structure of notedurations. Through the augmentation, we provide music readerswith additional information that supports the identification of rhyth-mic relations in a composition. To evaluate the introduced approachand the usability of the rhythmic fingerprint, we conducted a userstudy with both novices and music experts.
ELATED W ORK
Rhythm is one of the essential features of music [14, 23, 44]. Under-standing its characteristics and relations is crucial for music analysistasks such as structure analysis, pattern identification, and interpre-tation [43]. During such analytical processes, the structure of musiccan reveal rhythmic relations. However, analyzing rhythm and itsreflection in the music’s structure requires to be proficient in musictheory, which is a prerequisite that challenges novices and evenintermediately-experienced musicians [4]. Rhythm visualizationsoffer one way to overcome this obstacle, exploiting the human’svisual cognition ability. The augmentation of CMN with abstractvisualizations enables close and distant reading [20]. In previouswork, we applied such a combination to aid harmony analysis with harmonic fingerprints [31] visually.
Music Structure Analysis – A common music structure analysis task is to extract the temporal sections, often referred to by letteredlabels (i.e., A , B , C ,... ) [34]. These sections correspond to partsof the composition such as introduction, exposition, or coda andbuild the musical form. The so-derived structure segmentation ofmusic depends on musical features such as harmony, rhythm, andmelody [34]. While key changes and modulations provide an obvi-ous partitioning [9], towards phrase endings, particularly of major a r X i v : . [ c s . H C ] S e p ections, music tends to slow down, i.e., note durations becomelonger. Also, musical phrases often start with similar rhythmic pat-terns that indicate suitable segmentation split points. Consequently,the rhythmic qualities of a piece of music merit particular attention.Hence, it is often possible to utilize rhythm to infer the structureof a musical piece [26]. Boundaries that constitute the temporalsegments in the music structure frequently correlate with rhythmicchanges as in the occurrence of novel rhythmic patterns [18, 34]. Onthe other hand, the repetition of rhythmic patterns helps to identifythe musical contents that belong to one section [34]. Radial hierar-chy visualizations are particularly suitable to aid analysts in locatingnovelty and repetitions due to the hierarchy exhibited by rhythm andits cyclical appearance [10, 29, 39]. Radial Hierarchy Visualization – To visualize the hierarchical rela-tionship between different durations occurring in rhythmic patterns,Longuet-Higgins coined the rhythm tree [29]. The hierarchical datastructure within such trees is often visualized through tree maps [19]or
Icicle plots [24]. While these approaches effectively utilize spacethrough their squared appearance [19, 49], they are not suitable torepresent the temporal aspect of rhythm accurately. To classify jointvisualizations of hierarchical and time-series data, Draper et al. de-vise a taxonomy for radial hierarchy visualizations [10]. Beforedistinguishing them according to their exhibited design pattern, vi-sualizations are divided into three categories:
Polar Plot , Ring , and
Space Filling . The
Polar Plot radiates lines that convey semanticsfrom an origin to display the relationships between these branches.In contrast,
Ring visualizations display nodes on the circumferenceof a circle and highlight their relationships through connecting lines.For the design proposed in this paper, we employ a space-fillingvisualization to convey the semantics denoted to the hierarchy andto enable comparison. Schulz et al. concisely describe these require-ments to generate a tree-layout identical to a fan chart [10] [38].Draper et al. assign the visualization to their third category,
Space-Filling , within the concentric pattern [10] similar to
Sunburst visual-izations [40]. We transfer the concept of fan charts to the durationhierarchy since their structure is almost identical.
Rhythm Visualization – It is possible to visualize and analyzerhythmic aspects from musical compositions on several levels ofabstraction [20]. While aggregated visualization techniques facilitateretrieving an overview of the underlying data, they struggle withconveying the details. In Digital Humanities, scholars often carefullyexamine data to understand its meaning and underlying relations onthe close and distant reading level [5].Typically, readers of music perform close reading using the CMNthat encodes all rhythmic details [32]. For instance, the binary string as a rudimentary visual representation can aid the analysis of rhythmby considering the presence or absence of sound [27]. This ideahas been improved by the
Time Unit Box System [45] and its gen-eralization,
Drum Tablature [39]. These alternative representationsallow music analysts to focus on rhythmic characteristics at a closereading level. Detaching such alternative representations hindersusers in making a connection to the well-established CMN. Anotherapproach by Robledo exploits texture in the background of the CMN,which is highlighting rhythmic units [37] to improve their separation.These concepts use a linear sequence to communicate the rhythmicaspects visually but do not support the cyclic nature of rhythmicpatterns [39] or their symmetry [27]. Circular representations aremore suitable to address these two characteristics [27, 39].For instance, the necklace notation displays rhythmic events onthe circumference of a circle [39]. By exploiting the metaphor ofa clock [15], it supports the intuition of the connection betweenrhythmic repetitions and time. This way of presentation is suitablefor unveiling the regularity inherent to many rhythmic patterns [39].Similarly, the
Polygon notation represents beats in music by con-nected lines. These connections build up polygons enabling a com-parison of rhythmic patterns. Such abstract visualizations facilitate the understanding of underlying patterns but aggravate the connec-tion to the original representation. In the end, both close and distantreading is essential when it comes to reading music. Consequently,it is desirable to find solutions that support both concepts withoutsacrificing each other’s advantages.
Augmenting Sheet Music – In a review of visualizations for textualdata, Jänicke et al. emphasize the importance of combined closeand distant reading [20]. While the latter cannot replace the former,distant reading can point the reader towards interesting spots, e.g.,patterns in the data. Regarding the discussed abstract visualizations,there is a lack of such a connection between them and the CMN.Musicians usually achieve such a combination by augmenting vi-sualizations above or below the sheet music systems, for example,through syllables [8, 11]. In the
Generative Theory of Tonal Music (GTTM), Lerdahl et al. provide numerous annotations through ge-ometrical shapes, brackets, and trees to visualize rhythm features,such as articulation and meter [25]. While the elaborated approachesprovide an abstraction over the CMN, they are hard to compare dueto their lack of conventional structure and color, using only black andwhite [8, 11, 25]. Besides, their abstraction is either too narrow (e.g.,syllables) or too general (e.g., multi-measure tree-like structures inGTTM). To cope with this challenge, we propose a compromiseby introducing an abstract fingerprint visualization for the musicalfeature of rhythm, which can be attached to sheet music to ensurethe visual connection to the CMN.
HYTHMIC F INGERPRINT D ESIGN
Our proposed rhythmic fingerprint exploits the clock metaphor [15],a psychological anchor [50] in human cognition, and the rhythmtree [29], a music theory concept, to reflect rhythmic aspects.
Characteristics of Rhythm – Occasionally, the term of rhythm synonymously represents the phenomena of meter, accent, and tim-ing [28]. In this work, we define rhythm as referring to two primarydimensions: onset and duration [23]. We consider notes as the pres-ence of sound and its absence as rests. To quantify onset, we operatewithin the context of a single measure and calculate according toa predefined tatum [35] of 32 nd notes. Typically, the duration of anote scales with a predefined tactus [35] of quarter notes dependingon the time signature.There are diverse types of time signatures such as simple, com-pound, complex, and additive meter [36]. To address all specificattributes of each time signature type exceeds the scope of this pa-per. The following detailed description addresses the simple timesignature , but we later explain in Sect. 6 how we could extend ourapproach to deal with other time signatures like compound meter .A fundamental quality of duration is hierarchy [29]. This char-acteristic emerges in the rhythm tree as depicted in Fig. 2(a). Therhythm tree is similar to the phrase structure tree that Chomsky intro-duced for the syntax analysis in linguistics [6], due to the structuralsimilarities between language and music [1].Another characteristic of the rhythm tree is the geometric progres-sion of note durations [41]. The duration of notes adheres to d ( i ) = r i (1)where i ∈ ( − ∞ , ] ∩ Z and r =
2. Here, i represents the index of alayer in the rhythm tree (see Fig. 2(b)) and r reflects the common ratio between successive layers in the rhythm tree (i.e., the durationlength bisects at every step from top to bottom). For example, thelongest note (i.e., a whole note) has a duration of d ( ) =
4, whereasthe shortest note (i.e., a 32 nd note) has a duration of d ( − ) = . a) The rhythm tree [29] putsthe duration of notes and restswithin measures into relation. (b) This rhythm tree shows the relationship between subsequent dura-tion layers for simple time signatures only. We use two complementary color scales that preserve the duration order for rests and notes. (c) A skeleton ofthe rhythmic fin-gerprint. (d) Three voices in a mea-sure. Notes and rests arecolored according to (b).
Figure 2: We utilize (a) the hierarchy present in the rhythm tree together with (c) the clock metaphor [15] to create a space-filling concentricvisualization, the rhythmic fingerprint . (d) Within that, we encode notes and rests in the disk that corresponds to their duration. Also, we fill the arcat the note’s or rest’s offset with (b) the color corresponding to their duration. These illustrations are only suitable for simple time signatures.
The rhythm tree represents the complexity of rhythm in a measureas the distribution of rhythmic contents in the tree’s hierarchy. Thismeasure of complexity is similar to the Kolmogorov complexity [16].For example, the first rhythm (1) is more com-plex than the second rhythm (2), which is simplyrepeating a quarter note four times. Such com-plexity of rhythm affects the understanding andperformance of musical pieces [44]. Related to this complexity isthe measurement of evenness, which refers to onsets equally dis-tributed over time [46]. In this example, rhythm (2) has a higherevenness than rhythm (1) due to identical inter-onset intervals. The inter-onset interval measures the temporal distance between twosubsequent onsets. Considering rhythmic evenness helps to classifyrhythmic patterns [46]. Maximal evenness depends on whether allinter-onset intervals are equal, as it is the case for rhythm (2). Inthis case, the rhythm is even mirror-symmetrical. The notion ofsymmetry is another relevant factor in the analysis of the structureand rhythm [21, 48]. Thus, considering symmetry and evenness forvisual designs supporting rhythmic analysis is essential.
Design Rationale – Reading a radial clock is a common task thatis anchored in human intuition [15, 50]. Fuchs et al. show theadvantage of this representation for identification and comparisontasks compared to other representations [12]. Therefore, we use aradial template (see Fig. 2(c)) for the proposed rhythmic fingerprintdesign that is based on the clock metaphor [15] to reflect the temporaland repetitive aspects of rhythm. In the circular design, we layoutthe content of a measure by starting at the top center continuing in aclockwise direction. Then we map the structure of the rhythm treeto this layout (see Fig. 2(b)). Since a whole note has the longestduration, one traversal of the circle represents a duration of d ( ) = a ( i ) = td ( i ) (2)where i ∈ ( − ∞ , ] ∩ Z , t =
4, and d is the function given in Equa-tion 1. Here, i represents the layer index in the rhythm tree(see Fig. 2(b)) and t the tactus -level of a quarter note. Since a isstrictly increasing, there are more outer than inner disk arcs, whichis favorable as they have more display space devoted to them. Wedefine the tatum to a 32 nd note (i.e., i ∈ [ − , ] ∩ Z ). This generaldefinition allows for a flexible extension to a shorter tatum such as64 th notes if required. The starting point of an arc depicts a note’s onset. Therefore, thestarting point of each disk’s first arc is located at the top center withinthe fingerprint’s circle as indicated in Fig. 2(d). For the remainingarcs, we retrieve the starting points according to s ( i ) = (cid:26) cx (cid:12)(cid:12)(cid:12) x ∈ [ , a ( i )] ∩ Z (cid:27) (3)where i ∈ ( − ∞ , ] ∩ Z , c = ◦ , and a is the function given inEquation 2. Here, i represents the index of a disk and c the circum-ference of the circle. The space between two starting points in adisk visually reflects the duration of the arc beginning at the formerstarting point.To convey the rhythmic contents of a measure, we color thearcs using the color scales illustrated in Fig. 2(b). The outer disksuse a lighter color to avoid a visual bias towards their arcs sinceshorter notes tend to appear more often than longer notes. Theexemplary measure in Fig. 3 is a constructed composition of notesand rests to illustrate how three voices are encoded by the proposedrhythmic fingerprint. The third voice plays a whole note that fillsthe innermost disk, while the second voice comprises two half restsfilling the second disk. The first voice contains multiple notes ofdecreasing duration encoded through arcs in different disks. Thisexample illustrates the conflict between a note and a rest of the sameduration at the same offset (i.e., a half note and a half rest at offset 0).As both would address the same arc, we prefer to show the presenceof sound rather than its absence. Similarly, we aggregate multiplenotes or rests of the same duration at the same offset since theirmultiplicity does not change the perceived rhythm [26].Apart from the durations and onsets covered by Equation 1 and Equa-tion 3, music contains further rhythmic phenomena, such as dottednotes whose duration is given by the extension of Equation 1: d n ( i ) = n ∑ k = d ( i − k ) (4)where i ∈ ( − ∞ , ] ∩ Z . Here, n reflects the number of dots thatextend a note, i represents the layer index in the rhythm tree, and d is the function given in Equation 1. There are two additionalmechanisms the rhythmic fingerprint utilizes to accommodate thoseappearances. We visualize a dotted note whose base duration belongsto disk i in disk i + igure 3: The rhythmic fingerprint encodes simultaneous voices. The first (notes) and second voice (rests) do overlap in the first half seg-ment. Our design favors notes over rests (here, the first half note).The third voice has no overlap with the other voices. Augmenting Sheet Music with Rhythmic Fingerprints – Mu-sicXML [13] is a standard file format that is used to share digitalsheet music and also provides layout information. MusicXML hasbeen widely adopted due to the rise of services like MuseScore ,IMSLP , and their joint initiative, OpenScore which distributescompositions in the MusicXML format. We leverage this advan-tage to extract rhythm features, including onset and duration forall notes in every voice, from music sheets using music21 [7]. Atthe same time, we render the MusicXML with OpenSheetMusicDis-play (OSMD), an open-source library as a Scalable Vector Graphics.By extending OSMD with D3.js [2], we can place the rhythmic fin-gerprints on top of the score. To ensure flawless augmentation,we enlarge the space between the musical systems to position therhythmic fingerprints without overlapping with the CMN. SE C ASES
We present three use cases supported by the rhythmic fingerprintintroduced in Sect. 3. These use cases illustrate how the rhythmicfingerprint design facilitates music analysis and interpretation tasks.
Interpretation of Rhythm – Rhythm exhibits a multitude of charac-teristics, as we outline in Sect. 2. To judge a composition’s complex-ity, analysts need to examine each measure to extract the encodedrhythm. The rhythmic fingerprints support this task as they enablejudging the complexity based on the double encoding of rhythmthrough color and position. Higher color diversity indicates a highercomplexity, while the opposite holds for a narrow color spectrum.Along with the complexity of rhythm comes its evenness. This fea-ture depends on onset and duration, which are difficult to measure,especially in polyrhythms. Nonetheless, evenness is vital to com-pare rhythmic patterns and classify music based on it. The rhythmicfingerprints support this task as they reflect evenness in their visualpatterns. For performance preparations, a reader could be interestedin finding out if and where a composition is lively or slow. The colordistribution of the rhythmic fingerprints aids this task, especially forlonger sheet music, where a majority of darker colors hint at slowmusic while lighter colors at the outer layers indicate the opposite.
Music Structure Analysis – In Sect. 2, we discussed how rhythmaffects the structure of music. The repetition of and the change inrhythmic patterns constitute section boundaries. To detect rhythmic https://musescore.com/ https://imslp.org/ https://openscore.cc/ https://opensheetmusicdisplay.org/ changes and repetition, an analyst would need to closely analyzesingle measures by examining all notes to compare them with thecontent of other measures. Typically, this manual analysis process istedious and time-consuming. The annotated rhythmic fingerprintscan be of help through the visual patterns. Instead of delving into theCMN, the rhythmic fingerprints guide the user to recurrent rhythmicpatterns. By keeping the connection to the CMN, analysts can verifyhypotheses generated based on the fingerprints. Comparative Analysis of Compositions – The rhythmic finger-prints facilitate the comparative analysis of multiple compositionsthrough the combination of rhythm characteristics and music struc-ture. Instead of individually comparing measures and their notesacross compositions, the rhythmic fingerprints aggregate multiplevoices into visual patterns. The combination of their consistent struc-ture and color encoding establishes a visual appearance of measuresas single units that are more accessible to an analyst. They can visu-ally match the exhibited patterns to identify rhythmic similarities,the characteristics of the present rhythms, and music structure. Therhythmic fingerprints enable pattern matching due to their preat-tentive nature using color and consistent positioning. Comparingtwo scores with rhythmic fingerprints, a music analyst can distin-guish a fast introduction from a slow opening, for example. Theformer features short-lasting notes which appear in the outer disksof the rhythmic fingerprints, creating bright ring-like appearances.The slow-paced opening consists of longer notes that the rhythmicfingerprints display at their center, creating a dark and compact rep-resentation. The analyst can conclude the difference between thetwo pieces of music at a glance, easing the comparative analysis.
VALUATION
To evaluate the rhythmic fingerprint introduced in Sect. 3, we con-ducted a qualitative user study similar to our previous work [31].We tasked each participant to identify patterns in two music sheets,one with the rhythmic fingerprints and the second one without. Thisapproach enables us to assess the advantages of the rhythmic finger-print augmentation and elicit qualitative feedback from the users. Tohighlight differences between novices and experts, we divided theparticipants into two separate groups based on their expertise levelregarding music theory knowledge and rhythm analysis.
Our qualitative assessment of the rhythmic fingerprints comprisesthree steps: (1) The introduction of the design, (2) the analysis withand without the fingerprints, and (3) gathering feedback during theanalysis. Hereinafter, we describe the used datasets, the participantcharacteristics, and the study procedure.
Data Sets and Ground Truth – During the course of this study,the participants analyzed two music sheets,
Aria ( MS ) [33] and Variation VII ( MS ) [33] from the Goldberg Variations by JohannSebastian Bach (see Appendix). To ease the assessment of theanalysis task, we exploit available ground truth [3]. Both musicsheets MS and MS are equally long, consisting of 64 measures,and have the time signatures and , respectively. We chose thesecompositions due to their comparable complexity and musical form.The 64 measures are divided into two parts, namely P and P . Eachpart covers 32 measures and consists of a repeated pair of sections: AB for P and CD for P with a resulting form ABABCDCD asindicated by the lettered labels in Fig. 4. This overall structure isreflected by the rhythmic patterns, allowing us to compare the resultsbetween the unmodified CMN and the augmented music sheet.
Participants – We conducted our study with eight participants thatwe divide into two groups of different expertise levels. The firstgroup consists of four novices ( N - N ) who have little experience inmusic theory (mean score of 2) and never performed rhythm analysisbefore (mean score of 1). The second group includes four experts a) We asked N to highlight recurring patterns in MS without the rhythmicfingerprints. Besides identifying pattern D by three sub-patterns, he couldonly identify small parts (e.g., in A ) or none (e.g., C ) of the other patterns. (b) We asked N to highlight recurring patterns in MS with the rhythmicfingerprints. He came up with the pattern C , treated AB as a single repeatingpattern, and found multiple nested patterns throughout larger patterns. Figure 4: During the user study, we instructed the participants to find and mark as many recurring patterns as possible by colored rectangleannotations within the presented music sheet using the CMN (a) without and (b) with the rhythmic fingerprints augmentation. ( E - E ) who have more advanced experience in music theory (meanscore of 2.5) and are more proficient in rhythm analysis (mean scoreof 3.25). All participants in both groups have a university degreeand are aged 29 . ± . Procedure – The evaluation comprised three phases. First, theparticipants provided demographic information and their level ofexpertise using a Likert scale from beginner (1) to expert (5).During the second phase, we familiarized the participants with therhythmic fingerprint design. Then, we verified their understandingof the explanations by presenting examples each participant had tosolve correctly. We continued explaining the rhythmic fingerprintbased on an exemplary music sheet to introduce polyphonic rhythmsthey would encounter during the analysis.In the last study step, the participants analyzed two music sheets:one with and another without the rhythmic fingerprints. We alsorandomized the condition order such that each condition applies toone novice and one expert. For every condition, we asked the partic-ipant to identify recurrent rhythmic patterns that are exact matchesthroughout all voices, limiting each study pass to 30 minutes. Even-tually, we elicited feedback regarding the analysis and the rhythmicfingerprints, depending on the condition, during an interview. Werepeated the same process for the second pass and concluded with acomparison of both analysis sessions in the final interview to learnabout strategies for the given comparison task.
We elicited qualitative feedback from each participant and assessedtheir performance by comparing their results with the ground truth.
Pattern Identification Strategies – During the analysis, the par-ticipants followed diverse strategies to identify rhythmic patterns.Without the rhythmic fingerprints, most of the participants focusedon a single stave. N stated that he “ focused on the second voicebecause it was easier as it contained fewer notes ”. Meanwhile, N started differently and “ looked at the melody voice, becausethat’s the most distinctive ”. Later, both combined their findings withthe respective other staff to match rhythmic patterns. E took anotherapproach as they argued that “ th and 32 nd notes are [...] quitememorable or quarters as a starting point, they do not occur [...]often ”. This idea concurs with E who found that “ nd s alwaysstand out extremely ”. This strategy focuses on salient features of thevisual appearance created by the CMN to form rhythmic patterns. Given a music sheet with the rhythmic fingerprint augmentation,the participants, except for N and E , favored the fingerprints tospot recurrent rhythmic patterns. Consequently, many of the usersfocused on the rhythmic fingerprints while neglecting the CMN. Still, E “ tried to double-check [the results] ” and referred to the CMN asmeans of verification. To identify recurrent rhythmic patterns withthe rhythmic fingerprints, the participants preferred salient aspects ofthe fingerprints. As examples they gave the “ dark colour [which] isstriking ” ( N ), “ long rhythmic runs ” ( E ), the “ yellow on the outside ”( N ), and “ the green colour [which] is [...] prominent ” ( E ). Usefulness of the Rhythmic Fingerprints – When asked how theaugmentation supported the analytical tasks, the participants gavevarious insights. All of them, except for N , appreciated the rhythmicfingerprints as they eased the analysis. Not only “ have [they] greatlyaccelerated the analysis ” ( N ) but they “ create visual[ly] clearpatterns ” ( N ) such that one “ could easily find the similarities ” ( N )and “ larger continuous patterns ” ( N ). During his first analysiswith the CMN alone (see Fig. 4(a)), N did not manage to find anyground truth patterns. By contrast, in his second analysis with therhythmic fingerprints (see Fig. 4(b)), he was able to identify thetwo parts, P and P , as well as their repeating sections AB and CD ,respectively. In need of the next analytical step, E fittingly noticedthat “ a fingerprint would be helpful for a new push because it wasmuch easier to look for things that stand out ” while analyzing theCMN alone. E even goes so far as to say that “ someone who hasnever dealt with rhythm before [could] still do an analysis of rhythmhere ” and “ [wouldn’t] need to know notes ”. Challenges – None of the participants encountered the rhythmicfingerprint before our evaluation. The participants faced differentchallenges during the analysis. At start, many participants “ found itextremely difficult to get used to the fingerprint ” ( E ). Apart fromthe steep learning curve, the participants struggled with the currentcolor usage. Particularly, E found that “ the colors are too similar,especially the red [ones] ” which E emphasized, concerned that “ the32 nd s are difficult to see ” in contrast to the white background. Theyfurther elaborated on the consequence being that “ it is more difficultto search on the outer circle compared to the inner one ” which led N to “ look at them almost as closely as the notes ”, hampering theusefulness of the rhythmic fingerprints. If not looked at them closelyenough, N and E feared “ a likelihood of confusion because thecircles look very similar ” ( N ). N and E experienced a similarrritation as they “ lost the context [when only focused] on the glyphs ”( N ) and had a hard time “ to remember the fingerprints and to findthem somewhere else ” ( E ). ISCUSSION
Based on the study results we assess the pattern identification per-formance of the participants. With the fingerprints, each participanthad a similar or better performance than with the CMN only. Sincewe provided an elaborated introduction on the rhythmic fingerprintsbefore performing the actual analysis task, both novices and expertsunderstood and thus profited from the rhythm visualization. Thisfamiliarity could be a reason for the trust of the users towards therhythmic fingerprints. Although N and E primarily relied on theCMN, the remaining participants mainly worked with the visual-ization. Although some users reported a steep learning curve forthe rhythmic fingerprints, N argued that the use of color may in-spire musicians to analyze the rhythm. E added that the rhythmicfingerprints could give a new impulse during the rhythm analysis. The main discussion points gathered during the rhythm analysisprocess and the user’s feedback are: (1) the fingerprint inherentlyencodes rhythmic complexity, (2) the fingerprint does not separatebetween different voices, and (3) the fingerprint encodes a limitednumber of time signatures.
Rhythm Complexity – The rhythm tree and the rhythmic finger-prints, express complexity through the diversity of onsets and du-rations. E accurately noticed that a greater variety corresponds tohigher complexity. Therefore, the amount of present notes influ-ences readability. In the final interview, E reported that due to thelimited space of the inner disks and the adjacent placement, it waschallenging to distinguish the colored arcs. Analogously, E statedthat due to the smaller size, the outer disks were more challenging tocompare than the inner disks. While the CMN enables music readersto view multiple voices and their rhythmic progression separately,the rhythmic fingerprint aggregates the commonalities and revealsthe rhythm perceived by listeners. Consequently, the fingerprints donot replace the well-established CMN, which can be used by ana-lysts to understand the details of multiple voices. We also learnedthat it is currently difficult to see the arcs at the outer disk due to thesmall lightness difference to the background. To address this issueor other visual impairment issues, it is no effort to change the colorscale according to the reader’s needs. Voice Separation – In a multi-voice setting, the current design doesnot support readers distinguishing separate voices as in the CMN.To achieve this, readers need to examine the CMN to understandvoice separation. Due to our decision to emphasize the presenceof notes over rests in cases of overlap with the same offset andduration, the rhythmic fingerprint aggregates simultaneous notesand rests. As outlined in Sect. 3, we favor the presence of soundover its absence. Consequently, the rhythmic fingerprint is notbidirectional. It visually simplifies parallel rhythmic content toextract perceived rhythm, which is not altered by the multiplicity ofnotes with identical onset and duration. Due to this simplification,the rhythmic fingerprint design does not support multiple voiceanalysis. Tailoring the design to support such tasks would increasethe design complexity since more information would be represented.
Time Signature – One traversal of the rhythmic fingerprint repre-sents a whole note’s duration. Thus, even longer durations cannotbe displayed correctly. Simple time signatures, such as common, , or cut time, can be analyzed without any issues. E noticed thatthe design of the rhythm visualization allowed him to determinea composition’s time merely by looking at the appearance of therhythmic fingerprint. Meanwhile, none of the other participantsfaced issues during their analysis due to our design choice. Contrary,the visualization does not properly accommodate all possible time signatures, such as . Since such time signatures would require morethan one traversal, only two instances of the rhythmic fingerprintcould adequately display the rhythmic content of such a measure. While the study showed that the rhythmic fingerprints supportthe analysis of rhythmic patterns even when used by novice read-ers, there are still restrictions and limitations of the proposedapproach. First, there are numerous time signatures of varyingcomplexity [36]. The most common time signatures are sim-ple (e.g., ) and compound (e.g., ) meter. The current fin-gerprint skeleton (see Fig. 2(c)) only addresses simple meters.While it is theoretically possible to visualizecompound meters with the current design, itdoes not properly accommodate for the tertiarygrouping, e.g., three eighth notes that are en-compassed by a dotted quarter note. The twoskeletons to the right show how and differ(i.e., red lines in the 2 nd inner disk) regardingtheir skeletons. The main difference betweenthe duple and the triple meter is the perceptionof emphasized beats in a measure [29], which isespecially relevant for performers who need toknow which notes to emphasize. Typically, in meter, dotted quarternotes are indicating the regular pulse of the composition.There are different directions in which to extend the proposeddesign further. We plan to research how the design can supportother time signatures [36] with the additional use of interactiontechniques. Consequently, we want to tailor the visualization tothe individual needs of music analysts to overcome the currentrestrictions of static rhythmic fingerprints. For instance, DetailOutside and
Detail Inside are interaction techniques designed toincrease the readability in dense fan charts [40]. We want to providea visual music analysis interface that integrates such interactionopportunities with digital sheet music to facilitate the access basedon MusicXML [13]. Moreover, we plan to extend the temporalrange for the fingerprints from single measures to larger sectionsof a musical composition to provide overviews about the rhythmiccontent. With this, we take the view that music analysis can furtherbenefit from visualization research for close and distant reading,an important concept in digital humanities [20]. To investigate thepotential of other designs, we plan to organize design workshops andcompetitions to elicit new ideas for further musical visualizationssuitable for music analysis tasks.
ONCLUSION
We proposed a visualization for rhythm by exploiting a concept ofmusic theory, the rhythm tree, to extend traditional music notation.The augmentation of sheet music with the rhythmic fingerprintscombines typical close reading with distant reading of sheet music.To evaluate our approach, we conducted a qualitative user studywith four novices and four experts. The user performance assess-ment indicates that our visualization improves the identification ofrhythmic patterns. During the analysis, users were able to determinethe music structure and characteristics of rhythm with the help ofthe rhythmic fingerprints. Through the participants’ feedback, wealso identified difficulties that analysts face while working with theproposed rhythmic fingerprints. These include the readability ofcomplex rhythms, the separation of voices, and the display of irra-tional or complex time signatures. We plan to compare differentlayout strategies and color scales in future work to address thesechallenges. We aim to combine the rhythmic with the harmonicfingerprints from our previous work [31], enabling the joint analysisof harmony and rhythm based on MusicXML.
EFERENCES [1] T. Bershadskaya. Analogies and Parallels in the Structure of Musicand Verbal Languages. In
Language, Music, and Computing , pp. 3–10,2015. doi: 10.1007/978-3-319-27498-0_1[2] M. Bostock, V. Ogievetsky, and J. Heer. D Data-Driven Documents.
IEEE Trans. on Vis. and Comp. Graph. , 17(12):2301–2309, 2011. doi:10.1109/TVCG.2011.185[3] W. Breig. Bachs Goldberg-Variationen als zyklisches Werk.
Archiv fürMusikwissenschaft , 32(4):243–265, 1975. doi: 10.2307/930762[4] W. W.-Y. Chan. A Report on Musical Structure Visualization, 2007.[5] M. F. Cheema, S. Jänicke, and G. Scheuermann. Enhancing CloseReading. In
Digital Humanities 2016, Conf. Abstracts, JagiellonianUniversity & Pedagogical University , pp. 758–761, 2016.[6] N. Chomsky.
Syntactic Structures . Number 4 in Janua Linguarum.Series minor. De Gruyter Mouton, 14 ed., 1985.[7] M. S. Cuthbert and C. Ariza. Music21: A Toolkit for Computer-AidedMusicology and Symbolic Music Data. In
Proc. of ISMIR , pp. 637–642.International Society for Music Information Retrieval, 2010.[8] B. Dalby. Toward an Effective Pedagogy for Teaching Rhythm: Gordonand Beyond.
Music Educators Journal , 92(1):54–60, 2005. doi: 10.2307/3400228[9] R. B. Dannenberg and M. Goto. Music Structure Analysis from Acous-tic Signals. In
Handbook of Signal Processing in Acoustics , pp. 305–331. Springer, 2008.[10] G. M. Draper, Y. Livnat, and R. F. Riesenfeld. A Survey of RadialMethods for Information Visualization. In
IEEE Trans. on Vis. andComp. Graph. , 2009. doi: 10.1109/TVCG.2009.23[11] D. P. Ester, J. W. Scheib, and K. J. Inks. Takadimi: A Rhythm Systemfor All Ages.
Music Educators Journal , 2006. doi: 10.2307/3878473[12] J. Fuchs, F. Fischer, F. Mansmann, E. Bertini, and P. Isenberg. Eval-uation of Alternative Glyph Designs for Time Series Data in a SmallMultiple Setting. In
Conf. on Human Factors in Computing Systems ,pp. 3237–3246, 2013. doi: 10.1145/2470654.2466443[13] M. Good. MusicXML: An Internet-Friendly Format for Sheet Music.In
XML Conf. Proc. , 2001.[14] F. Gouyon and S. Dixon. A Review of Automatic Rhythm DescriptionSystems.
Computer Music Journal , 29(1):34–54, 2005. doi: 10.1162/comj.2005.29.1.34[15] C. Guo, S. Wei, M. Li, Z. C. Qian, and Y. V. Chen. Comparison ofCircle and Dodecagon Clock Designs for Visualizing 24-Hour CyclicalData. In
Lecture Notes in Computer Science , 2017. doi: 10.1007/978-3-319-58640-3_5[16] V. Hashemi and S. Ramezani. Inducing Meter for a Rhythm andMeasures for Rhythm Complexity, 2008.[17] C. Hultberg. Approaches to Music Notation: The printed score asa mediator of meaning in Western tonal tradition.
Music EducationResearch , 4(2):185–197, 2002. doi: 10.1080/1461380022000011902[18] K. Jensen, J. Xu, and M. Zachariasen. Rhythm-Based Segmentation ofPopular Chinese Music. In
Proc. of ISMIR , 2005.[19] B. Johnson and B. Shneiderman. Tree-Maps: A Space-Filling Ap-proach to the Visualization of Hierarchical Information Structures. In
Proc. of the Conf. on Vis. , 1991. doi: 10.1109/visual.1991.175815[20] S. Jänicke, G. Franzini, M. F. Cheema, and G. Scheuermann. OnClose and Distant Reading in Digital Humanities: A Survey and FutureChallenges. In
Eurographics Conf. on Vis. , pp. 83–103, 2015. doi: 10.2312/eurovisstar.20151113[21] D. Kempf. What is symmetry in music?
Int. Rev. of the Aesth. and Soc.of Music , 27(2):155–165, 1996. doi: 10.2307/3108344[22] R. Khulusi, J. Kusnick, C. Meinecke, C. Gillmann, J. Focht, andS. Jänicke. A Survey on Visualizations for Musical Data.
ComputerGraphics Forum , 2020. doi: 10.1111/cgf.13905[23] C. L. Krumhansl. Rhythm and Pitch in Music Cognition.
PsychologicalBulletin , 126(1):159, 2000. doi: 10.1037/0033-2909.126.1.159[24] J. B. Kruskal and J. M. Landwehr. Icicle Plots: Better Displays forHierarchical Clustering.
The American Statistician , 37(2):162–168,1983. doi: 10.2307/2685881[25] F. Lerdahl and R. S. Jackendoff.
A Generative Theory of Tonal Music .MIT Press, 1996.[26] F. Levé, R. Groult, G. Arnaud, C. Séguin, R. Gaymay, and M. Giraud. Rhythm Extraction from Polyphonic Symbolic Music. In
Proc. of theInt. Soc. of Music Information Retrieval Conf. , 2011.[27] Y. Liu and G. T. Toussaint. Mathematical Notation, Representation,and Visualization of Musical Rhythm: A Comparative Perspective.
Int. J. Mach. Learn. Comput. , 2(3):261–265, 2012. doi: 10.7763/ijmlc.2012.v2.127[28] J. London. Rhythm. Grove Music Online, Oxford Music Online,2020. Retrieved July 27, 2020, from 10.1093/gmo/9781561592630.-article.45963.[29] H. C. Longuet-Higgins. The Perception of Music.
Interdis-ciplinary Science Reviews , 3(2):148–156, 1978. doi: 10.1179/030801878791926065[30] D. Malandrino, D. Pirozzi, and R. Zaccagnino. Visualization and MusicHarmony: Design, Implementation, and Evaluation. In
Int. Conf. Inf.Vis. , pp. 498–503, 2018. doi: 10.1109/iV.2018.00092[31] M. Miller, A. Bonnici, and M. El-Assady. Augmenting Music Sheetswith Harmonic Fingerprints. In
Proc. of the ACM Symp. on DocumentEngineering , pp. 1–10, 2019. doi: 10.1145/3342558.3345395[32] M. Miller, J. Häußler, M. Kraus, D. Keim, and M. El-Assady. Analyz-ing visual mappings of traditional and alternative music notation. In
IEEE VIS Workshop on Visualization for the Digital Humanities , 2018.[33] MuseScore. Goldberg Variations, 2016. Retrieved May 31, 2020, from https://musescore.com/user/13172/scores/1671046 .[34] M. Müller.
Music Structure Analysis , pp. 167–236. Springer, 2015.doi: 10.1007/978-3-319-21945-5_4[35] M. Müller.
Tempo and Beat Tracking , pp. 303–353. Springer, 2015.doi: 10.1007/978-3-319-21945-5_6[36] S. M. Rashid, D. De Roure, and D. L. McGuinness. A Music TheoryOntology. In
Proc. of Semantic Applications for Audio and Music , pp.6––14. ACM, 2018. doi: 10.1145/3243907.3243913[37] D. L. Robledo. Method of Representing Rhythm in Music Notationand Display Therefore, 2010. US Patent 7,763,790.[38] H. J. Schulz, Z. Akbar, and F. Maurer. A Generative Layout Approachfor Rooted Tree Drawings. In
IEEE Pacific Visualization Symp. , 2013.doi: 10.1109/PacificVis.2013.6596149[39] W. A. Sethares.
Rhythm and Transforms . Springer, 2007. doi: 10.1007/978-1-84628-640-7[40] J. T. Stasko and E. Zhang. Focus+Context Display and NavigationTechniques for Enhancing Radial, Space-Filling Hierarchy Visualiza-tions. In
IEEE Symp. on Inf. Vis. , pp. 57–65. IEEE Computer Society,2000. doi: 10.1109/INFVIS.2000.885091[41] K. Stone. Problems and Methods of Notation.
Perspectives of NewMusic , 1(2):9–31, 1963. doi: 10.2307/832100[42] H. Strayer. From Neumes to Notes: The Evolution of Music Notation.
Musical Offerings , 4(1):1–14, 2013. doi: 10.15385/jmo.2013.4.1.1[43] K. Swanwick.
Musical Knowledge: Intuition, Analysis and MusicEducation . Routledge, 1994.[44] E. Thul and G. T. Toussaint. Rhythm Complexity Measures: A Compar-ison of Mathematical Models of Human Perception and Performance.In
Proc. of the Int. Soc. of Music Information Retrieval Conf. , 2008.[45] G. Toussaint. The Geometry of Musical Rhythm. In
Discrete and Com-putational Geometry , pp. 198–212, 2005. doi: 10.1007/11589440_20[46] G. Toussaint. Computational geometric aspects of rhythm, melody, andvoice-leading. In
Computational Geometry: Theory and Applications ,2010. doi: 10.1016/j.comgeo.2007.01.003[47] G. Toussaint.
The Geometry of Musical Rhythm . Chapman and Hal-l/CRC, 2nd ed., 2020. doi: 10.1201/9781351247771[48] G. Toussaint, L. Matthews, M. Campbell, and N. Brown. Measur-ing musical rhythm similarity: Transformation versus feature-basedmethods.
JIMS , 6(1):23–53, 2012. doi: 10.4407/jims.2012.12.002[49] H. van de Wetering, N. Klaassen, and M. Burch. Space-ReclaimingIcicle Plots. In
IEEE Pac. Vis. Symp. , pp. 121–130. IEEE, 2020. doi:10.1109/PacificVis48177.2020.4908[50] I. Vessey. Cognitive Fit: A Theory-Based Analysis of the GraphsVersus Tables Literature.
Decision Sciences , 22(2):219–240, 1991. doi:10.1111/j.1540-5915.1991.tb00344.x
PPENDIX Allegretto Figure A.1: The dataset “Aria” from the Goldberg Variations by Johann Sebastian Bach without the fingerprints ( MS ). Figure A.2: The dataset “Variation VII” from the Goldberg Variations by Johann Sebastian Bach without the fingerprints ( MS ). Allegretto Figure A.3: The dataset “Aria” from the Goldberg Variations by Johann Sebastian Bach with the fingerprints ( MS ). Figure A.4: The dataset “Variation VII” from the Goldberg Variations by Johann Sebastian Bach with the fingerprints ( MS ).igure A.5: The dataset MS without the fingerprint augmentation annotated by N . The labels illustrate the underlyingground truth patterns. More examples annotated by the other participants can be found here: https://osf.io/jx8dy/.Figure A.6: The dataset MS with the fingerprint augmentation annotated by N2