Linking Sheet Music and Audio – Challenges and New Approaches

4 Linking Sheet Music and Audio Recordings
(a) Score-audio synchronization on the measure-level. Time segments in the audio stream are mapped to
individual measures in the score representation. The depicted audio track contains a repetition. Therefore,
the according score measures have to be mapped to both audio segments.
(b) Score-audio mapping on the detail level of pieces of music. The score and the audio data are segmented
into individual pieces of music. Afterwards, the correct score-audio pairs have to be determined.
Figure 2 Examples for score-audio synchronization on different detail levels.
to deviations in the global music structure. In Section 4, we discuss common computational
approaches to sheet music-audio synchronization and present various strategies how the
resulting global differences between documents can be handled within the synchronization
pipeline. Finally, in Section 5, we describe some applications and novel interfaces that are
based on synchronization results. We conclude the paper with an outlook on future work. A
discussion of relevant work can be found in the respective sections.
2 Task Specification
The goal of music synchronization is the generation of semantically meaningful bidirectional
mappings between two music documents representing the same piece of music. Those
documents can be of the same data type (e.g., audio-audio synchronization) or of different
data types (e.g., score-audio synchronization or lyrics-audio synchronization). In the case of
score-audio synchronization the created linking structures map regions in a musical score,
e.g., pages or measures, to semantically corresponding sections in an audio stream (see Figure
2).
Although the task of score-audio synchronization appears to be straightforward, there
exist several aspects along which the task and its realization can vary (see Figure 3). The