Volume 1, Issue 1, January 2011 - DROPS - Schloss Dagstuhl

92 11041 – Multimodal Music Processing
source separation could be improved when some additional information about the sources
such as the musical score, beats, or the sung melody is known. A potential application
of active listening involving audio-informed source separation and watermarking was also
discussed.
As second topic, we discussed possible strategies for combining features that are calculated
at different frame rates or that represent different semantics levels. In particular, it was
shown that, besides the well known early and late fusion strategies, it may be appropriate to
consider other approaches that iteratively (or jointly) exploit information given by the various
modalities to improve estimates for the other modalities. In the same spirit, an example of
cross-modality search was discussed. More specifically, it was shown how cross-correlation
between semantic functions automatically extracted from each stream can lead to efficient
inter-modality search for information retrieval or content authoring. Here, specific examples
on music videos were given including a short demonstration on an audio-based video search
prototype.
As a third topic, the strategy of considering the time evolution of audio features simultaneously
at different temporal levels was addressed. This so-called multi-level feature
integration was discussed with a specific focus on the appropriateness to combine both early
and late fusion strategies. Finally, a set of questions was selected intended as initial input
for further group discussions.
3.31 Personalization in Multimodal Music Retrieval
Markus Schedl (Universität Linz, AT)
License Creative Commons BY-NC-ND 3.0 Unported license
© Markus Schedl
This talk provided an overview of current research endeavors and existing solutions in
multimodal music retrieval, where the term “multimodal” relates to two aspects. Accounting
for the context of a piece of music or an artist constitutes the first aspect, while the second
one relates to incorporating the user context. Adding “traditional” signal-based audio
analysis, I proposed a model that incorporates three broad categories of influence factors for
music retrieval and music similarity computation: music content, music context, and user
context. The music context is introduced as all the information that is important to the
music, albeit not directly extractable from the audio signal. Such information includes, for
example, editorial or collaboratively assembled meta-data, lyrics in textual form, cultural
background of an artist, or images of album covers. The user context, in contrast, is defined
by various external factors that influence how a listener perceives music. It is therefore
strongly related to user modeling and personalization, both facets of music information
research that have not gained large attention by the MIR community so far. In my estimation,
adding personalization aspects to existing music retrieval systems (for example, playlist
generators, recommender systems, or visual browsers) constitutes one key issue in future
MIR research.