accelerating the mixing phase in studio recording productions by ...

ACCELERATING THE MIXING PHASE IN STUDIO RECORDINGPRODUCTIONS BY AUTOMATIC AUDIO ALIGNMENTNicola MontecchioUniversity of PadovaDepartment of Information Engineeringnicola.montecchio@dei.unipd.itArshia ContInstitut de Recherche et CoordinationAcoustique/Musique (IRCAM)arshia.cont@ircam.frABSTRACTWe propose a system for accelerating the mixing phase ina recording production, by making use of audio alignmenttechniques to automatically align multiple takes of excerptsof a music piece against a performance of the whole work.We extend the approach of our previous work, based on sequentialMontecarlo inference techniques, that was targetedat real-time alignment for score/audio following. The proposedapproach is capable of producing partial alignmentsas well as identifying relevant regions in the partial resultswith regards to the reference, for better integration withina studio mix workflow. The approach is evaluated usingdata obtained from two recording sessions of classical musicpieces, and we discuss its effectiveness for reducing manualwork in a production chain.1. INTRODUCTIONThe common practice in productions of studio recordingsconsists of several phases. At first the raw audio materialis captured and stored on a support. This material is subsequentlycombined and edited in order to produce a mix,which is finalized in the mastering phase for commercialrelease. Nowadays, the whole process revolves around acomputer Digital Audio Workstation (DAW).In the case of instrumental recording, the initial task involvescapturing a complete reference run-through of the entirepiece, after which additional takes of specific sectionsare recorded to allow the mixing engineer to mask performancemistakes or reduce eventual environmental noises.The role of a mixing engineer is to integrate these takeswithin the global reference in order to achieve a seamlessfinal mix [2]. The first step in preparing a mix session consistsin arranging the takes with regards to the global ref-Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page.c○ 2011 International Society for Music Information Retrieval.erence. Figure 1 shows a typical DAW session preparedout of a reference run-through (the top track) and additionaltakes aligned appropriately. Those takes usually require furthercleanup as they commonly include noise or conversationthat are not useful for the final mix. This means that,in addition to alignment, the mixing engineer identifies cutpointsfor each take that correspond to relevant regions inthe reference. The additional takes are finally blended withthe reference by crossfading short overlapping audio regionsto avoid perceptual discontinuities.Figure 1. A typical DAW mixing session.The purpose of this work is to facilitate the process ofmixing by integrating automatic (audio to audio) alignmenttechniques into the production chain. Special care is taken toconsider existing practices within the workflow, such as automaticidentification of interest points. In contrast to mostliterature on audio alignment, we are concerned with twoessential aspects: the ability to identify a partial alignmentwith an unknown starting position and the detection of regionsof interest inside the alignment. Moreover our approachpermits to achieve different degrees of accuracy dependingon efficiency requirements.Using audio material collected from two real-life recordingsessions, we show that it is possible to optimize the operationsof sound engineers by automating time-consumingtasks. We further discuss how such framework can be integratedpragmatically within common DAW software.

2. RELATED WORKAt the application level, alignment techniques were alreadyintroduced in the literature in [3]. Alignment of audio tothe symbolic representation of a piece was integrated intothe workflow, permitting the automation of the editing processthrough operations such as pitch and timing corrections.The application of these approaches is precluded inthe present context by the requirement of accessing a symbolicrepresentation of the music. Nonetheless, despite thislimitation, the work provides important insights in the integrationwithin a DAW setup.At the technological level, audio alignment has often beenthe subject of extensive research; an overview of classicalapproaches in literature can be found in [6]. In contrast totraditional methods, an important aspect of this work is theconsideration of partial results and detection of interest regions.An audio alignment method with similar aims wasintroduced in [7], that explicitly deals with the synchronizationof recordings that have different structural forms.3. GENERAL ARCHITECTUREThe proposed methodology was devised assuming that ageneric algorithm is available that is capable of aligningaudio sequences without a known starting position. Eventhough methods such as HMM or DTW [4] could have beenused for this aim, we chose to exploit our previous work [6]on sequential Montecarlo inference because of its straightforwardapplicability to the present context, its flexibilityregarding the degree of accuracy given by the availability ofsmoothing algorithms and the possibility to trade accuracyfor computational efficiency in an direct way.In the first phase a rough alignment is produced as in Figure2(a); the initial uncertainty in the alignment is due to thefact that the initial position is not known a priori. In a secondphase we identify a sufficiently long region of the alignmentthat can be reasonably approximated by a straight line, as inFigure 2(b); this region intuitively corresponds to the “correct”section of the alignment. These two phases solve thetask of placing the takes along the reference (Figure 1).The remaining steps address the tasks in which a moreaccurate alignment is required. In the third phase, the initialportion of the alignment is corrected, starting from aposition inside the region found in the previous phase andusing a reversed variant of the alignment algorithm (Figure2(c)). Finally, a refined alignment is produced by exploitinga smoothing algorithm for sequential Montecarloinference, as shown in Figure 2(d).4. METHODOLOGYThe four phases described in the previous section are highlightedin Figure 2 and described below in detail.reference time [s]reference time [s]reference time [s]reference time [s]1900 1920 1940 19601910 1930 19501900 1920 1940 19601900 1920 1940 1960p(x 0) = U(0, L)0 10 20 30 40take time [s](a) Initial alignment, using sequential Montecarlo inference.convergence region0 10 20 30 40take time [s]interest region(b) Identification of the interest region of the alignment.p(x (b)B ) = diag(1, −1) p(x B|z 0 . . . z B)0 10 20 30 40take time [s](c) Correction of the beginning of the alignment.p(x (fb)0 ) = diag(1, −1) p(x (b)0 |z B . . . z 0)0 10 20 30 40take time [s](d) Final alignment obtained using smoothed inference.Figure 2. Alignment methodology.

4.1 Initial AlignmentThe alignment problem is formulated as the tracking of aninput data stream along a reference, using motion equations.4.1.1 System State RepresentationThe system state is modeled as a two-dimensional randomvariable x = (s, t), representing the current position in thereference audio and tempo respectively; s is measured inseconds and t is the speed ratio of the performances. Theincoming signal processing frontend is based on spectralfeatures extracted from the FFT analysis of an overlapping,windowed signal representation, with hop size ∆T . In orderto use sequential Montecarlo methods to estimate the hiddenvariable x k = (s k , t k ) using observation z k at time frame k,we assume that the state evolution is Markovian.4.1.2 Observation ModelingLet p(z k |x k ) denote the likelihood of observing an audioframe z k of the take given the current position along the referenceperformance s k . We consider a simple spectral similaritymeasure, defined as the Kullback-Leibler divergencebetween the power spectra at frame k of the take and at times k in the reference.4.1.3 System State Transition ModelingLet p(x k |x k−1 ) denote the pdf for the state transition; wemake use of tempo estimation in the previous frame, assumingthat it does not change too quickly:[ ]skp(x k |x k−1 ) = N ( | µt k , Σ)k[ ]sk−1 + ∆T tµ k =k−1t k−1[ ]σ2Σ = s ∆T 00 σt 2 ∆TIntuitively, this corresponds to a performance where tempois rather steady but can fluctuate; the parameters σ 2 t and σ 2 scontrol respectively the variability of tempo and the possibilityof local mismatches that do not affect the overalltempo estimate.4.1.4 Inference AlgorithmSequential Montecarlo inference methods work by recursivelyapproximating the current distribution of the systemstate using the technique of Sequential Importance Sampling:a random measure {x i k , wi k }Ns i=1is used to characterizethe posterior pdf with a set of N s particles over thestate domain and associated weights, and is updated at eachtime step as in Algorithm 1. In particular, q(x k |x k−1 , z k ) isthe particle sampling function. In our implementation thiscorresponds to the transition probability density function; inthis case the algorithm is known as condensation algorithm.An optional resampling step is used to address the degeneracyproblem, common to particle filtering approaches;this is discussed in detail in [1,5] and in the next paragraph.The decoding of position and tempo is carried out bycomputing the expected value of the resulting random measure(which is efficiently computed as E[x k ] = ∑ N si=1 xi k wi k ).Algorithm 1: SIS Particle Filter - Update stepfor i = 1 . . . N s dosample x i k according to q(xi k |xi k−1 , z k)w i k ←ŵk i ← wi p(z k |x i k )p(xi k |xi k−1 )k−1 q(x i k |xi k−1 ,z k)ŵi k ∑j ŵj k∀i = 1 . . . N sN eff ← ( ∑ N si=1 (wi k )2 ) −1if N eff < resampling threshold thenresample x 1 k . . . xNs kaccording to ddf wk 1 . . . wNs kwk i ← N s−1 ∀i = 1 . . . N s4.1.5 InitializationInitialization plays a central role in the performance of thealgorithm; in a probabilistic context this corresponds to anappropriate choice of the prior distribution p(x 0 ).In a real-time setup the player is expected to start the performanceat a well known point of the reference; this fact isexploited in the design of the algorithm by setting an appropriatelyshaped prior distribution, typically a low-varianceone around the beginning.In the proposed situation however the initial point is notknown (it represents indeed the aim of our interest). To copewith this, the prior distribution p(x 0 ) is set to be uniformover the whole duration L of the reference performance; thealgorithm is expected to “converge” to the correct positionafter a few iterations. Figure 3 shows the evolution of theprobability distribution for the position of the input at differentmoments of the alignment.4.1.6 Degeneracy Issues w.r.t. Realtime AlignmentA relevant parameter of Algorithm 1 is the resampling threshold.The variable N eff , commonly known as effective samplesize, is used to estimate the degree of degeneracy whichaffects the random measure; degeneracy is related to thevariance of the weights {wk i }Ns 1 , and it is proven to be alwaysincreasing in absence of resampling. In a degeneratesituation most particles have close-to-zero weight, resultingin most of the computation being spent in updating particleswhich are subject to numerical approximation errors.Resampling is introduced to obviate this issue. Intuitively,resampling replaces a random measure of the true distributionwith an equivalent one (in the limit of N s → ∞) that isbetter suited for the inference algorithm. Since resampling

0 100 200 300 400 500 600ref. time [s](a) prior distribution (k = 0)0 100 200 300 400 500 600ref. time [s](b) k = 1reference time[s]1920 1940 1960 19800 10 20 30 40take time[s]0 100 200 300 400 500 600ref. time [s](c) k = 10Figure 4. Identification of the interest region.125 130 135 140ref. time [s](d) k = 50Figure 3. Evolution of p(s k |z 1 . . . z k ).introduces other problems (in particular, sample impoverishment,i.e., a small number of particles is selected multipletimes) its usage should be limited, thus producing thenecessity for a threshold on the effective sample size.In the real-time score following case [6] the mass of thedistribution is always concentrated around a small regionof the domain thus allowing the resampling threshold to berelatively low. In contrast, in a situation such as the onedepicted in Figure 3, the sparsity of the distribution in theinitial phases of the alignment imposes a much higher resamplingthreshold, otherwise many relevant hypotheses aresoon lost in the resampling phase and cannot be recovered.4.2 Identification of the Interest RegionThis phase aims at identifying a region of the alignment obtainedpreviously where it is certain that the alignment isindeed “correct”. As depicted in Figure 2(b), a typical alignmentcan be subdivided into two regions, the first one beingcharacterized by irregular oscillations (because not enoughdata has been observed yet in order to select the most probablehypothesis with enough confidence) and the second oneresembling a straight line; we will refer to the former asconvergence region and to the latter as interest region.As can be inferred by observing the plot in Figure 2(b),the most important characteristic of the interest region is itsslope. From a technical point of view, the slope should beas constant as possible for the alignment region to be significant.From a musical perspective it should be roughlyunitary, implying that the performance tempos of the singletake and the reference are approximately the same. In additionto that, the duration of the interest region should belong enough to discard noisy sections of the alignment.The interest region is identified in the following man-ner: each of many initial candidate regions w 1 . . . w W isiteratively expanded as long as it meets the criteria exposedabove; the longest of the resulting intervals is elected as theinterest region, unless none of them matches the requirements,in which case the alignment is identified as incorrect.The process described above is depicted in Figure 4(dashed horizontal lines represent the regions progressivelyexamined by the algorithm) and formalized in Algorithm 2.Algorithm 2: Identification of interest regionw 1 , . . . , w W ← regularly spaced intervals in [0, L]candidates ← ∅ for i = 1 . . . W dowhile |w i | < L dow i ← max(0, w starti−∆T), min(L, w endi+∆T)a i ← slope of LS-fit line for points in w ie i ← mean difference with LS-fit line in w iif a i ∈ [1 − ∆A, 1 + ∆A] ∩ e i < ∆E thencandidates ← candidates ∪ ielsebreakif |candidates| > 0 theninterest region ← max w ii∈candidateselsealignment is incorrect4.3 Correction of the Convergence RegionIn order to fix the convergence region of the alignment, weexploit again the sequential Montecarlo inference methodologyof 4.1, with some adaptations. The general idea isto run the algorithm “backwards”, i.e., to align the timereversedaudio streams, starting from a point in the previousalignment that is known to be correct.The starting point B is chosen inside the region of interest.The prior distribution for the backward alignmentis equal to that of the forward alignment at B, howeverwith the value of the velocity for each particle inverted:p(x (b)B ) = diag(1, −1)p(x B|z 0 . . . z B ). The audio stream of

the take is then reversed and processed by Algorithm 1, as inFigure 2(c). Experimentation shows that a narrow uniformor gaussian prior centered in (B, −1) T are for practical purposesequivalent to the form of p(x (b)B) mentioned above.4.4 Smoothing InferenceSequential Montecarlo inference algorithms are typically foronline estimation; this implies that at each instant only theinformation about the past is exploited, instead of the wholeobservation sequence. In the context of an offline applicationhowever these real-time constraints can be dropped.Both the Forward/Backward and Viterbi inference algorithmscan be deduced, respectively estimating the probability distributionat each instant given the full observation sequenceand the Maximum A Posteriori alignment. The running timeof both algorithms is quadratic in the number of particles,however this issue can be mitigated by an appropriate choiceof the prior distribution p(x (fb)0 ) such as a resampling ofdiag(1, −1) p(x (b)0 ) with a smaller numer of samples.5. EVALUATIONAn ideal evaluation of the efficacy of the proposed methodologyin the context discussed in Section 1 should aim atmeasuring the amount of work saved in production with respectto the current workflow. A discussion of our currentwork in this area is presented in Section 6.Below we evaluate the efficacy of the proposed approachregarding the initial phase of laying out the takes as in Figure1. The accuracy of the alignment in terms of latency andaverage error was evaluated in our previous work [6]; a similaranalysis could not be performed in this case, due to thelack of a (manually annotated) reference linking the timingsof each musical event for all takes to the reference recording.Moreover, in this situation the aim is rather to positioncorrectly the highest number of takes against the reference,rather than to align them with the highest possible precision.5.1 Dataset descriptionWe collected the recordings produced in two real-life sessionsby different groups of sound engineers, consisting ofapproximately 3 hours of audio data. The first one is arecording session of the second movement of J. Brahms’sextet op. 18; the second one was produced shortly after thepremiere of P. Manoury’s “Tensio”, for string quartet andlive electronics, in December 2010. Table 1 summarizestheir characteristics.5.2 Experimental ResultsWe performed the alignment of each take in the two databasesaccording to the procedure introduced in Section 4. We selectthe center point of the interest region identified in thedataset n. of rec. duration [s]ref. takes (avg,std) totalBrahms 20 + ref. 588.8 112.8, 92.0 2844.0Manoury 49 + ref. 2339.4 113.5, 94.0 7900.4Table 1. Datasets used for evaluation.second phase as the alignment reference for the whole take(we do not performs the optional two last steps).In all the test we executed, we set the number of particlesN s to be proportional to the duration of the reference (60particles per second). Our implementation aligns a minuteof audio in 2.29s for N s = 10 5 on a laptop computer with a2.4 Ghz Intel i5 processor (a single core is used).5.2.1 Brahms DatasetFor this dataset, a manual placement of all the takes withrespect to the reference recording was performed using amusical score, in order to evaluate the correctness of the automaticprocedure. Aural inspection of the data showed thatnone of the recordings but one presented undesired noises.All the takes but one were correctly aligned. In the unsuccessfulcase, the length of the recording itself was onesecond shorter than the minimum length for an interest region(15s); using last alignment point as a reference, theplacement of this take also results to be correct.5.2.2 Manoury DatasetThe dataset contains a complete run-through and 49 separatetakes. The particularity of this dataset is the presenceof undesired material for the final mix in many of the individualtakes (such as speech, practice sessions, volume andcalibration tests). Out of 49 takes, 14 contain exclusivelynoise and 21 partially. In the former case we consider thealignment correct if the file is discarded, in the latter we aimat aligning correctly the interesting portion of the take. Thisis in sharp contrast with the “cleanness” of the Brahms setand presents difficulties that were not foreseen when formulatingthe alignment procedure.Contrarily to the Brahms dataset, the evaluation of thealignment precision was done a posteriori: instead of performinga manual alignment in advance, the results of theautomatic alignment were checked. The reason for this liesbehind the length (approximately 40 minutes) and complexityof the music: even with the score at our disposal, it wasimmediately evident that a manual alignment would havetaken a very long time. It is precisely this difficulty thatsound engineers had to face.Our first experiment aligning this dataset yielded ratherpoor results on the 21 files containing noise regions of significantlength (in some cases up to more than one minute);since in almost all cases the noisy portion was at the beginning,we decided to directly align the reversed audio streams

in the first phase. With this simple adaptation the results areas follows: of the 35 files containing interesting regions, 26were correctly aligned; all of the 14 takes that contained exclusivelynoises were correctly discarded by the algorithm.The absence of false positives (no noise-only takes weremistakenly aligned) and the correct positioning of all thealigned files suggest that the simple algorithm for identificationof the interest region is robust enough to be appliedto rather short audio segments, yielding the possibility of repeatingthe alignment algorithm multiple times on differentsubregions of the audio in order to avoid noisy sections.6. WORKFLOW ADAPTATIONThe audio industry has established over the years commonstandards for mixing that are adopted in most professionalstudio records worldwide. Integration of new technologieswithin existing workflow therefore requires special attentionto existing practices within the community. To this end, weconducted several interviews with sound engineers.From an R&D standpoint, an ideal integration would bea direct implementation of this technology into the graphicaluser interface of common DAW softwares to maximizeusability. Such integration would allow novel possibilities,such as linking two tracks by means of their alignment anddefining the placement of transition points between themfor crossfading, avoiding any destructive editing regardingthe discarded audio regions. Such integration requires directcontact with software houses which are mostly close topublic domain development.An alternative solution is represented by standalone alignmenttools, whose outputs should be directly importable intoa commercial DAW. Virtually all the major DAWs and videopost production systems support the Open Media Framework(OMF) and the Advanced Authoring Format (AAF),respectively owned by Avid Technology, Inc. and by theAdvanced Media Workflow Association (AMWA) 1 . Theseare employed as interchange formats to allow interoperabilitybetween different software. An alignment software, thatwe are currently developing, could automatically constructan initial session using an interchange format that audio engineerscan use in their DAW to start the mixing process.7. CONCLUSIONS AND FUTURE WORKIn this paper, we attempted to address two issues: Introducingnovels tools generalizing audio matching algorithmsto partial alignment with relevant region detection, and theirintegration within realistic studio mixing procedure to acceleratemixing session preparation for audio engineers. Thefirst task involves adapting audio alignment techniques tosituations where there is no specific prior knowledge on thestarting point of the alignment. Such considerations wouldallow audio engineers to automatically obtain a global viewof many different individual takes with regards to a referencerun-through recording in a typical recording session, aswell as providing access to relevant parts within each take;this is a time-consuming task if done manually. We furtherdiscussed how this procedure can realistically be integratedinto common mixing workflows.Applications of the proposed technology are not limitedto the preparation of the initial mixing session: mid-level informationobtained during the alignment task can in fact befurther integrated in a studio mixing workflow. For example,our audio alignment provides useful information aboutthe tempo of a performance with regards to the referencethat can be employed as an important factor for the mixingengineer. Such integration requires further collaborationwith audio engineers to determine an optimal exploitation ofthese informations in the context of existing practices.8. REFERENCES[1] M. S. Arulampalam, S. Maskell, and N. Gordon. ATutorial on Particle Filters for Online Nonlinear/Non-Gaussian Bayesian Tracking. IEEE Transactions on SignalProcessing, 50:174–188, 2002.[2] B. Bartlett and J. Bartlett. Practical Recording Techniques:The step-by-step approach to professional audiorecording. Focal Press, 2008.[3] R. Dannenberg and N. Hu. Polyphonic Audio Matchingfor Score Following and Intelligent Audio Editors.Proc. of the International Computer Music Conference(ICMC), 2003.[4] S. Dixon and G. Widmer. Match: a Music AlignmentTool Chest. Proc. of the 6th International Conferenceon Music Information Retrieval (ISMIR), 2005.[5] R. Douc, O. Cappe, and E. Moulines. Comparison ofResampling Schemes for Particle Filtering. In Proc. ofthe 4th International Symposium on Image and SignalProcessing and Analysis (ISPA), 2005.[6] N. Montecchio and A. Cont. A Unified Approach to RealTime Audio-to-Score and Audio-to-Audio AlignmentUsing Sequential Montecarlo Inference Techniques. InProc. of the IEEE International Conference on Acoustics,Speech, and Signal Processing (ICASSP), 2011.[7] M. Müller and D. Appelt. Path-Constrained Partial MusicSynchronization. In Proc. of the 34th InternationalConference on Acoustics, Speech and Signal Processing(ICASSP), 2008.1 http://www.avid.com, http://www.amwa.tv