Addressing database mismatch in forensic speaker ... - ATVS

Addressing database mismatch in forensic speaker recognitionwith Ahumada III: a public real-casework database in SpanishDaniel Ramos , Joaquin Gonzalez-Rodriguez ,Javier Gonzalez-Dominguez and Jose Juan Lucena-Molina ¡ATVS - Biometric Recognition Group, Escuela Politecnica SuperiorC./ Francisco Tomas y Valiente 11, Universidad Autonoma de Madrid E-28049 Madrid, Spain¡ Acoustics and Image Processing Department, Criminalistic ServiceDireccion General de la Policia y de la Guardia Civil, Ministerio del Interior, Madrid, Spaindaniel.ramos@uam.esAbstractThis paper presents and describes Ahumada III, a speechdatabase in Spanish collected from real forensic cases. In itscurrent release, the database presents ¢£ male speakers recordedusing the systems and procedures followed by Spanish GuardiaCivil police force. The paper also explores the usefulness ofsuch a corpus for facing the important problem of database mismatchin speaker recognition, understood as the difference betweenthe database used for tuning a speaker recognition systemand the data which the system will handle in operationalconditions. This problem is typical in forensics, where variabilityin speech conditions may be extreme and difficult to model.Therefore, this work also presents a study evaluating the impactof such problem, for which a corpus quoted as NIST4M(NIST MultiMic MisMatch) has been constructed from NISTSRE 2006 data. NIST4M presents microphone data both in theenrolled models and in the test segments, allowing the generationof trials in a variety of strongly mismatching conditions.Database mismatch is simulated by eliminating some microphonechannels of interest from the background data, and computingscores with speech from such microphones in unknowntesting conditions as usually happens in forensic speaker recognition.Finally, we show how the incorporation of AhumadaIII as background data is useful to face database mismatch inreal-world forensic conditions.1. IntroductionThe interest in using automatic systems for forensic speakerrecognition has increased in the last years. The main reasons forthis are the improvement of accuracy in the technology [1] and amore comprehensive study about the role of automatic speakerrecognition in forensic science [2]. However, speaker recognitiontechnology have still important challenges to address. First,speech data scarcity remains a problem for automatic systems.Many forensic cases involve recovered questioned recordingspresenting a small amount of speech, often under unfavourablequality conditions. Recent research has focused on increasingthe robustness and accuracy of speaker recognition technol-This work has been supported by the Spanish Ministry of Educationunder project TEC2006-13170-C02-01. J. G.-D. also thanks SpanishMinistry of Education for supporting his doctoral research. We alsothank Ana Rosa Gonzalez-Sanz and people from the Acoustics and ImageProcessing Department from Guardia Civil for their important effortin collecting data for forensic purposes.ogy under short-duration conditions [3]. Second, session variabilitymismatch introduces variation to different utterances ofthe same speaker, typically due to factors such as transmissionchannel, speaking style, speaker emotional state, environmentalconditions, recording devices, etc. This variability seriously degradesthe performance of a system, and its compensation hasbeen subject of abundant recent research [4, 5]. In fact, this hasbeen a major topic of research in recent Speaker RecognitionEvaluations (SRE) conducted by NIST [1]. Finally, we identifythe database mismatch problem, understood as the variation inthe conditions between the dataset used for tuning an automaticspeaker recognition system (referred to as background or developmentdatabase) and the data used in real-world operationalconditions (known as evaluation or operational database).Database mismatch is a typical situation in forensic speakerrecognition, mainly because the limitation in the availability ofreal-casework databases for system tuning, and also becausethe conditions of the speech in real-world forensic recordingare extremely variable. Due precisely to session variability andextreme conditions, there are several frequent situations wherethe database mismatch may constitute a serious problem. Onthe one hand, problematic incriminatory questioned speech maytypically include:¤ Telephone wire-taps. The development database may nothave speech data recorded in the same conditions as thewire-tapping system. This problem may become seriousif an effort has not been made by the forensic laboratoryin order to collect field data using their recording systemat hand. The acquisition of such database is usually expensiveand time-consuming, as the final corpus shouldhave a significant size and represent the session variabilityof telephone recordings used in casework.¤ Distant or hidden microphone, in cases where such a deviceis present on an individual or a prepared room. Inthis case, the environmental conditions of the recordingsare extremely variable depending on the room characteristics,the microphone position and layout, etc. It is virtuallyimpossible to simulate such conditions for the purposeof recording a representative database, especially ifit is desired to take all possible cases into account.On the other hand, control speech recordings from a givensuspect may be problematic in the following usual situations:¤ Telephone conversations for which the suspect recognizesto be the author, either recorded at police depen-

dencies or wire-tapped. This is exactly the same case asfor a recovered wire-tap.¤ Speech recorded using a microphone. Although therecording conditions are much more controlled than inthe hidden microphone case for questioned recordings,variability in the room layout, environmental conditionsand microphone types makes extremely difficult to predictwhich kind of speech will be needed for a representativebackground database.Although database mismatch is commonly seen as a problemin the community and among forensic experts, to theauthors’ knowledge, rigorous experimental studies regardingthis important issue in forensic speaker recognition have beenmainly focused on telephonic databases. With the release ofNIST multi-microphone databases in SRE 2005 and 2006 [1],a richer dataset has been available for research in the topic. Infact, realistic databases for forensic speaker recognition are importantfor two main reasons: first, more data will be availablefor research in database mismatch; and second, forensic laboratorieswill be able to improve the robustness of their speakerrecognition systems in casework conditions.Driven by these needs, this paper presents the Ahumada IIIdatabase, a real-casework publicly available corpus in Spanish,which has been acquired by the Acoustics and Image ProcessingDepartment of Spanish Guardia Civil. In its current release,Ahumada III Release 1 (Ah3R1), it includes speech data fromreal forensic cases recovered using one of the typical recordingsystems from Guardia Civil, namely analog magnetic tapescontaining GSM tappings. Moreover, the database is being extendedwith more material under this platform and also usingSITEL, a Spanish nationwide digital tapping system. Ah3R1includes variability in conditions such as noise, environmentalcharacteristics, emotional state, country and region of originand dialect of speakers, etc. Next releases will significantly increasethe amount of data presenting strong variability from realcases.This work also explores the database mismatch problem usingthe presented Ahumada III database and a corpus generatedfrom multi-microphone data from NIST SRE 2006, namely theNIST4M database (NIST MultiMic MisMatch). This paper isorganized as follows. First, the Ahumada III database is describedin the context of Guardia Civil operative procedures forforensic speaker recognition. The experimental section thenpresents results which illustrate the impact of database mismatchin system performance in two different ways: databasemismatch using NIST multi-microphone corpora, and robustnessin real-casework conditions using Ahumada III. Resultsshow the importance of the database mismatch problem andencourages research in the field and data collection. Finally,conclusions are drawn.2. Addressing database mismatch in realcasework: the Ahumada III databaseIn the last years, Spanish Guardia Civil has done a significanteffort on the application of automatic speaker recognition systemsto forensic voice evidences [2]. Following a Bayesianframework, and pursuing transparency in their reports, muchof this effort has concentrated in the assessment of their systemin the sake of testability, as a demanding requirement inthe new paradigm of forensic identification sciences [6]. Mostvoice evidences arriving to Guardia Civil laboratories have twopossible origins. First, digitized analog magnetic recordingsfrom GSM mobile calls are typical in cases between 1995 and2004. From those recordings of this type received in the lastten years, those authorized (case by case) by the correspondingjudge after a trial, have been added to a database registered inthe Spanish Ministerio del Interior, known as Base de Datos deRegistros Acústicos (BDRA) 1 . Second, nationwide digital interceptionsystem (SITEL) has been used since 2005 by the twoSpanish State Police forces. This system records digital wiretapsdirectly connected to all mobile telephone operators.With the purpose of robustness in real-case conditions andproper assessment of systems, Guardia Civil has recorded severalspeech databases in the last decade. Back in 1998, the AhumadaI corpus was collected [7] containing 100 male speakersin telephone and microphone recordings. A subcorpus ofAhumada I spontaneous telephone speech was used in NISTSpeaker Recognition Evaluations both in 2000 and 2001.complementary database known as Gaudi with 100 femalespeakers was recorded in 2001 2 . From 2004 to 2006, the Baezadatabase was recorded, including GSM and microphone spontaneousconversational speech sessions recorded at the sametime, from ¡¢£ males and ¡¢ females. Baeza has shown, upto this moment, the most relevant reference population to beused in real cases, as it was recorded through the same SpanishGSM network as in actual cases. In 2008, all 100 male speakersfrom Ahumada I are still available and are to be again recordedthrough GSM and SITEL. In this way, Ahumada II (as is to beknown) will constitute a major contribution for the analysis oflong-term stability and degradation of speaker features after tenyears. As most new cases come through SITEL recordings andare to be, by this time, evaluated with Baeza as reference population,also in 2008 almost 100 speakers from Baeza (differentfrom Ahumada speakers) are to be recorded again through SI-TEL. This database will be known as Ahumada IV and will beused for system assessment.2.1. The Ahumada III databaseAhumada III consists of authorized conversational speech fromreal cases both from BDRA and SITEL. The expected size ofthe database in number of speakers and variety of conditionsaddressed is huge both in terms of number of available calls andamount of data. However, as conditions are not uniform, andspeech recordings have to be authorized one by one, differentreleases of the database will be progressively available.Ahumada III Release 1 (Ah3R1) consists ¢£ of speakersfrom a number of real cases with GSM BDRA calls acrossSpain, with a variety of country of origin of speakers, emotionaland acoustic conditions, and dialects in the case of Spanishspeech.The unique low variability dimension is gender,as all of them are male speakers. All ¢£ speakers in Ah3R1have two minutes of speech available from a single phone callto be used as unquestioned (control) recording, with the purposeof model training or voice characterization. Additionally,ten speech segments for £ speakers and five segments for £speakers are included for testing issues, each one from a differentcall. Such fragments present between ¡ and ¢¤ secondsof speech, with an average duration of £ seconds. An evaluationprotocol, equivalent to that of NIST SRE [1], is availableand ready to be used. In this way, experienced NIST SRE par-1 With reference public scientific file number 1981420003 fromSpanish Guardia Civil, Orden Ministerial INT/3764/2004 de 11 denoviembre.2 A 25 Ahumada male subcorpus and an equivalent 25 Gaudi femalesubcorpus are freely available in http://atvs.ii.uam.es/databases.jsp.A

ticipants can run and assess immediately their systems on theAh3R1 data. Ah3R1 is publicly available for research purposesunder a license agreement 3 to be signed with Guardia Civil(contact: crim-acustica@guardiacivil.es). Sample speech segmentsare ready to be directly heard in ATVS website in orderto perceive the quality and diversity of Ah3R1 recordings.3. Experiments on database mismatchIn this section we present results illustrating the database mismatchproblem, and how Ahumada III will improve accuracyin real forensic casework. For score computation, the ATVSGMM system has been used, where speech data known to comefrom a given speaker is represented using Gaussian MixtureModels (GMM) adapted from an Universal Background Model(UBM). GMM of ££¢ mixtures have been used for modellingMFCC-based features with channel compensation based on featurewarping and factor analysis [4].3.1. Database mismatch with NIST4M databaseAn evaluation corpus, namely NIST4M (NIST MultiMic Mis-Match), has been constructed using NIST SRE 2006 data.This corpus aims at simulating unfavourable session variability.Moreover, utterances in NIST SRE 2006 from all speakers havingmicrophone conversations have been combined in NIST4Mfor trial generation, having multimicrophone data both in theenrolled models and in the test segments. Database mismatchhas been simulated by eliminating all the data for a given microphonefrom the background data, using such microphone onlyin NIST4M evaluation data. Thus, trials for that microphonewill be performed without considering such kind of data in thebackground set, and therefore under database mismatch. Thissituation is usual in forensic speaker recognition.For the construction of NIST4M, half of the telephone utterancesof each speaker were selected as testing segments andthe rest as training data. For microphone data, microphonesin one session per speaker has been used for training data, and¢ microphones from each of the rest of sessions are used fortesting. The microphone types changed with different speakers,in such a way that the data for all microphone types tend tobe balanced. Trials were generated by performing all possiblecombinations between the same speaker utterances for targettrials, and the next £¢ speakers for non-target trials, avoidingdifferent-gender comparisons.Development data has been selected from past NIST SRE.Microphone data from NIST SRE 2005 has been divided inthree sub-sets, separately used for training factor analysis parameters,UBM and T-Norm cohorts. This data has been complementedwith NIST SRE 2005, 2004 and 2002 telephone data,as well as speech from Switchboard I 4 .For experiments presented here microphones (¡ ) and¤ (¡¤) as quoted by NIST [1] have been chosen. Both microphonespresent different characteristics, being ¡ a distantmicrophone ¡¤ and a higher-quality device. Two different testingconditions have been defined. On the one hand, we definea same-microphone condition where trials containing onlya given microphone both in the training and testing speech datahave been selected. For this condition, in the case of ¡ thereare target and ¤¢¤ non-target scores, and in the case of ¡¤there are £ target and ¤ £ non-target scores. Such numbersare quite small, and therefore results obtained for this condition3 Available at http://atvs.ii.uam.es/databases.jsp4 See LDC website http://www.ldc.upenn.edu/should be carefully interpreted. However, we have consideredthem here in order to illustrate database mismatch when sessionvariability is controlled. On the other hand, we define amicrophone-model condition, where trials containing a givenmicrophone in the models and every possible channel in thetest segment have been considered. For this condition, when¡ is selected there are £¢ target and ¢££¤ non-target scores;whereas ¤£¤ target and ¤¢£¢ non-target scores have been generatedfor ¡¤.For database mismatch simulation, on the one hand, thescores for the ¡ microphone-model condition in a databasematching situation will have data from NIST SRE 2005 ¡microphone in the UBM, factor analysis data and T-Norm cohort.On the other hand, in database mismatch conditions datarecorded over NIST SRE 2005 ¡ microphone will be eliminatedfrom the UBM, factor analysis data and T-Norm cohort.The same procedure is applied to ¡¤. It is needed to remark thatthe data from a given microphone represents only a £¢¤ ¤¥ ofthe whole data used for training UBM and factor analysis, andonly £ up to £ models in the whole T-Norm cohort. This isimportant in order to evaluate the impact of the elimination ofsuch a relatively small amount of data from the background set.3.1.1. ResultsTable 1 shows the effect of database mismatch for the samemicrophonecondition described above. Performance is measuredin the form of equal error rate (EER) and Detection CostFunction (DCF) as defined by NIST [1]. Results show thatdatabase mismatch leads to a significant degradation in performance,especially for the DCF performance measure. Howeverthese results should be carefully interpreted, as the number ofscores in each condition is scarce.DB match DB mismatchEER DCF EER DCF¡ ¡ ¤ ¡¤vs. 15.2137 0.0391 17.094 0.0472vs. 1.4815 0.0033 2.2222 0.0143Table 1: Performance of same-microphone condition underdatabase mismatch.Figure 1 illustrates the effects of database mismatch for themicrophone-model condition. This aims to simulate a typicalforensic situation, where speech from the suspect may havebeen recorded in particular microphone conditions for whicha background database is not available. DET curves show asignificant degradation in discrimination performance when themicrophone of the considered model is eliminated from thebackground data.3.2. Experiments with Ahumada IIIIn this section, the use of Ahumada III for compensatingdatabase mismatch in real-casework data is illustrated. The simulationhas been conducted by adding Ahumada III data to theUBM trained with NIST SRE data described in Section 3.1. Althoughmore accurate results can be obtained by using Baeza orBDRA databases from Guardia Civil, we selected a NIST-SREbasedbackground dataset for the illustration of database mismatch,and also for transparency, because any laboratory havingthe NIST SRE databases can test the same results in AhumadaIII. The same ATVS GMM system has been used for performingthe experiments, except for session variability compensa-

False Rejection Probability (in %)4020105210.50.20.1m5 match: EER−DET = 7.28m5 mismatch no m5: EER−DET = 8.82m3 match: EER−DET = 21.06m3 mismatch no m3: EER−DET = 22.600.1 0.2 0.5 1 2 5 10 20 40False Acceptance Probability (in %)Figure 1: DET curves for the microphone-model condition, illustratingthe effect of database mismatch.False Rejection Probability (in %)4020105210.50.20.1Ah3 mismatch: EER−DET = 13.58Ah3 1/3 match: EER−DET = 12.19Ah3 1/2 match: EER−DET = 11.470.1 0.2 0.5 1 2 5 10 20 40False Acceptance Probability (in %)Figure 2: DET curves for experiments using Ahumada III.Database mismatch is reduced by increasingly adding AhumadaIII data to the NIST-SRE-based UBM.tion, which has not considered factor analysis. Moreover, dueto data scarcity in Ah3R1, we will not explore effects in factoranalysis-basedsession variability compensation or T-Norm cohorts,but it will be considered for future work as next releasesof Ahumada III will be available.In order to simulate the effect of using Ahumada III in realcasework, we have used a cross-validation procedure, where werotate subsets of the speakers used for UBM training ( £ or ¢£from the ¢£ speakers considered in the test) and compute scoreswith the rest of available speakers ( £ or £). We call both conditionsthe £ ¢ and £ match respectively. Thus, we obtain anumber of ££ target trials for both conditions, £¡££ non-targettrials for the £ ¢ match condition and ¤¡££ non-target trials forthe £ match condition. In both situations, the proportion ofAhumada III speech data with respect to NIST SRE data in theUBM does not exceed £¤¥ after silence removal.Figure 2 shows the DET plots of the proposed experiments.It is shown that the inclusion of Ahumada III in the UBM increasesdiscrimination accuracy, as expected. Moreover, theincrease is highly significant considering the small percentage(less than £¤¥) of Ahumada III data with respect to NIST SREdata in the UBM.4. ConclusionsThis paper has presented the first release of Ahumada III, aspeech corpus in Spanish from real police investigations. Thedatabase is being built as a tool for aiding forensic laboratoriesto assess and tune their speaker recognition systems in themost possible realistic conditions. In this sense, the importantproblem of database mismatch has been explored using thepresented Ahumada III database and also other multichannelcorpora such as NIST4M, a database constructed from NISTSRE 2006 multichannel data. In this work, database mismatchhas been simulated in UBM modelling, channel compensationbased on factor analysis and T-Norm cohorts. As a result, it hasbeen observed that database mismatch has a critical impact inthe performance of systems, even if the background data matchingthe operational speech conditions is relatively small with respectto the whole dataset used for tuning. It is also shown howdatabases such as Ahumada III help to compensate for this importantproblem. There have been many topics which have notbeen explored in this work, such as the influence of databasemismatch in session variability, its impact in the calibration¡¢andcomputation and a more rigorous study for the different microphonechannels in the NIST4M corpus. However, with AhumadaIII new releases, scientists will be able to deeply exploresuch problems in a realistic forensic framework.5. References[1] M. A. Przybocki, A. F. Martin, and A. N. Le,“NIST speaker recognition evaluations utilizing the Mixercorpora-2004, 2005, 2006,” IEEE Transactions on Audio,Speech and Language Processing, vol. 15, no. 7, pp. 1951–1959, 2007.[2] J. Gonzalez-Rodriguez, Phil Rose, D. Ramos, Doroteo T.Toledano, and J. Ortega-Garcia, “Emulating DNA: Rigorousquantification of evidential weight in transparent andtestable forensic speaker recognition,” IEEE Transactionson Audio, Speech and Language Processing, vol. 15, no. 7,pp. 2072–2084, 2007.[3] Benoît Fauve, Nicholas Evans, and John Mason, “Improvingthe performance of text-independent short durationSVM- and GMM-based speaker verification,” in Proc. ofOdyssey, Stellenbosch, South Africa, 2008.[4] P. Kenny, G. Boulianne, P. Ouellet, and P. Dumouchel,“Speaker and session variability in gmm-based speaker verification,”IEEE Transactions on Audio, Speech and SignalProcessing, vol. 15, no. 4, pp. 1448–1460, 2007.[5] R. Vogt and S. Sridharan, “Explicit modelling of sessionvariability for speaker verification,” Computer Speech andLanguage, vol. 22, no. 1, pp. 17–38, 2007.[6] M. J. Saks and J. J. Koehler, “The coming paradigm shiftin forensic identification science,” Science, vol. 309, no.5736, pp. 892–895, 2005.[7] J. Ortega-Garcia, J. Gonzalez-Rodriguez, and V. Marrero-Aguilar, “AHUMADA: A large speech corpus in Spanishfor speaker characterization and identification,” SpeechCommunication, vol. 31, no. 2-3, 2000.