Proceedings Fonetik 2009 - Institutionen för lingvistik

More documents

Recommendations

Info

Proceedings, FONETIK 2009, Dept. of Linguistics, Stockholm UniversityProject presentation: Spontal – multimodal database ofspontaneous speech in dialogJonas Beskow, Jens Edlund, Kjell Elenius, Kahl Hellmer, David House & Sofia StrömbergssonKTH Speech Music & Hearing, Stockholm, SwedenAbstractWe describe the ongoing Swedish speech databaseproject Spontal: Multimodal database ofspontaneous speech in dialog (VR 2006-7482).The project takes as its point of departure thefact that both vocal signals and gesture involvingthe face and body are important in everyday,face-to-face communicative interaction,and that there is a great need for data withwhich we more precisely measure these.IntroductionSpontal: Multimodal database of spontaneousspeech in dialog is an ongoing Swedish speechdatabase project which began in 2007 and willbe concluded in 2010. It is funded by the SwedishResearch Council, KFI - Grant for largedatabases (VR 2006-7482). The project takes asits point of departure the fact that both vocalsignals and gesture involving the face and bodyare key components in everyday face-to-faceinteraction – arguably the context in whichspeech was borne – and focuses in particular onspontaneous conversation.Although we have a growing understandingof the vocal and visual aspects of conversation,we are lacking in data with which we can makemore precise measurements. There is currentlyvery little data with which we can measure withprecision multimodal aspects such as the timingrelationships between vocal signals and facialand body gestures, but also acoustic propertiesthat are specific to conversation, as opposed toread speech or monologue, such as the acousticsinvolved in floor negotiation, feedback andgrounding, and resolution of misunderstandings.The goal of the Spontal project is to addressthis situation through the creation of a Swedishmultimodal spontaneous speech database richenough to capture important variations amongspeakers and speaking styles to meet the demandsof current research of conversationalspeech.Scope60 hours of dialog consisting of 120 half-hoursessions will be recorded in the project. Eachsession consists of three consecutive 10 minuteblocks. The subjects are all native speakers ofSwedish and balanced (1) for gender, (2) as towhether the interlocutors are of opposing genderand (3) as to whether they know each otheror not. This balance will result in 15 dialogs ofeach configuration: 15x2x2x2 for a total of 120dialogs. Currently (April, 2009), about 33% ofthe database has been recorded. The remainderis scheduled for recording during 2010. All subjectspermit, in writing (1) that the recordingsare used for scientific analysis, (2) that the analysesare published in scientific writings and (3)that the recordings can be replayed in front ofaudiences at scientific conferences and suchlike.In the base configuration, the recordings arecomprised of high-quality audio and highdefinitionvideo, with about 5% of the recordingsalso making use of a motion capture systemusing infra-red cameras and reflectivemarkers for recording facial gestures in 3D. Inaddition, the motion capture system is used onvirtually all recordings to capture body andhead gestures, although resources to treat andannotate this data have yet to be allocated.Instruction and scenariosSubjects are told that they are allowed to talkabout absolutely anything they want at anypoint in the session, including meta-commentson the recording environment and suchlike,with the intention to relieve subjects from feelingforced to behave in any particular manner.The recordings are formally divided intothree 10 minute blocks, although the conversationis allowed to continue seamlessly over theblocks, with the exception that subjects are informed,briefly, about the time after each 10minute block. After 20 minutes, they are alsoasked to open a wooden box which has beenplaced on the floor beneath them prior to therecording. The box contains objects whoseidentity or function is not immediately obvious.The subjects may then hold, examine and190
Proceedings, FONETIK 2009, Dept. of Linguistics, Stockholm UniversityFigure 1. Setup of the recording equipment used to create the Spontal database.discuss the objects taken from the box, but theymay also chose to continue whatever discussionthey were engaged in or talk about somethingentirely different.Technical specificationsThe audio is recorded on four channels using amatched pair of Bruel & Kjaer 4003 omnidirectionalmicrophones for high audio quality,and two Beyerdynamic Opus 54 cardioid headsetmicrophones to enable subject separation fortranscription and dialog analysis. The two omni-directionalBruel & Kjaer microphones areplaced approximately 1 meter from each subject.Two JVC HD Everio GZ-HD7 high definitionvideo cameras are placed to obtain a goodview of each subject from a height that is approximatelythe same as the heads of both ofthe participating subjects. They are placedabout 1.5 meters behind the subjects to minimizeinterference. The cameras record in mpeg-2encoded full HD with the resolution1920x1080i and a bitrate of 26.6 Mbps. To ensureaudio, video and motion-capture synchronizationduring post processing, a record playeris included in the setup. The turntable is placedbetween the subjects and a bit to the side, in fullview of the motion capture cameras. The markerthat is placed near the edge on the platter rotateswith a constant speed (33 rpm) andenables high-accuracy synchronization of theframe rate in post processing. The recordingsetup is illustrated in Figure 1.Figure 2 shows a frame from each of thetwo video cameras aligned next to each other,so that the two dialog partners are both visible.The opposing video camera can be seen in thecentre of the image, and a number of tripodsholding the motion capture cameras are visible.The synchronization turn-table is visible in theleft part of the left pane and the right part of theright pane. The table between the subjects iscovered in textiles, a necessary precaution asthe motion capture system is sensitive to reflectingsurfaces. For the same reason, subjectsare asked to remove any jewelry, and other shinyobjects are masked with masking tape.Figure 3 shows a single frame from the videorecording and the corresponding motioncapturedata from a Spontal dialog. As in Figure2, we see the reflective markers for the motion-capturesystem on the hands, arms, shoulders,trunk and head of the subject. Figure 4 is a3D data plot of the motion capture data fromthe same frame, with connecting lines betweenthe markers on the subject’s body.191
Page 1 and 2:
Department of LinguisticsProceeding
Page 3 and 4:
Proceedings, FONETIK 2009, Dept. of
Page 5 and 6:
Page 7 and 8:
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17 and 18:
Page 19 and 20:
Page 21 and 22:
Page 23 and 24:
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
Page 37 and 38:
Page 39 and 40:
Page 41 and 42:
Proceedings, FOETIK 2009, Dept. of
Page 43 and 44:
Page 45 and 46:
Page 47 and 48:
Page 49 and 50:
Page 51 and 52:
Page 53 and 54:
Page 55 and 56:
Page 57 and 58:
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
Page 127 and 128:
Page 129 and 130:
Page 131 and 132:
Page 133 and 134:
Page 135 and 136:
Page 137 and 138:
Page 139 and 140: Proceedings, FONETIK 2009, Dept. of
Page 189: Proceedings, FONETIK 2009, Dept. of
Page 203 and 204: Proceedings, FOETIK 2009, Dept. of
Page 205 and 206: Proceedings, FOETIK 2009, Dept. of
Page 227: Department of LinguisticsPhonetics
show all

Proceedings Fonetik 2009 - Institutionen för lingvistik

Create successful ePaper yourself

Delete template?

Save as template?