Proceedings Fonetik 2009 - Institutionen för lingvistik

More documents

Recommendations

Info

Proceedings, FONETIK 2009, Dept. of Linguistics, Stockholm UniversityReal vs. rule-generated tongue movements as an audiovisualspeech perception supportOlov Engwall and Preben WikCentre for Speech Technology, CSC, KTHengwall@kth.se, preben@kth.seAbstractWe have conducted two studies in which animationscreated from real tongue movementsand rule-based synthesis are compared. Wefirst studied if the two types of animations weredifferent in terms of how much support theygive in a perception task. Subjects achieved asignificantly higher word recognition rate insentences when animations were shown comparedto the audio only condition, and a significantlyhigher score with real movementsthan with synthesized. We then performed aclassification test, in which subjects should indicateif the animations were created frommeasurements or from rules. The results showthat the subjects as a group are unable to tell ifthe tongue movements are real or not. Thestronger support from real movements henceappears to be due to subconscious factors.IntroductionSpeech reading, i.e. the use of visual cues in thespeaker’s face, in particular regarding the shapeof the lips (and hence the often used alternativeterm lip reading), can be a very importantsource of information if the acoustic signal isinsufficient, due to noise (Sumby & Pollack,1954; Benoît & LeGoff, 1998) or a hearingimpairment(e.g., Agelfors et al., 1998; Siciliano,2003). This is true even if the face is computeranimated. Speech reading is much morethan lip reading, since information is also givenby e.g., the position of the jaw, the cheeks andthe eye-brows. For some phonemes, the tip ofthe tongue is visible through the mouth openingand this may also give some support. However,for most phonemes, the relevant parts of thetongue are hidden, and “tongue reading” istherefore impossible in human-human communication.On the other hand, with a computeranimatedtalking face it is possible to maketongue movements visible, by removing partsin the model that hide the tongue in a normalview, thus creating an augmented reality (AR)display, as exemplified in Fig. 1.Since the AR view of the tongue is unfamiliar,it is far from certain that listeners are ableto make use of the additional information in asimilar manner as for animations of the lips.Badin et al. (2008) indeed concluded that thetongue reading abilities are weak and that subjectsget more support from a normal view ofthe face, where the skin of the cheek is showninstead of the tongue, even though less informationis given. Wik & Engwall (2008) similarlyfound that subjects in general found littleadditional support when an AR side-view as theone in Fig. 1 was added to a normal front view.There is nevertheless evidence that tonguereading is possible and can be learned explicitlyor implicitly. When the signal-to-noise ratiowas very low or the audio muted in the studyby Badin et al. (2008), subjects did start tomake use of information given by the tonguemovements – if they had previously learnedhow to do it. The subjects were presented VCVwords in noise, with either decreasing or increasingsignal-to-noise ratio (SNR). The groupwith decreasing SNR was better in low SNRconditions when tongue movements were displayed,since they had been implicitly trainedon the audiovisual relationship for stimuli withhigher SNR. The subjects in Wik & Engwall(2008) started the word recognition test in sentenceswith acoustically degraded audio by afamiliarization phase, where they could listento, and look at, training stimuli with both nor-Figure 1. Augmented reality view of the face.30
Proceedings, FONETIK 2009, Dept. of Linguistics, Stockholm Universitymal and degraded audio. Even though the totalresults were no better with the AR view thanwith a normal face, the score for some sentenceswas higher when the tongue was visible.Grauwinkel et al. (2007) also showed that subjectswho had received explicit training, in theform of a video that explained the intra-oral articulatormovement for different consonants,performed better in the VCV recognition taskin noise than the group who had not receivedthe training and the one who saw a normal face.An additional factor that may add to the unfamiliarityof the tongue movements is thatthey were generated with a rule-based visualspeech synthesizer in Wik & Engwall (2008)and Graunwinkel et al. (2007). Badin etal. (2008) on the other hand created the animationsbased on real movements, measured withElectromagnetic Articulography (EMA). In thisstudy, we investigate if the use of real movementsinstead of rule-generated ones has anyeffect on speech perception results.It could be the case that rule-generatedmovements give a better support for speechperception, since they are more exaggeratedand display less variability. It could howeveralso be the case that real movements give a bettersupport, because they may be closer to thelisteners’ conscious or subconscious notion ofwhat the tongue looks like for different phonemes.Such an effect could e.g., be explainedby the direct realist theory of speech perception(Fowler, 2008) that states that articulatory gesturesare the units of speech perception, whichmeans that perception may benefit from seeingthe gestures. The theory is different from, butclosely related to, and often confused with, thespeech motor theory (Liberman et al, 1967;Liberman & Mattingly, 1985), which stipulatesthat speech is perceived in terms of gesturesthat translate to phomenes by a decoder linkedto the listener's own speech production. It hasoften been criticized (e.g., Traunmüller, 2007)because of its inability to fully explain acousticspeech perception. For visual speech perception,there is on the other hand evidence (Skipperet al., 2007) that motor planning is indeedactivated when seeing visual speech gestures.Speech motor areas in the listener’s brain areactivated when seeing visemes, and the activitycorresponds to the areas activated in thespeaker when producing the same phonemes.We here investigate audiovisual processing ofthe more unfamiliar visual gestures of thetongue, using a speech perception and a classificationtest. The perception test analyzes thesupport given by audiovisual displays of thetongue, when they are generated based on realmeasurements (AVR) or synthesized by rules(AVS). The classification test investigates ifsubjects are aware of the differences betweenthe two types of animations and if there is anyrelation between scores in the perception testand the classification test.ExperimentsBoth the perception test (PT) and the classificationtest (CT) were carried out on a computerwith a graphical user interface consisting of oneframe showing the animations of the speechgestures and one response frame in which thesubjects gave their answers. The acoustic signalwas presented over headphones.The Augmented Reality displayBoth tests used the augmented reality side-viewof a talking head shown in Fig. 1. Movementsof the three-dimensional tongue and jaw havebeen made visible by making the skin at thecheek transparent and representing the palateby the midsagittal outline and the upper incisor.Speech movements are created in the talkinghead model using articulatory parameters, suchas jaw opening, shift and thrust; lip rounding;upper lip raise and retraction; lower lip depressionand retraction; tongue dorsum raise, bodyraise, tip raise, tip advance and width. Thetongue model is based on a component analysisof data from Magnetic Resonance Imaging(MRI) of a Swedish subject producing staticvowels and consonants (Engwall, 2003).Creating tongue movementsThe animations based on real tongue movements(AVR) were created directly from simultaneousand spatially aligned measurements ofthe face and the tongue for a female speaker ofSwedish (Beskow et al., 2003). The MovetrackEMA system (Branderud, 1985) was employedto measure the intraoral movements, usingthree coils placed on the tongue, one on the jawand one on the upper incisor. The movementsof the face were measured with the Qualisysmotion capture system, using 28 reflectors attachedto the lower part of the speaker’s face.The animations were created by adjusting theparameter values of the talking head to optimallyfit the Qualisys-Movetrack data (Beskowet al., 2003).31
Page 1 and 2: Department of LinguisticsProceeding
Page 3 and 4: Proceedings, FONETIK 2009, Dept. of
Page 29: Proceedings, FONETIK 2009, Dept. of
Page 41 and 42: Proceedings, FOETIK 2009, Dept. of
Page 81 and 82:
Proceedings, FONETIK 2009, Dept. of
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
Page 127 and 128:
Proceedings, FOETIK 2009, Dept. of
Page 129 and 130:
Page 131 and 132:
Page 133 and 134:
Page 135 and 136:
Page 137 and 138:
Page 139 and 140:
Page 141 and 142:
Page 143 and 144:
Page 145 and 146:
Page 147 and 148:
Page 149 and 150:
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185 and 186:
Page 187 and 188:
Page 189 and 190:
Page 191 and 192:
Page 193 and 194:
Page 195 and 196:
Page 197 and 198:
Page 199 and 200:
Page 201 and 202:
Page 203 and 204:
Page 205 and 206:
Page 207 and 208:
Page 209 and 210:
Page 211 and 212:
Page 213 and 214:
Page 215 and 216:
Page 217 and 218:
Page 219 and 220:
Page 221 and 222:
Page 223 and 224:
Page 225 and 226:
Page 227:
Department of LinguisticsPhonetics
show all

Proceedings Fonetik 2009 - Institutionen för lingvistik

Create successful ePaper yourself

Delete template?

Save as template?