Proceedings Fonetik 2009 - Institutionen för lingvistik

More documents

Recommendations

Info

Proceedings, FOETIK 2009, Dept. of Linguistics, Stockholm UniversityCross - modal Clustering in the Acoustic - ArticulatorySpaceG. Ananthakrishnan and Daniel M. eibergCentre for Speech Technology, CSC, KTH, Stockholmagopal@kth.se, neiberg@speech.kth.seAbstractThis paper explores cross-modal clustering inthe acoustic-articulatory space. A method toimprove clustering using information frommore than one modality is presented. Formantsand the Electromagnetic Articulography measurementsare used to study corresponding clustersformed in the two modalities. A measurefor estimating the uncertainty in correspondencesbetween one cluster in the acousticspace and several clusters in the articulatoryspace is suggested.IntroductionTrying to estimate the articulatory measurementsfrom acoustic data has been of specialinterest for long time and is known as acousticto-articulatoryinversion. Though this mappingbetween the two modalities expected to be aone-to-one mapping, early research presentedsome interesting evidence showing nonuniqueness,in this mapping. Bite-block experimentshave shown that speakers are capableof producing sounds perceptually close to theintended sounds even though the jaw is fixed inan unnatural position (Gay et al., 1981). Mermelstein(1967) and Schroeder (1967) haveshown, through analytical articulatory models,that the inversion is unique to a class of areafunctions rather than a unique configuration ofthe vocal tract.With the advent of measuring techniqueslike Electromagnetic Articulography (EMA)and X-Ray Microbeam, it was possible to collectsimultaneous measurements of acousticsand articulation during continuous speech. Severalattempts have been made by researchers toperform acoustic-to-articulatory inversion byapplying machine learning techniques to theacoustic-articulatory data (Yehia et al., 1998and Kjellström and Engwall, 2009). The statisticalmethods applied to the problem of mappingbrought a new dimension to the concept ofnon-uniqueness in the mapping. In the deterministiccase, one can say that if the same acousticparameters are produced by more than one articulatoryconfiguration, then the particularmapping is considered to be non-unique. It isalmost impossible to show this using real recordeddata, unless more than one articulatoryconfiguration produces exactly the same acousticparameters. However, not finding such instancesdoes not imply that non-uniquenessdoes not exist.Qin and Carreira-Perpinán (2007) proposedthat the mapping is non-unique if, for a particularacoustic cluster, the corresponding articulatorymapping may be found in more than onecluster. Evidence of non-uniqueness in certainacoustic clusters for phonemes like /ɹ/, /l/ and/w/ was presented. The study by Qin quantizedthe acoustic space using the perceptual Itakuradistance on LPC features. The articulatoryspace was clustered using a nonparametricGaussian density kernel with a fixed variance.The problem with such a definition of nonuniquenessis that one does not know what isthe optimal method and level of quantizationfor clustering the acoustic and articulatoryspaces.A later study by Neiberg et. al. (2008) arguedthat the different articulatory clustersshould not only map onto a single acoustic clusterbut should also map onto acoustic distributionswith the same parameters, for it to becalled non-unique. Using an approach based onfinding the Bhattacharya distance between thedistributions of the inverse mapping, they foundthat phonemes like /p/, /t/, /k/, /s/ and /z/ arehighly non-unique.In this study, we wish to observe how clustersin the acoustic space map onto the articulatoryspace. For every cluster in the acousticspace, we intend to find the uncertainty in findinga corresponding articulatory cluster. It mustbe noted that this uncertainty is not necessarilythe non-uniqueness in the acoustic-toarticulatorymapping. However, finding this uncertaintywould give an intuitive understanding202
Proceedings, FOETIK 2009, Dept. of Linguistics, Stockholm Universityabout the difficulties in the mapping for differentphonemes.Clustering the acoustic and articulatoryspaces separately, as was done in previous studiesby Qin and Carreira-Perpinán (2007) as wellas Neiberg et al. (2008), leads to hard boundariesin the clusters. The cluster labels for theinstances near these boundaries may estimatedincorrectly, which may cause an over estimationof the uncertainty. This situation is explainedby Fig. 1 using synthetic data where wecan see both the distributions of the syntheticdata as well as the Maximum A-posterioriProbability (MAP) Estimates for the clusters.We can see that, because of the incorrect clustering,it seems as if data belonging to one clusterin mode A belongs to more than one clusterin mode B.In order to mitigate this problem, we havesuggested a method of cross-modal clusteringwhere both the available modalities are madeuse of by allowing soft boundaries for the clustersin each modality. Cross-modal clusteringhas been dealt with in detail under several contextsof combining multi-modal data. Coen(2005) proposed a self supervised methodwhere he used acoustic and visual features tolearn perceptual structures based on temporalcorrelations between the two modalities. Heused the concept of slices, which are topologicalmanifolds encoding dynamic states. Similarly,Belolli et al.(2007) proposed a clusteringalgorithm using Support Vector Machines(SVMs) for clustering inter-related text datasets.The method proposed in this paper does notmake use of correlations, but mainly uses coclusteringproperties between the two modalitiesin order to perform the cross-modal clustering.Thus, even non-linear dependencies (uncorrelated)may also be modeled using thissimple method.TheoryWe assume that the data is a Gaussian MixtureModel (GMM). The acoustic space Y = {y 1 ,y 2 …y } with ‘’ data points is modelled using‘I’ Gaussians, {λ 1 , λ 2 ,…λ I } and the articulatoryspace X = {x 1 , x 2 …x } is modelled using ‘K’Gaussians, {γ 1 ,γ 2 ,….γ K }. ‘I’ and ‘K’ areobtained by minimizing the BayesianInformation Criterion (BIC). If we know whicharticulatory Gaussian a particular data pointbelongs to, say, γ k The correct acousticGaussian ‘λ n ’ for the the ‘n th ’ data point havingacoustic features ‘y n ’ and articulatory features‘x n ’ is given by the maximum cross-modal a-posteriori probabilityλ = arg max P( λ | x , y , γ )n i n n k1≤i ≤I= arg max p( xn, yn| λi , γ k )*P( λi | γ k )*P( γ k )1≤i≤I(1)The knowledge about the articulatory clustercan then be used to improve the estimate of theFigure 1. The figures above show a synthesizedexample of data in two modalities. The figuresbelow show how MAP hard clustering may bringabout an effect of uncertainty in the correspondencebetween clusters in the two modalities.correct acoustic cluster and vice versa as shownbelowγ = arg max p( x , y | λ , γ )*P( γ | λ )*P( λ )n n n i k k i i1≤k ≤K(2)Where P(λ|γ) is the cross-modal prior and thep(x,y|λ,γ) is the joint cross-modal distribution.If the first estimates of the correct clusters areMAP, then the estimates of the correct clustersof the speech segments are improvedFigure 2. The figures shows an improved86420286420246 4 2 0 2 4 6 8864202Mode AMode A46 4 2 0 2 4 6 8Mode A46 4 2 0 2 4 6 824 2 0 2 4 6 8 10 12performance and soft boundaries for the syntheticdata using cross-modal clustring, here the effect ofuncertainty in correspondences is less.121086420121024 2 0 2 4 6 8 10 1286420121086420Mode BMode BMode B24 2 0 2 4 6 8 10 12203
Page 1 and 2:
Department of LinguisticsProceeding
Page 3 and 4:
Proceedings, FONETIK 2009, Dept. of
Page 5 and 6:
Page 7 and 8:
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17 and 18:
Page 19 and 20:
Page 21 and 22:
Page 23 and 24:
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
Page 37 and 38:
Page 39 and 40:
Page 41 and 42:
Proceedings, FOETIK 2009, Dept. of
Page 43 and 44:
Page 45 and 46:
Page 47 and 48:
Page 49 and 50:
Page 51 and 52:
Page 53 and 54:
Page 55 and 56:
Page 57 and 58:
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
Page 127 and 128:
Page 129 and 130:
Page 131 and 132:
Page 133 and 134:
Page 135 and 136:
Page 137 and 138:
Page 139 and 140:
Page 141 and 142:
Page 143 and 144:
Page 145 and 146:
Page 147 and 148:
Page 149 and 150:
Page 151 and 152: Proceedings, FONETIK 2009, Dept. of
Page 201: Proceedings, FONETIK 2009, Dept. of
Page 205 and 206: Proceedings, FOETIK 2009, Dept. of
Page 227: Department of LinguisticsPhonetics
show all

Proceedings Fonetik 2009 - Institutionen för lingvistik

Create successful ePaper yourself

Delete template?

Save as template?