Dynamic Time Warping (DTW) for Single Word and Sentence ...

DTW for Single Word and Sentence Recognizers
Dynamic Time Warping (DTW)
for Single Word and Sentence
Recognizers
Reference: Huang et al. Ch. 8.2.1, Waibel/Lee, Ch. 4
(some papers online at http://csl.ira.uka.de > MMMK)

DTW for Single Word and Sentence Recognizers
Simplified Decoding
Speech
Feature
extraction
Speech
features
2
Trained
Phoneme
classifiers
Decision
/h/
...
Hypotheses
(phonemes)
/h/ /e/ /l/ /o/
/w/ /o/ /r/ /l/ /d/

DTW for Single Word and Sentence Recognizers
Simplified Classifier Training
Aligned
Speech
Feature
extraction
Speech
features
/h/ /e/ /l/ /o/ /h/ /e/ /l/ /o/
3
Train
Classifier
/e/
Improved
Classifiers
Train Classifier
Use aligned speech
vectors (e.g. all frames
of phoneme /e/) to train
the reference vectors of
/e/ (=Codebook):
- kmeans
- LVQ

DTW for Single Word and Sentence Recognizers
Classification
Xuedong Huang, Alex Acero, Hsiao-Wuen Hon:
Spoken Language Processing, Chapter 4
&
R. Duda and P. Hart: Pattern Classification and
Scene Analysis. John Wiley & Sons, 2000 (2 nd edition)

DTW for Single Word and Sentence Recognizers
Pattern Recognition Review
• Static Patterns, no dependence on Time or Sequential Order
• Approaches
– Knowledge-based approaches:
1. Compile knowledge
2. Build decision trees
– Connectionist approaches:
1. Automatic knowledge acquisition, "black-box" behavior
2. Simulation of biological processes
– Statistical approaches:
1. Build a statistical model of the "real world"
2. Compute probabilities according to the models
5

DTW for Single Word and Sentence Recognizers
Pattern Recognition
statistical
6
Pattern
Recognition
supervised unsupervised
parametric nonparametric
linear nonlinear
knowledge /
connectionist

DTW for Single Word and Sentence Recognizers
Pattern Recognition
• Important Notions
– Supervised - Unsupervised Classifiers
– Parametric - Non-Parametric Classifiers
– Linear - Non-linear Classifiers
• Classical Statistical Methods
– Bayes Classifier
– K-Nearest Neighbor
• Connectionist Methods
– Perceptron
– Multilayer Perceptrons
7

DTW for Single Word and Sentence Recognizers
Supervised - Unsupervised
• Supervised training:
Class to be recognized is known
for each sample in training data.
Requires a priori knowledge of
useful features and knowledge
/labeling of each training token
(cost!).
• Unsupervised training:
Class is not known and structure
is to be discovered automatically.
Feature-space-reduction
example: clustering, auto-associative nets
8

DTW for Single Word and Sentence Recognizers
Unsupervised Classification
F2
+ +
+
+ + +
• Classification:
+
+ +
+
+
+
+
+ + +
+ + +
+ +
+ +
+ + + +
+
+
+
+
+ + + + +
+ + + +
– Classes Not Known: Find Structure
9
+
F1

DTW for Single Word and Sentence Recognizers
Unsupervised Classification
F2
+ +
+
+ + +
• Classification:
– Classes Not Known: Find Structure
– Clustering
+
+ +
+
+
+
+
+ + +
+ + +
+ +
+ +
+ + + +
+
+
+
+
+ + + + +
+ + + +
– How to cluster? How many clusters?
10
+
F1

DTW for Single Word and Sentence Recognizers
Supervised Classification
Income
+ +
+
+ +
• Classification:
– Classes Known: Creditworthiness: Yes-No
– Features: Income, Age
– Classifiers
+ credit-worthy
+
+ +
+
+
+
+
+ + +
+ + +
+ +
+ +
+ + + +
+
+
+
+
+ + + + +
+ + + +
11
+
non-credit-worthy
Age

DTW for Single Word and Sentence Recognizers
Classifier Design in Practice
• Need a priori probability P ( ) i (not too bad)
• Need class conditional PDF p
• Problems:
– limited training data
– limited computation
– class-labeling potentially costly and prone to error
– classes may not be known
– good features not known
• Parametric Solution:
– Assume that p (x / )
has a particular parametric form
i
– Most common representative: multivariate normal density
12
(x / i )

DTW for Single Word and Sentence Recognizers
Mixtures of Gaussian Densities
Often the shape of the set of vectors that belong to one class does not
look like what can be modeled by a single Gaussian.
A (weighted) sum of Gaussians can approximate many more densities:
In general, a class can be modeled as a mixture of Gaussians:
13

DTW for Single Word and Sentence Recognizers
Gaussian Densities
• The most often used model for (preprocessed) speech signals are
Gaussian densities.
• Often the "size" of the parameter spaces is measured in "number of
densities“
• A multivariate Gaussian density looks like this:
Its parameters are:
• The mean vector µ (a vector with d coefficients)
• The covariance matrix (a symmetric dxd matrix),
if X indep. is diagonal
• The determinant of the covariance matrix| |
14

DTW for Single Word and Sentence Recognizers
Codebooks of Reference Vectors
Definition: A collection of reference vectors is called a codebook.
Ways how to find (good) codebooks:
• A) Simple but suboptimal: use centers of equally sized hypercubes
• B) Possibly independent from data: build tree structure of feature space
• C) Run the k-means (or LBG) algorithm
• D) Use the "learning vector quantization" (LVQ) algorithm
15

DTW for Single Word and Sentence Recognizers
Finding Codebooks of Reference Vectors
Tree-Structured Feature Space
1. initialize one node for all data
2. compute the direction of the largest deviation for all nodes
3. split the node with the largest deviation in two (hyperplane through mean)
4. if not yet satisfied then goto step 2
Quantization:
starting at root node, compare vector x with current node's hyperplane,
descend to node that corresponds to the side of the hyperplane on which x lies
16

DTW for Single Word and Sentence Recognizers
Finding Codebooks of Reference Vectors
The k-Means Algorithm
1. Step 1 Initialize: given value of k, sample vectors v 1 , ... v T
Intialize any k means (e.g. µ i = v i )
2. Step 2 Nearest-Neighbor classification: Assign every vector v i
to its class' centroid µ f(i)
3. Step 3 Codebook update: Replace every mean µ i by the mean of
all sample vectors that have been assigned to it
4. Step 4 Iteration: If not satisfied, yet, then go to step 2
possible stop-criterion:
• A fixed number of iterations
• The average (maximum) distance |v i - µ f(i) | is below a fixed value
• The derivative of the distance is below a fixed value (nothing happens any
more)
• Theoretically k-means can only converge to a local optimum
• Initialization is often critical - repeat k-means for several codebook sets
• OR: Linde-Buzo-Gray (LBG) Algorithm, I.e. start with a 1-vector codebook and use
splitting algorithm to obtain 2-vector, …, M-vector codebook
17

DTW for Single Word and Sentence Recognizers
Problems of Classifier Design
• Features:
– What and how many features should be selected?
– Any features?
– The more the better?
– If additional features not useful (same mean and covariance),
classifier will automatically ignore them?
18

DTW for Single Word and Sentence Recognizers
Curse of Dimensionality
• Adding more features
– Adding independent features may help
– BUT: adding indiscriminant features may lead to worse
performance!
• Reason:
– Training Data vs. Number of Parameters
– Limited training data.
• Solution:
– select features carefully
– reduce dimensionality
– Principle Component Analysis
19

DTW for Single Word and Sentence Recognizers
Trainability
• Two-phoneme classification example (Huang et al.),
Phonemes modeled by Gaussian mixtures
• Parameters are trained with a varied set of training
samples
20

DTW for Single Word and Sentence Recognizers
Simplified Decoding and Training
Speech
Aligned
Speech
Feature
extraction
Feature
extraction
Speech
features
Speech
features
/h/ /e/ /l/ /o/ /h/ /e/ /l/ /o/
21
Decision
(apply trained
classifiers)
/h/
...
Train
Classifier
/e/
Hypotheses
(phonemes)
/h/ /e/ /l/ /o/
/w/ /o/ /r/ /l/ /d/
Improved
Classifiers
Train codebook
- kmeans
- LVQ

DTW for Single Word and Sentence Recognizers
Where do we go from here?
Dynamic Programming and Single Word Recognition
• Comparing Complete Utterances
• Problems: Endpoint Detection and Speech Detection
• Time Warping
• The Minimal Editing Distance Problem
• Dynamic Programming
• Dynamic Time Warping (DTW)
• Constraints for the DTW-Path
• The DTW Search Space
• DTW with Beam Search
• The Principles of Building Speech Recognizers
• Isolated Word Recognition with Template Matching
22

DTW for Single Word and Sentence Recognizers
Compare Complete Utterances
What we had so far:
• Record a sound signal
• Compute frequency representation
• Quantize/classify vectors
We now have:
• A sequence of pattern vectors
What we want:
• The similiarity between two such sequences
Obviously: The order of vectors is important!
vs.
23
=>

DTW for Single Word and Sentence Recognizers
Comparing Complete Utterances
Comparing speech vector sequences has to overcome three problems:
1) Speaking rate characterizes speakers (speaker dependent!)
if the speaker is speaking faster, we get fewer vectors
2) Changing speaking rate by purpose: e.g. talking to a foreign person
3) Changing speaking rate non-purposely: speaking disfluencies
vs.
So we have to find a way to decide which vectors to compare to another
• Impose some constraints (compare every vector to all others is too costly)
24

DTW for Single Word and Sentence Recognizers
Endpoint Detection
When comparing two recorded utterances we face two problems:
1) When does the speech begin?
• We might not have any mechanism to signal the recognizer
when it should listen.
2) Varying length of utterance
• Utterances might be of different length (speaking rate, …)
• One or both utterances can be preceeded or followed by a period
of (possibly non-voluntarily recorded) silence
vs.
25

DTW for Single Word and Sentence Recognizers
Speech Detection
Solution to Problem (1) When does speech begin?
A: Push-to-talk scenario: only listen when user pushes button
B: Always on scenario: always listen, only consider speech regions
Select Speech Regions:
Use SNR: works well if SNR > 30dB, otherwise problematic
Compute signal power:
p[i..j] = k=i..j s[k]2 , then apply a threshold t to detect speech
t
26

DTW for Single Word and Sentence Recognizers
Approaches to Alignment of Vector Sequences
First idea to overcome the varying length
of Utterances
Problem (2):
1. Normalize sequence length
2. Make a linear alignment
between the two sequences
Linear alignment can handle the problem of different speaking rates
But …..
What about varying speaking rates?
27

DTW for Single Word and Sentence Recognizers
Approaches to Alignment of Vector Sequences
• Linear alignment can handle the problem of different speaking rates
• But: it can not handle the problem of varying speaking rates during
the same utterance.
28

DTW for Single Word and Sentence Recognizers
Alignment of Speech Vectors May Be Non-Bijective
Given: two sequences x 1 ,x 2 ,...,x n and y 1 ,y 2 ,...,y m
Wanted: alignment relation R (not function), where (i,j) is in R
iff x i is aligned with y j .
y
x
1 2 3 4 5 6 7 8 9 ...
1 2 3 4 5 6 7 8 9 ...
It is possible that …
… more than one x is aligned to the same y (e.g. x 3, x 4)
… more than one y is aligned to the same x (e.g. y 8, y 9)
… more than an x or a y has no alignment partner at all (e.g. y 6)
29

DTW for Single Word and Sentence Recognizers
y
Solution: Time Warping
x
Given:
two sequences x 1 ,x 2 ,...,x n and y 1 ,y 2 ,...,y m
Wanted:
alignment relation R, where (i,j) is in R
iff x i is aligned with y j ., i.e. are looking for
a common time-axis:
30
y
y 8,y 9 align to
same x
y 6 has no partner
x 3,x 4 align to the same y
x

DTW for Single Word and Sentence Recognizers
What is the best alignment relation R?
Distance Measure between two utterances:
For a given path R(i,j), the
distance between x and y is
the sum of all
local distances d(x i ,y j )
In our example:
d(x 1 ,y 1 ) + d(x 2 ,y 2 ) + d(x 3 ,y 3 )
+ d(x 4 ,y 3 ) + d(x 5 ,y 4 ) +
d(x 6 ,y 5 ) + d(x 7 ,y 7 ) + ...
Question:
How can we find a path that
gives the minimal overall
distance?
31

DTW for Single Word and Sentence Recognizers
Utterance Comparison by Dynamic Time Warping
How can we apply the DP algorithm for the minimal editing distance to the
utterance comparison problem?
Differences and Questions:
• What do editing steps correspond to?
• We "never" really get two identical vectors.
• We are dealing with continuous and not discrete signals here
Answers:
• We can delete/insert/substitute vectors. Define cost for del/ins, define
cost for substituting = distance between vectors
• No two vectors are the same? So what.
• Continuous signals => we get continuous distances (no big deal)
The DTW-Algorithm:
• Works like the minimal editing distance algorithm
• Minor modification: Allow different kinds of steps (different
predecessors of a state)
• Use vector-vector distance measure as cost function
32

DTW for Single Word and Sentence Recognizers
Dynamic Programming
How can we find the minimal editing distance?
Greedy algorithm?
• Always perform the step that is currently the cheapest.
• If there are more than one cheapest step …
• … take any one of them.
Obvious: can't guarantee to lead to the optimal solution.
Solution: Dynamic Programming (DP)
DP is frequently used in operations research, where
consecutive decisions depend on each other and whose
sequence must lead to optimal results.
33

DTW for Single Word and Sentence Recognizers
Key Idea of Dynamic Programming
The key idea of DP is:
If we would like to take our system into a state s i , and
• we know the costs c 1 ,...,c k for the optimal ways to get
• from the start to all states q 1 ,...,q k from which we can go to s i ,
• then the optimal way to s i goes over the
state q l where l = argmin j c j
34

DTW for Single Word and Sentence Recognizers
The Dynamic Programming Matrix (1)
To find the minimal editing distance from
x 1 ,x 2 ,...,x n to y 1 ,y 2 ,...,y m
we can define an algorithm inductively:
Let C(i, j) denote the
minimal editing distance from x1 ,x2 ,...,xi to y1 ,y2 ,...,yj .
Then we get:
• C(0,0) = 0 (no characters no editing)
• C(i, j) is either (whichever is smallest):
• C(i-1, j-1) plus the cost for
replacing xi with yj • or C(i-1, j) plus the cost for deleting xi • or C(i, j-1) plus the cost for inserting yj 35

DTW for Single Word and Sentence Recognizers
The Dynamic Programming Matrix (2)
Usually for the minimal editing distance:
• The cost for deleting or inserting a character is 1
• The cost for replacing x i with y j is 0 (if x i = y j ) or 1 (else)
• Might be useful to define other costs for special purposes
Eventually:
• Remember for each state (i-1, j-1) which one was
the best predecessor (backpointer)
• Find the sequence of editing steps by
backtracing the predecessor pointers
from the final state
36

DTW for Single Word and Sentence Recognizers
The Minimal Editing Distance Problem
Given: two character sequences (words) x 1 ,x 2 ,...,x n and y 1 ,y 2 ,...,y m
Wanted: the minimal number (and sequence) of editing steps that are needed to
convert x to y
The editing cursor starts at x 0 , an editing step can be one of:
• Delete the character x i under the cursor
• Insert a character x i at the cursor position
• Replace character x i at the cursor position with y j
• Moving the cursor to the next character (no editing), we can't go back
Example: Convert x = “BRAKES" to y = “BAKERY":
One possible solution:
• B = B, move curser to next character
• Delete character x 2 = R
• A = A, move curser to next character, K = K, move, E = E, move
• Replace character x 5 = S with character y 5 = R
• Insert character y 6 = Y (sequence not necessarily unique)
Often referred to as Levinshtein distance, keep in mind, we will revisit this when we
talk about how to measure the Performance (Word Accuracy) of a recognizer
37

DTW for Single Word and Sentence Recognizers
Levinshtein
Y
R
E
K
A
B
B R A K E S
B=B, move to next
38

DTW for Single Word and Sentence Recognizers
Levinshtein
Y
R
E
K
A
B
B R A K E S
Delete character x2=R 39

DTW for Single Word and Sentence Recognizers
Levinshtein
Y
R
E
K
A
B
B R A K E S
A=A, move to next
40

DTW for Single Word and Sentence Recognizers
Levinshtein
Y
R
E
K
A
B
B R A K E S
41
K=K, move to next

DTW for Single Word and Sentence Recognizers
Levinshtein
Y
R
E
K
A
B
B R A K E S
42
E=E, move to next

DTW for Single Word and Sentence Recognizers
Levinshtein
Y
R
E
K
A
B
B R A K E S
43
replace character x 6 = S
with character y 5 = R

DTW for Single Word and Sentence Recognizers
Y
R
E
K
A
B
Levinshtein
B R A K E S
44
insert character y 6 = Y

DTW for Single Word and Sentence Recognizers
Levinshtein
Sequence is not necessarily unique!
Y
R
E
K
A
B
B R A K E S
insert character y 5 = R
45
replace character x 6 = S with y 6 = Y

DTW for Single Word and Sentence Recognizers
What we could do already
We can build a first simple isolated-word recognizer using DTW
We can build a preprocessor such that recorded speech can be processed
by the recognizer
We can recognize speech using DTW and print the score for each of its
reference patterns:
Example:
• Build recognizer that can recognize two words w 1 and w 2
• Collect training examples (in real life: a lot of data)
• Skip the optimization phase (don't need development set)
• Collect evaluation data (a few examples per word)
• Run tests on evaluation data and report results
46

DTW for Single Word and Sentence Recognizers
Compare Complete Utterances
=>
47
Ref Word 2
Ref Word 1
Hypothese

DTW for Single Word and Sentence Recognizers
Isolated Word Recognition with Template Matching
• For each word in the vocabulary, store at least one reference pattern
• When multiple reference patterns are available, either use all of
them or compute an average
• During recognition
• Record a spoken word
• Perform pattern matching with all stored patterns (or at least
with those that can be used in the current context)
• Compute a DTW score for every vocabulary word (when using
multiple references, compute one score out of many, e.g. average or
max)
• Recognize the word with the best DTW score
• This approach works only for very small vocabularies and/or for
speaker-dependent recognition
48

DTW for Single Word and Sentence Recognizers
What we can‘t do yet:
OR Problems with Pattern Matching
• Need endpoint detection
• Need collection of reference patterns (inconvenient for user)
• Works only well for speaker-dependent recognition
(difficult to cover variations)
• High computational effort (esp. for large vocabularies),
proportional to vocabulary size
• Large vocabulary also means: need huge amount of training data
since we need training samples for each word
• Difficult to train suitable references (or sets of references)
• Poor performance when the environment changes
• Unsuitable where speaker is unknown and no training is feasible
• Unsuitable for continuous speech, coarticulation
(combinatorial explosion of possible patterns)
• Impossible to recognize untrained words
• Difficult to train/recognize subword units
49

DTW for Single Word and Sentence Recognizers
Make a Wish
• We would like to work with speech units shorter than words
-> each subword unit occurs often, training is easier, need
less data
• We want to recognize speech from any speaker, without prior training
-> store "speaker-independent" references
• We want to recognize continuous speech not only isolated words
-> handle coarticulation effects, handle sequences of words
• We would like to be able to recognize words that have not been trained
-> train subword units and compose any word out of these
(vocabulary independence)
• We would prefer a sound mathematical foundation
• Solution (particularly sucessful for ASR): Hidden Markov Models
50

DTW for Single Word and Sentence Recognizers
Approaches to Alignment of Vector Sequences
First idea to overcome the varying length
of Utterances
Problem (2):
1. Normalize sequence length
2. Make a linear alignment
between the two sequences
Linear alignment can handle the problem of different speaking rates
But …..
What about varying speaking rates?
51

DTW for Single Word and Sentence Recognizers
Approaches to Alignment of Vector Sequences
Linear alignment can handle the problem of different speaking rates
But: it can not handle the problem of varying speaking rates during
the same utterance.
52

DTW for Single Word and Sentence Recognizers
Alignment of Speech Vectors May Be Non-Bijective
Given: two sequences x 1 ,x 2 ,...,x n and y 1 ,y 2 ,...,y m
Wanted: alignment relation R (not function), where (i,j) is in R
iff x i is aligned with y j .
y
x
1 2 3 4 5 6 7 8 9 ...
1 2 3 4 5 6 7 8 9 ...
It is possible that …
… more than one x is aligned to the same y (e.g. x3, x4) … more than one y is aligned to the same x (e.g. y8, y9) … more than an x or a y has no alignment partner at all (e.g. y6) 53

DTW for Single Word and Sentence Recognizers
y
x
Solution: Time Warping
Given:
two sequences x 1 ,x 2 ,...,x n and y 1 ,y 2 ,...,y m
Wanted:
alignment relation R, where (i,j) is in R
iff x i is aligned with y j ., i.e. are looking for
a common time-axis:
54
y
y 8,y 9 align to same
y 6 has no partner
x 3,x 4 align to the same y
x

DTW for Single Word and Sentence Recognizers
What is the best alignment relation R?
Distance Measure between two utterances:
For a given path R(i,j), the
distance between x and y is
the sum of all
local distances d(x i ,y j )
In our example:
d(x 1 ,y 1 ) + d(x 2 ,y 2 ) + d(x 3 ,y 3 )
+ d(x 4 ,y 3 ) + d(x 5 ,y 4 ) +
d(x 6 ,y 5 ) + d(x 7 ,y 7 ) + ...
Question:
How can we find a path that
gives the minimal overall
distance?
55

DTW for Single Word and Sentence Recognizers
Dynamic Programming
How can we find the minimal editing distance?
Greedy algorithm?
• Always perform the step that is currently the cheapest.
• If there are more than one cheapest step …
• … take any one of them.
Obvious: can't guarantee to lead to the optimal solution.
Solution: Dynamic Programming (DP)
DP is frequently used in operations research, where
consecutive decisions depend on each other and whose
sequence must lead to optimal results.
56

DTW for Single Word and Sentence Recognizers
Key Idea of Dynamic Programming
The key idea of DP is:
If we would like to take our system into a state s i , and
• we know the costs c 1 ,...,c k for the optimal ways to get
• from the start to all states q 1 ,...,q k from which we can go to s i ,
• then the optimal way to s i goes over the
state q l where l = argmin j c j
57

DTW for Single Word and Sentence Recognizers
DTW review
Goal: identify example pattern that is most similar to unknown input
compare patterns of different length
Note: all patterns are preprocessed 100 vectors / second of speech
DTW: Find alignment between unknown input and the example pattern
that minimizes the overall distance
Find average vector distance, but which frame-pairs ?
One Example Pattern
t 1 t 2 t M
t 1 t 2
?
58
Input = unknown pattern
t N
Euclidean Distance

DTW for Single Word and Sentence Recognizers
Example Pattern 1
DTW review
Apply transition function
t 1 t 2 t M
t 1 t 2
Unknown pattern
59
t N
Editing Distance,
for spell-checkers..
C(x,y)=
min (
C(x-1,y) + d ins(x,y),
C(x-1,y-1) + d sub(x,y),
C(x,y-1) + d del(x,y)
)
Weighted Distance
for SR..:
1
2
2
2
1

DTW for Single Word and Sentence Recognizers
The Dynamic Programming Matrix (2)
Usually for the minimal editing distance:
• The cost for deleting or inserting a character is 1
• The cost for replacing x i with y j is 0 (if x i = y j ) or 1 (else)
• Might be useful to define other costs for special purposes
Eventually:
• Remember for each state (i-1, j-1) which one was
the best predecessor (backpointer)
• Find the sequence of editing steps by
backtracing the predecessor pointers
from the final state
60

DTW for Single Word and Sentence Recognizers
DTW-Steps
Many different warping steps are possible and have been used.
Examples:
General rule is:
Cumulative cost of
destination =
best-of (cumulative cost
of source
+ cost of step
+ distance in destination)
Myers and Sakoe/Chiba
evaluated performance
and speed of DTW for a
wide range of
parameters: only small
differences in
performance for a fairly
wide range
symmetric
(editing distance)
61
Bakis
Itakura weighted

DTW for Single Word and Sentence Recognizers
DTW optimization
How to save computation time?
Solution: Add constraints
- define start and end point
- path should be close to diagonal,
- no two successive horizontal or vertical steps
Example Pattern 1
t 1 t 2 t M
X
t 1 t 2
X
Unknown pattern
62
t N
Shaded fields cannot
be reached and do not
need to be computed
(inactive)

DTW for Single Word and Sentence Recognizers
Constraints for the DTW-Path
• Endpoint constraints: we want the path not to skip a part
at the beginning or end of utterance
• Monotonicity conditions: we can't go back in time
(for neither utterance)
• Local continuity: possible types of motions (slope and
direction) of the path, no jumps etc.
• Global path constraints: path should be close to diagonal
•Slope weighting: DTW-path should be somehow
"smooth"
BTW: For most word recognition applications the warping path itself need not be
computed, only the accumulated distance D is required
63

DTW for Single Word and Sentence Recognizers
DTW with Beam Search
Idea:
do not consider steps to be possible out of states that have "too high"
cumulative distances.
Approaches:
• "expand" only a fixed number of states per column of DTW-matrix
• Expand only states that have a cumulative distance
less than a factor (the beam) times the best distance so far
64

DTW for Single Word and Sentence Recognizers
What we could do already
We can build a first simple isolated-word recognizer using DTW
We can build a preprocessor such that recorded speech can be processed
by the recognizer
We can recognize speech using DTW and print the score for each of its
reference patterns:
Example:
• Build recognizer that can recognize two words w 1 and w 2
• Collect training examples (in real life: a lot of data)
• Skip the optimization phase (don't need development set)
• Collect evaluation data (a few examples per word)
• Run tests on evaluation data and report results
65

DTW for Single Word and Sentence Recognizers
Single word recognizer
=>
Ref Word 2
Ref Word 1
66
Hypothese

DTW for Single Word and Sentence Recognizers
Search in Automatic Speech Recognition
Isolated Word Recognition:
• For every applicable vocabulary word:
• Compute it´s score (DTW)
• Compute it´s likelihood (forward algorithm)
• Report the word with the best likelihood / score
Problems, Optimization, Drawbacks
Continuous Speech Recognition:
• Not only find state sequence within words ...
… BUT ...
• … also find the word sequence
(i.e. a sequence through the huge state graph)
Problems, Strategies, Optimization, Drawbacks
67

DTW for Single Word and Sentence Recognizers
DTW optimization (2)
Pruning beam: Eliminate losers (score > best score + )
Reference pattern
Pattern 1 Pattern 2 Pattern 3
t N
t1 t2 Unknown pattern
68
Best scoring field at
this frame
Fields that have such a
bad score, that it is save
to ‘deactivate’ them
Fields with no active
predecessors don’t need
to be computed
This word is no
longer ‘active’

DTW for Single Word and Sentence Recognizers
DTW Summary
• Optimization of DTW:
• Usually, only interested in final score
• Algorithm requires only values in current and previous
frame
• Keep it simple: Do not allocate new storage but overwrite
stuff that is no longer needed
• For most transition patterns, one frame is enough
• Drawbacks of DTW:
• Does not generalize
• Speaker dependent
• Need example(s) for each word from each speaker
• Gets computationally expensive for large vocabularies
69

DTW for Single Word and Sentence Recognizers
Isolated Word Recognition with Template Matching
• For each word in the vocabulary, store at least one reference pattern
• When multiple reference patterns are available, either use all of
them or compute an average
• During recognition
• Record a spoken word
• Perform pattern matching with all stored patterns (or at least
with those that can be used in the current context)
• Compute a DTW score for every vocabulary word (when using
multiple references, compute one score out of many, e.g. average or
max)
• Recognize the word with the best DTW score
• This approach works only for very small vocabularies and/or for
speaker-dependent recognition
70

DTW for Single Word and Sentence Recognizers
From Isolated to Continuous Speech
• Sloppier
• Higher speaking rate
• Combinatorial explosion of things that can be said
• Spontaneous effects: restarts, fragments, noise
• Co-articulation: Did you dija ..
• Segmentation: how to find word boundaries
• Solution: reduce to known problems
71

DTW for Single Word and Sentence Recognizers
Plan 1: Cut Continuous Speech Into Single
Words
• Write magic algorithm that segments speech into 1-word
chunks
• Run DTW/Viterbi on each chunk
• BUT: Where are the boundaries????
• No reliable segmentation algorithm for detecting word
boundaries other than doing recognition itself, due to:
– Co-articulation between words
– Hesitations within words
– Hard decisions lead to accumulating errors
• integrated approach works better
72

DTW for Single Word and Sentence Recognizers
Plan 2: Two level DTW
Level 1 DTW: For each point in time (or each remotely likely word begin time)
Compute DTW with open end within all words
Level 2 DTW: For each word end and start point from first DTW
Combine words in second DTW based on calculated with-in scores
Problem: Extremely costly for larger vocabularies and longer words
Pattern
1
Pattern
2
73
Assumed word
begin frame
Active
corresponding
word end frames
(with scores)

DTW for Single Word and Sentence Recognizers
Plan 3: Depth First Search
Well known search strategies: depth first search vs. breath first search:
In speech recognition, this corresponds roughly to:
time-asynchronous vs. time-synchronous
In time-synchronous search, we expand every hypothesis simultaneously frame by frame.
In time-asynchronous search, we expand partial hypotheses that have different durations.
74

DTW for Single Word and Sentence Recognizers
Plan 4: One stage Dynamic Programming
• Within words:
– Viterbi
• Between words:
– For the first state of each words, consider the best last state of all
words as predecessor.
– This last state or the first state in the current word is predecessor.
• first state means any state that can be the first of a word
• last state means any state that has a transition out of the word
75

DTW for Single Word and Sentence Recognizers
Search with vs without Language Model
without grammar with grammar
• without grammar: The best predecessor state is the same for all
word-initial states
=> expand only the word-final state that has the best score in a frame
• with grammar: The best predecessor state depends also on the
word transition probability/penalty
76

DTW for Single Word and Sentence Recognizers
Reference Sentence
Compare Complete Utterances / Words
77
?
Hypothesis
= recognized
sentence

DTW for Single Word and Sentence Recognizers
Reference Sentence
Compare Smaller Units
78
Hypothesis
= recognized
sentence

DTW for Single Word and Sentence Recognizers
What we can‘t do yet:
OR Problems with Pattern Matching
• Need endpoint detection
• Need collection of reference patterns (inconvenient for user)
• Works only well for speaker-dependent recognition
(difficult to cover variations)
• High computational effort (esp. for large vocabularies),
proportional to vocabulary size
• Large vocabulary also means: need huge amount of training data
since we need training samples for each word
• Difficult to train suitable references (or sets of references)
• Poor performance when the environment changes
• Unsuitable where speaker is unknown and no training is feasible
• Unsuitable for continuous speech, coarticulation
(combinatorial explosion of possible patterns)
• Impossible to recognize untrained words
• Difficult to train/recognize subword units
79

DTW for Single Word and Sentence Recognizers
Make a Wish
• We would like to work with speech units shorter than words
-> each subword unit occurs often, training is easier, need
less data
• We want to recognize speech from any speaker, without prior training
-> store "speaker-independent" references
• We want to recognize continuous speech not only isolated words
-> handle coarticulation effects, handle sequences of words
• We would like to be able to recognize words that have not been trained
-> train subword units and compose any word out of these
(vocabulary independence)
• We would prefer a sound mathematical foundation
• Solution (particularly sucessful for ASR): Hidden Markov Models
80

DTW for Single Word and Sentence Recognizers
Speech Production as Stochastic Process
• The same word / phoneme / sound sounds different every time it is uttered
• We can regard words / phonemes as states of a speech production process
• In a given state we can observe different acoustic sounds
• Not all sounds are possible / likely in every state
• We say: in a given state the speech process "emits" sounds according to
some probability distribution/density
• The production process can make transitions from one state into another
• Not all transitions are possible, transitions have different probabilities
When we specify the probabilities for sound-emissions (emission
probabilities) and for the state transitions, we call this a model.
81

DTW for Single Word and Sentence Recognizers
What‘s different?
Reference in terms of state sequence of statistical
models, models consists of
prototypical references vectors
82
Hypothesis
= recognized
sentence

DTW for Single Word and Sentence Recognizers
Formal Definition of Hidden Markov Models
A Hidden Markov Model =(A,B,) is a five-tuple consisting of:
S The set of States S={s 1 ,s 2 ,...,s n } – n is the number of states
The initial probability distribution , (s i ) = P(q 1 = s i)
probability of s i being the first state of a sequence
A The matrix of state transition probabilities: 1 ≤ i, j ≤ n
A=(a ij ) with a ij = P(q t+1= s j|q t = s i) going from state s i to s j
B The set of emission probability distributions/densities,
B={b 1 ,b 2 ,...,b n } where b i (x)=P(o t = x|q t = s i) is the probability
of observing x when the system is in state s i
V Set of symbols -- v is the number of distinct symbols
The observable feature space can be discrete:
V={x 1 ,x 2 ,...,x v }, or continuous V=R d
83

DTW for Single Word and Sentence Recognizers
Generating an Observation Sequence
The term "hidden" refers to seeing observations and drawing
conclusions without knowing the hidden sequence of states
84

DTW for Single Word and Sentence Recognizers
Keep in Mind for Next Session
85
Hypothesis
= recognized
sentence