Data coding of, and transition time in, ASL ... - Jonathan Keane

Introduction Methods and Coding Results Conclusions References
DATA CODING OF, AND TRANSITION TIME IN,
ASL FINGERSPELLING
Jonathan Keane
University of Chicago

Introduction Methods and Coding Results Conclusions References
Outline
Introduction
Methods and Coding
Coding Principles
Data collection
Our coding method
Coding Principles again
Transitions
Results
Speed
Signer
Word type
Position
Individual Letters
Transitions again
Conclusions

Introduction Methods and Coding Results Conclusions References
Background – Fingerspelling
All handshapes are static except for -J- and -Z-.
Fingerspelling makes up anywhere from
12–35% of ASL discourse.
(Padden, 1991; Padden and Gunsauls, 2003)
j
z

Introduction Methods and Coding Results Conclusions References
The phonetics of Fingerspelling
Wilcox (1992) looks at about 7 words and describes some of the
dynamics of hand motion.
Tyrone et al. (1999) looks at fingerspelling by parkinsonian
signers from a phonetic perspective.
Brentari and Padden (2001); Cormier et al. (2008) both look at
the nativization process for fingerspelled words.
Quinto-Pozos (2010) described the rate of fingerspelling for two
signers within fluent discourse.

Introduction Methods and Coding Results Conclusions References
Coding Principles
What data looks like
data.mpg

Introduction Methods and Coding Results Conclusions References
Coding Principles
Data coding needs to be
Accurate
Accurate, detailed data is necessary for any linguistic analysis.
Reproducible
Coding should be able to be reproduced, and individual coders
should form some sort of consensus.
Quick
Coding time is often directly related to the amount of data we can
collect.
Easy
A coding system that requires little specialized training is better
than one that requires experts to use. (All else being equal)

Introduction Methods and Coding Results Conclusions References
Data collection
Recording specifications
Signers
2 signers, both are deaf of deaf parents, and native ASL users.
Video
2 video cameras recording at 60 FPS.
We collected 2 sessions for each signer
1 at a normal, conversational speed, and 1 at a careful speed.
There were 100 non-English words, 100 names, 100 nouns.
Each word was fingerspelled twice in each speed.
The video was then post processed and compressed for coding.

Introduction Methods and Coding Results Conclusions References
Data collection
Session details
Careful elicitation and data collection allowed us to maximize the
data we started with.
1. Output a logfile with the order of the words as they were
presented to the signer (although this could be improved further)
2. Segmentation – green button
3. First pass error detection – red button

Introduction Methods and Coding Results Conclusions References
Our coding method
3–4 humans hand coded apogees
Using ELAN, coders watched the videos at 20–40% speed.
Told to press a button whenever they thought there was an apogee.
Described as the point where the hand was maximally or
minimally open.
Could be described as the minimum instantaneous velocity of all
of the articulators.
Use discretion when coding apogees with movement, but be
consistent.
Not defined as the canonical form

Introduction Methods and Coding Results Conclusions References
Our coding method
The position of each apogee was
algorithmically determined.
Minimized the mean absolute distance between the points for each
word.
We accounted for errant, and missing presses by assigning a
violation cost for every apogee that was deleted or added.
The coders were already fairly close together.
Mean absolute deviation:
27.93 msec for all letters
62.52 msec for letters with movement

Introduction Methods and Coding Results Conclusions References
Our coding method
Leveraging known data
A first guess at the letter of each apogee was added using left edge
forced alignment.
Although the letters it assigns are not 100% accurate, they are close.
Finally someone trained in fingerspelling went through and verified
the location, and letter of each apogee. The vast majority of apogees
are unchanged.

Introduction Methods and Coding Results Conclusions References
Our coding method
Example

Introduction Methods and Coding Results Conclusions References
Our coding method
Verification
5404 apogees of 6594 in normal are unchanged. (∼ 81%)
Density
0.00 0.02 0.04 0.06 0.08 0.10
−200 −100 0 100 200
msec

Introduction Methods and Coding Results Conclusions References
Our coding method
Verification
Of the changed apogees, they are often shifted back by 2–3 frames.
Density
0.000 0.004 0.008
−200 −100 0 100 200
msec

Introduction Methods and Coding Results Conclusions References
Coding Principles again
How does our method look
Accurate
Much less chance for transcription or other errors compared to
traditional methods.
Reproducible
The first stage of coding is incredibly reproducible, to a high
degree of accuracy. The second step might be (we’re working on
testing this)
Quick
Varied, but each clip chunk (5–10 min of data) took 105 min for 3
coders, 75 min of which can be done simultaneously by 3 coders,
so 55 min at the fastest.
Easy
Our initial pass for coding can be done with very little training.
The verification task requires a bit of training in fingerspelling,
but since it’s more confirmation, it’s much easier and quicker than

Introduction Methods and Coding Results Conclusions References
Transitions
Example
staxi.mp4
Figure: T-A-X-I, normal speed.

Introduction Methods and Coding Results Conclusions References
Transitions
Example
staxi2.mp4
Figure: T-A-X-I, normal speed, slow motion

Introduction Methods and Coding Results Conclusions References
Transitions
Example, revisited
164 ms 249 ms 246 ms
| | | |
Figure: durations for T-A-X-I

Introduction Methods and Coding Results Conclusions References
Transitions
T-A-X-I, L-A-M-B, F-R-E-D, C-A-R-P, and P-U-H-U
100 ms 174 ms 183 ms
| | | |
214 ms 211 ms 191 ms
| | | |
252 ms 162 ms 182 ms
| | | |
153 ms 207 ms 182 ms
| | | |
225 ms 355 ms 224 ms
| | | |
Figure: durations for T-A-X-I, L-A-M-B, F-R-E-D, C-A-R-P, and P-U-H-U
(signer: s1 speed: normal)

Introduction Methods and Coding Results Conclusions References
Questions
1. When asked to sign at two different speeds, how much of a
difference is there between them
2. Is there individual variation
3. Does the type of word affect the speed of fingerspelling
4. Does the position of a letter in a word affect transition time
5. Do letters with movement take longer to execute
6. Does articulatory complexity change transition time

Introduction Methods and Coding Results Conclusions References
ANOVA table
Df Sum Sq Mean Sq F value Pr(>F)
wordtype 2 33.19 16.59 220.34 0.0000
speed 1 1425.48 1425.48 18927.22 0.0000
signer 1 13.90 13.90 184.55 0.0000
wordtype:speed 2 13.04 6.52 86.58 0.0000
wordtype:signer 2 0.20 0.10 1.36 0.2570
speed:signer 1 232.93 232.93 3092.84 0.0000
Residuals 12267 923.87 0.08
Table: ANOVA table for log(time)

Introduction Methods and Coding Results Conclusions References
Speed
between speeds
speed
time (msec)
200
400
600
800
careful
normal
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

Introduction Methods and Coding Results Conclusions References
Speed
ANOVA table
Df Sum Sq Mean Sq F value Pr(>F)
wordtype 2 33.27 16.64 219.11 0.0000
speed 1 1424.43 1424.43 18760.33 0.0000
signer 1 13.96 13.96 183.82 0.0000
wordtype:speed 2 13.57 6.79 89.38 0.0000
wordtype:signer 2 0.24 0.12 1.60 0.2020
speed:signer 1 234.20 234.20 3084.53 0.0000
Residuals 12354 936.53 0.08
Table: ANOVA table for log(time)

Introduction Methods and Coding Results Conclusions References
Signer
between signers, by speed
signer
time (msec)
200
400
600
800
s1
s2
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
careful
s1
s2
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
normal

Introduction Methods and Coding Results Conclusions References
Signer
ANOVA table
Df Sum Sq Mean Sq F value Pr(>F)
wordtype 2 33.27 16.64 219.11 0.0000
speed 1 1424.43 1424.43 18760.33 0.0000
signer 1 13.96 13.96 183.82 0.0000
wordtype:speed 2 13.57 6.79 89.38 0.0000
wordtype:signer 2 0.24 0.12 1.60 0.2020
speed:signer 1 234.20 234.20 3084.53 0.0000
Residuals 12354 936.53 0.08
Table: ANOVA table for log(time)

Introduction Methods and Coding Results Conclusions References
Word type
between wordtypes, by speed
wordtype
time (msec)
200
400
600
800
foreign name noun
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
careful
foreign name noun
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
normal

Introduction Methods and Coding Results Conclusions References
Word type
ANOVA table
Df Sum Sq Mean Sq F value Pr(>F)
wordtype 2 33.27 16.64 219.11 0.0000
speed 1 1424.43 1424.43 18760.33 0.0000
signer 1 13.96 13.96 183.82 0.0000
wordtype:speed 2 13.57 6.79 89.38 0.0000
wordtype:signer 2 0.24 0.12 1.60 0.2020
speed:signer 1 234.20 234.20 3084.53 0.0000
Residuals 12354 936.53 0.08
Table: ANOVA table for log(time)

Introduction Methods and Coding Results Conclusions References
Position
300
+
Short words (3 – 6 letters) – s1, normal
+
+
+
+
+
+
+
+
+
+
+
+
+
250
time (msec)
200
150
100
50
1 2 3 4 5
position

Introduction Methods and Coding Results Conclusions References
Position
300
+
+
Long words (8 – 10 letters) – s1, normal
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
250
time (msec)
200
150
100
+
+
+
+
1 2 3 4 5 6 7 8 9 10
position

Introduction Methods and Coding Results Conclusions References
Position
Possible explanations
1. memory limitations
2. articulation limitations
3. phonological chunking
4 letters ∼= 3 movements ∼= 1 ASL sign

Introduction Methods and Coding Results Conclusions References
Position
800
700
careful +
+
+
+
+
is different
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
time (msec)
600
500
400
300
+
1 2 3 4 5 6 7 8 9 10 11 12
position
+
+
+

Introduction Methods and Coding Results Conclusions References
Position
+
+
non-english is different
+
+
+
+
+
+
+
+
+
+
300
+
+
250
time (msec)
200
150
100
1 2 3 4 5 6 7 8 9 10
position

Introduction Methods and Coding Results Conclusions References
Individual Letters
T-A-X-I, L-A-M-B, F-R-E-D, C-A-R-P, and P-U-H-U
100 ms 174 ms 183 ms
| | | |
214 ms 211 ms 191 ms
| | | |
252 ms 162 ms 182 ms
| | | |
153 ms 207 ms 182 ms
| | | |
225 ms 355 ms 224 ms
| | | |
Figure: durations for T-A-X-I, L-A-M-B, F-R-E-D, C-A-R-P, and P-U-H-U
(signer: s1 speed: normal)

Introduction Methods and Coding Results Conclusions References
Individual Letters
T-A-X-I, L-A-M-B, F-R-E-D, C-A-R-P, and P-U-H-U
100 ms 174 ms 183 ms
t a x i
214 ms 211 ms 191 ms
l a m b
252 ms 162 ms 182 ms
f r e d
153 ms 207 ms 182 ms
c a r p
225 ms 355 ms 224 ms
p u h u
Figure: durations for T-A-X-I, L-A-M-B, F-R-E-D, C-A-R-P, and P-U-H-U
(signer: s1 speed: normal)

Introduction Methods and Coding Results Conclusions References
Individual Letters
400
transitions for high frequency letters
+
+
+
+
+
+
273 197 340 144 299
+
+
+
+
+
+
+
+
+ +
+
+
+
+
+
+
+
200
time (msec)
0
−200
−400
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
299 212 369 146 360
+
+
+
+
i o+
a t e
+
+
+

Introduction Methods and Coding Results Conclusions References
Individual Letters
transitions + for for low frequency letters
+
400
+
+
+
59 46 42 52 +
42
200
time (msec)
0
−200
−400
+
+
43 26 21 43 80
+
+
+
+
z j q x y
+

Introduction Methods and Coding Results Conclusions References
Individual Letters
Movement in -YsNormal.mp4
Figure: The first 10 instances of -Y- – not at the end of the word

Introduction Methods and Coding Results Conclusions References
time (msec)
Individual Letters
400
300
200
82 bigrams with + more + than 50 instances
+ +
+
+
+
+
+
+
+
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
100
+
+
+
+
+
+
+
+
32 36 52 4416560 36 48 25 36 56 33 31 36 28 21 28 24 8510510135 36 32 36 27 56 39 35 57 78 41 48 27 25 28 28 66 57 60 27 28 44 32 28 48 36 56 31 35 28 64 29 37 27 28 36 32 83 70 57 37 35 35 47 22 60 72 52 43 36 33 32 28 26 27 36 27 32 32 26 31
ac ad al aman ar as at ax be ca ce co de ea ee eg ek el en er es ex fi ge ho ia ic ie il in is it ji ka ke ko la le li mamemi na nd ne ng ni no nt ol on or osowpe po pr qu ra re ri sa se si ss st ta te ti to ua ue ui ur us va ve vi wi xe ze
letter

Introduction Methods and Coding Results Conclusions References
Individual Letters
Transition time by articulatory complexity
300
mean transition time
250
200
0 100 200 300 400 500
complexity score

Introduction Methods and Coding Results Conclusions References
Conclusions
1. When asked to fingerspell at different speeds, the spread is
significant.
2. There is individual variation overall and in speed, but not
wordtype.
3. Signers fingerspell slower on non-English words.
4. Signers seem to chunk their production into 3-4 letter chunks
with longer words.
5. Letters with movement take longer to execute.
6. The class of letters that have movement might need redefining:
-Y- and possibly -Q-.
7. Transition time does not seem to correlate with articulatory
complexity.

Introduction Methods and Coding Results Conclusions References
Future directions
1. More sophisticated modeling
2. Quantification of other articulatory features
3. Recognition related tasks
4. More signers (in progress!)

Introduction Methods and Coding Results Conclusions References
Thank you for coming.
I must also acknowledge the contributions of many who
contributed in ways big and small:
Fingerspelling data
Drucilla Ronchen and Andy Gabel
Main advisors
Jason Riggle and Diane Brentari
Other researchers
Susan Rizzo, Karen Livescu, Greg Shakhnarovich, Raquel
Urtasun, Erin Dhalgren, and Katie Henry.

Introduction Methods and Coding Results Conclusions References
Brentari, Diane, and C. Padden. 2001. Foreign vocabulary in sign languages:
A cross-linguistic investigation of word formation, chap. Native and foreign
vocabulary in American Sign Language: A lexicon with multiple origins,
87–119. Mahwah, NJ: Lawrence Erlbaum.
Cormier, Kearsy, Adam Schembri, and Martha E. Tyrone. 2008. One hand or
two nativisation of fingerspelling in asl and banzsl. Sign Language &
Linguistics 11.3–44.
Padden, Carol. 1991. Theoretical issues in sign language research, chap. The
Acquisition of Fingerspelling by Deaf Children, 191–210. The University of
Chicago press.
Padden, Carol, and Darline Clark Gunsauls. 2003. How the alphabet came to
be used in a sign language. Sign Language Studies 4.10–33.
Quinto-Pozos, David. 2010. Rates of fingerspelling in american sign
language. Poster at TISLR 10.
Tyrone, M.E., J. Kegl, and H. Poizner. 1999. Interarticulator co-ordination in
deaf signers with parkinson’s disease. Neuropsychologia 37.1271–1283.
Wilcox, Sherman. 1992. The phonetics of fingerspelling. John Benjamins
Publishing Company.

Introduction Methods and Coding Results Conclusions References
between wordtypes, by speed and by signer
wordtype
time (msec)
200
400
600
800
foreign name noun
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
careful
s1
foreign name noun
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
normal
s1
foreign name noun
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
careful
s2
foreign name noun
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
normal
s2