PhD thesis - School of Informatics - University of Edinburgh

Chapter 3. Tracking English Inclusions in German 85
occurring in advertising are conspicuous for nearly all of the respondents (98%). A
closer look at the data reveals that Marcadet et al. (2005) do not consistently annotate
abbreviations and acronyms expanding to English definitions as English. Conversely,
the English inclusion classifier presented in this thesis is designed to recognise them
as well. Moreover, person names like Ted Saskins are annotated as English and not
distinguished from real English inclusions as advocated in this thesis.
On the reconciled gold standard, the English inclusion classifier performs with an
F-score of 96.35 points (an accuracy of 98.95%, a precision of 97.78% and a recall of
94.96%). These scores are slightly better than those reported by Marcadet et al. (2005)
on this data set (98.67% accuracy). However, it is not entirely straightforward to com-
pare these scores as the gold standard annotation is reconciled. The few classification
errors are mainly due to English words like Team or Management being already listed
in the German lexicon. These anglicisms are strongly integrated in the German lan-
guage and have well established pronunciations. Therefore, such classification errors
are unlikely to cause pronunciation problems during TTS synthesis.
Given the results of both sets of evaluations, it can be concluded that the English
inclusion classifier performs well on randomly selected unseen mixed-lingual data in
different domains and compares well to an existing mixed-lingual LID approach.
3.5 Parameter Tuning Experiments
This section discusses a series of interesting parameter tuning experiments to optimise
the English inclusion classifier. These were the basis for the final design of the full
system which was evaluated in the previous section. These experiments include a task-
based evaluation of three different POS taggers and a task-based evaluation of two
search engines. All experiments involve the German development data for evaluation.
3.5.1 Task-based Evaluation of Different POS taggers
Throughout the entire process of error analysis, it was noticed that the performance of
the English inclusion classifier depends to some extent on the performance on the POS
tagger. Initially, the system made use of the POS tagger TnT (Brants, 2000b) trained
on the NEGRA corpus (Skut et al., 1997). Some classification errors result from errors