PhD thesis - School of Informatics - University of Edinburgh

Chapter 3. Tracking English Inclusions in German 93
ble 3.20). In the first experiment (ID1), the tagger’s standard feature set is used which
includes words, character sub-strings, word shapes, POS tags, abbreviations and NE
tags (Finkel et al., 2005). The resulting F-scores are high both for the internet and
space travel data (84.74 and 91.29 points) but extremely low for the EU data (13.33
points) due to the sparseness of English inclusions in that data set. ID2 involves the
same setup as ID1 but eliminating all features relying on the POS tags. The tagger
performs similarly well for the internet and space travel data as for ID1 but improves
by 8 points to an F-score of 21.28 for the EU data. This can be attributed to the fact that
the POS tagger does not perform with perfect accuracy particularly on data containing
foreign inclusions. Training the supervised tagger on POS tag information is therefore
not necessarily useful for this task, especially when the data is sparse. Despite the
improvement, there is a big discrepancy between the F-score which the ML classifier
produces for the EU data and those of the other two data sets.
Table 3.20 compares the best F-scores produced with the tagger’s own feature set
(ID2) to the best results of the English inclusion classifier presented in this thesis and
the baseline. The best English inclusion classifier is the full system combined with
consistency checking (Section 3.3.6). For the EU data, the English inclusion classi-
fier performs significantly better than the supervised tagger (χ 2 : d f = 1, p ≤ 0.05).
However, since this data set only contains a small number of English inclusions, this
result is not unexpected. It is therefore difficult to draw any meaningful conclusions
from these results. For the internet and space travel data sets, which contain many En-
glish inclusions, the trained maxent tagger and the English inclusion classifier perform
equally well, i.e. without statistical significance between the difference in performance
(χ 2 : d f = 1, p ≤ 1). The fact that the maxent tagger requires hand-annotated train-
ing data, however, represents a serious drawback. Conversely, the English inclusion
classifier does not rely on annotated data and is therefore much more portable to new
domains. Section 3.4.3 shows that it performs well on unseen data in three different
domains as well as on entirely new data provided by another research group. Given
the necessary lexicons, the English inclusion classifier can easily be run over new text
and text in a different language or domain without further cost. The time required to
port the classifier to a new language is the focus of attention in the next chapter.