PhD thesis - School of Informatics - University of Edinburgh

Chapter 3. Tracking English Inclusions in German 82
The F-score for the space travel test data is almost 7 points lower than that ob-
tained for the development set. This performance drop is caused by lower precision
and recall. Although the classifier is overgenerating on the development set for this
particular domain, the fact that the scores on the unseen test data are relatively con-
sistent across all three different domains is a positive result. Moreover, each data set
is relatively small which makes it difficult to draw clear conclusions. In fact, the test
and development data on space travel are slightly different in nature as can be seen in
Table 3.1. While both sets contain a similar percentage of English inclusions (2.8%
versus 3%), those in the test set are much less repeated than those in the development
set which is reflected in their type-token-ratios amounting to 0.33 and 0.15, respec-
tively. Therefore, the higher development test data scores could be due to the higher
number of repetitions of English inclusions in the space travel development data.
The full system F-scores for the EU test data are considerably higher than for the
development set (84.42 versus 65.22 points). This is not surprising since the EU de-
velopment data only contains 30 different English inclusions, less than 1% of all types,
which made it an unusual data set for evaluating the classifier on. Error analysis was
therefore focused mainly on the output of the other two data sets. The EU test data, on
the other hand, contains 86 different English inclusions, i.e. three times as many types
as in the development data (see Table 3.1). Considering that the English inclusion clas-
sifier yields an equally high performance on the unseen EU test data as on the other
two test data sets, it can be concluded that system design decisions and post-processing
rules are made general enough to apply to documents on different domains.
The best overall F-scores on all six data sets are obtained when combining the full
system with a second consistency checking run (Internet test data: F=84.78, Space
travel test data: F=84.66, EU test data: F=84.68). This second run essentially ensures
that all English inclusions found in the first run are consistently classified within each
document. This is done by applying an on-the-fly gazetteer which is generated auto-
matically. This setup was explained in more detail in Section 3.3.6. The results listed
in Table 3.14 show that the improvement in F-score is always caused by an increase in
recall, outweighing the smaller decrease in precision. While this improvement is es-
sential for document classification, particularly when comparing different classifiers,
it is unlikely to be beneficial when performing language classification on tokens in
individual sentences, for example during the text analysis of a TTS synthesis system.