PhD thesis - School of Informatics - University of Edinburgh

Chapter 3. Tracking English Inclusions in German 69
Domain Method Accuracy Precision Recall F-score
Internet Baseline 93.95% - - -
Full system 98.25% 92.75% 77.37% 84.37
TextCat 92.24% 33.57% 28.87% 31.04
Space Baseline 96.99% - - -
Full system 99.45% 89.19% 93.61% 91.35
TextCat 93.80% 20.73% 37.32% 26.66
EU Baseline 99.69% - - -
Full system 99.78% 59.26% 76.19% 66.67
TextCat 96.43% 2.54% 28.57% 4.66
Table 3.8: Performance of the English inclusion classifier compared to the baseline and
the performance of TextCat.
In order to get an idea of how a conventional LID system performs on the task
of recognising English inclusions embedded in German text, Table 3.8 also reports
the performance of TextCat, an automatic LID tool based on the character n-gram
frequency text categorisation algorithm proposed by Cavnar and Trenkle (1994) and
reviewed in Section 2.2. While this LID tool requires no lexicons, its F-scores are
low for the internet and space travel domains (31.04 and 26.66, respectively) and very
poor for the EU data (4.66). This confirms that the identification of English inclusions
is more difficult for this domain, coinciding with the result of the English inclusion
classifier. The low scores also prove that such conventional n-gram-based language
identification alone is unsuitable for token-based language classification, particularly
in case of closely related languages.
3.4.2 Evaluation of Individual System Modules
The full system described in Section 3.3 combines a lexicon lookup module, a search
engine module and a post-processing module in order to classify English inclusions in
German text. This section reports the performance of individual system modules of
the English inclusion classifier compared to those of the full system and the baseline
scores. It shows that the combination of individual models leads to a performance
increase of the system on mixed-lingual data.