PhD thesis - School of Informatics - University of Edinburgh

Chapter 4. System Extension to a New Language 114
development and test sets versus 0.29 and 0.25 in German development and test sets,
see Table 4.1). This means that the classifier is already less likely to miss inclusions
which minimises the effect of consistency checking for French.
Test set Development set
Method Acc P R F Acc P R F
French data
FS 98.10% 88.59% 84.11% 86.29 98.45% 91.60% 84.08% 87.68
FS+CC 98.08% 88.35% 84.30% 86.28 98.49% 91.39% 85.09% 88.13
German data
FS 97.93% 92.13% 75.82% 83.18 98.07% 93.48% 73.31% 82.17
FS+CC 98.13% 91.58% 78.92% 84.78 98.25% 92.75% 77.37% 84.37
Table 4.5: Evaluation of the full system (FS) with optional consistency checking (CC).
4.5 Chapter Summary
This chapter described how the English inclusion classifier was successfully converted
to a new language, French. The extended system is able to process either German or
French text for identifying English inclusions. The system is a pipeline made up of sev-
eral modules, including pre-processing, a lexicon, a search-engine, a post-processing
and a document consistency checking module. The extension of the core system was
carried out in only one person week and resulted in a system performance of 71.59
points in F-score on the French development data. A further week was spent on im-
plementing the post-processing module which boosted the F-score to 87.68 points. A
third week was required to select external language resources plus collect and annotate
French evaluation data in the domain of internet and telecoms.
The performance drop between the French development set and the unseen test sets
is relatively small (1.85 in F-score) which means that the system does not seriously
over- or undergenerate for this domain but results in an equally high performance on
new data. This chapter also demonstrated that the English inclusion classifier is easily
extendable to a new language in a relative short period of time and without having to