PhD thesis - School of Informatics - University of Edinburgh

Chapter 4. System Extension to a New Language 105
tools and therefore demanded more time and effort to be customised for French. The
core system was adapted in approximately one person week in total (Section 4.1). Fig-
ure 4.2 illustrates the system architecture after extending it to French. Note that the
system now involves an additional document-based language identification step after
pre-processing in which the base language of the document is determined by TextCat
(Cavnar and Trenkle, 1994). TextCat, a traditional language identification tool, per-
forms well on identifying the language of sentences and larger passages. This enables
running the English inclusion classifier over either German or French text without
having to specify the base language of the text manually. The base-language-specific
classifier components are therefore initiated purely based on TextCat’s language iden-
tification. For both the German and French newspaper articles, TextCat is always able
to identify the language correctly.
4.3.1 Pre-processing Module
The pre-processing module involves tokenisation and POS tagging (cf. Section 3.3.2).
First, the German tokeniser was adapted to French and a French part-of-speech (POS)
tagger was integrated into the system. The French tokeniser consists of two rule-based
tokenisation grammars. In the same way as the German version, it not only identifies
tokens surrounded by white space and punctuation but also resolves typical abbrevia-
tions, numerals and URLs. Both grammars are applied by means of improved upgrades
of the XML tools described in Thompson et al. (1997) and Grover et al. (2000). These
tools process an XML input stream and rewrite it on the basis of the rules provided.
The French TreeTagger (see Section 3.5.1.2) is used for POS tagging. It is freely avail-
able for research and is also trained for a number of other languages, including German
and English. The TreeTagger functions on the basis of binary decision trees trained on
a French corpus of 35,448 word forms and yields a tagging accuracy of over 94% on
an evaluation data set comprising of 10,000 word forms (Stein and Schmidt, 1995). 2
2 While trained models are available online, the tagged data set that was used to train and evaluate
the French TreeTagger model is not part of the distribution. Otherwise, the data could have been used
to train TnT, as that tagger resulted in a better performance of the English inclusion classifier on the
German development data (see Section 3.5.1).