PhD thesis - School of Informatics - University of Edinburgh

Chapter 3. Tracking English Inclusions in German 57
to tokenise the XML documents. The first grammar pre-tokenises the text into tokens
surrounded by white space and punctuation and the second grammar groups together
various abbreviations, numerals and URLs. Grammar rules also split hyphenated to-
kens. The two grammars are applied with lxtransduce 6 , a transducer which adds or
rewrites XML markup to an input stream based on the rules provided. lxtransduce
is an improved version of fsgmatch, the core program of LT-TTT (Grover et al.,
2000). The tokenised text is then POS-tagged using the statistical POS tagger TnT
(Trigrams’n’Tags). The tagger is trained on the TIGER Treebank (Release 1) which
consists of 700,000 tokens of German newspaper text (Brants et al., 2002) annotated
with the Stuttgart-Tübingen Tagset (Schiller et al., 1995), henceforth referred to as
STTS.
3.3.3 Lexicon Lookup Module
The lexicon module performs an initial language classification run based on a case-
insensitive lookup procedure using two lexicons, one for the base language of the text
and one for the language of the inclusions. The system is designed to search CELEX
Version 2 (Celex, 1993), a lexical database of German, English and Dutch. The Ger-
man database holds 51,728 lemmas and their 365,530 word forms and the English
database contains 52,446 lemmas representing 160,594 corresponding word forms. A
CELEX lookup is only performed for tokens which TnT tags as NN (common noun),
NE (named entity), ADJA or ADJD (attributive and adverbial or predicatively used adjec-
tives) as well as FM (foreign material). Anglicisms representing other parts of speech
are relatively infrequently used in German (Yeandle, 2001) which is the principal rea-
son for focussing on the classification of noun and adjective phrases. Before the lexicon
lookup is performed, distinctive characteristics of German orthography are exploited
for classification. So, all tokens containing German umlauts are automatically recog-
nised as German and are therefore not further processed by the system.
The core lexicon lookup algorithm involves each token being looked up twice, both
in the German and English CELEX databases. Each part of a hyphenated compound
is checked individually. Moreover, the lookup in the English database is made case-
insensitive in order to identify the capitalised English tokens in the corpus, the reason
6 http://www.ltg.ed.ac.uk/˜richard/lxtransduce.html