PhD thesis - School of Informatics - University of Edinburgh

Chapter 3. Tracking English Inclusions in German 56
3.3.1 Processing Paradigm
The underlying processing paradigm of the English inclusion classifier is XML-based.
As a markup language for NLP tasks, XML is expressive and flexible yet constrain-
able. Furthermore, there exists a wide range of XML-based tools for NLP applications
which lend themselves to a modular, pipelined approach to processing whereby lin-
guistic knowledge is computed and added incrementally as XML annotations. More-
over, XML’s character encoding capabilities facilitate multilingual processing. As il-
lustrated in Figure 3.1, the system for processing German text is essentially a UNIX
pipeline which converts HTML files to XML and applies a sequence of modules: a pre-
processing module for tokenisation and POS tagging, followed by a lexicon lookup, a
search engine module, post-processing and an optional document consistency check
which all add linguistic markup and classify tokens as either German or English. The
pipeline is composed partly of calls to LT-TTT2 and LT-XML2 (Grover et al., 2006) 4 for
tokenisation and sentence splitting. In addition, non-XML public-domain tools such
as the TnT tagger (Brants, 2000b) were integrated and their output incorporated into
the XML markup. The primary advantage of this architecture is the ability to integrate
the output of already existing tools with that of new modules specifically tailored to
the task in an organised fashion. The XML output can be searched to find specific
instances or to acquire counts of occurrences using the LT-XML2 tools.
3.3.2 Pre-processing Module
All downloaded Web documents are first of all cleaned up using TIDY 5 to remove
HTML markup and any non-textual information and then converted into XML. Alter-
natively, the input into the classifier can be in simple text format which is subsequently
converted into XML format. The resulting XML pages simply contain the textual in-
formation of each article. Subsequently, all documents are passed through a series of
pre-processing steps implemented using the LT-XML2 and LT-TTT2 tools (Grover et al.,
2006) with the output of each step encoded in XML.
Two rule-based grammars which were developed specifically for German are used
4 These tools are improved upgrades of the LT-TTT and LT-XML toolsets (Grover et al., 2000; Thompson
et al., 1997) and are available under GPL as LT-TTT2 and LT-XML2 at: http://www.ltg.ed.ac.
uk. 5http://tidy.sourceforge.net