tekom-Jahrestagung 2012 - ActiveDoc

Sprachtechnologie / Language Technology
A hardened service layer architecture
The first and most important task was to create a stable production environment.
To achieve this, a service layer was put on top of Moses SMT,
providing redundancy, load balancing, asynchronous processing, failover
support, industry-standard document format support, alignment-based
tag-handling, improved normalization and hardened (de)tokenization
and (lower/real)casing.
To achieve a separation of concerns, linguistic processing by Moses SMT
and text-engineering capabilities were strictly separated. As a result,
the hardened Moses SMT set-up allows the deployment of third-party
translation and language models, while still providing the text engineering
capabilities built on top of the translation workflow. Named entity
recognition (NER) and terminology management services, for example,
can be added without disrupting the Moses SMT models. From a technical
point of view, tagging and annotation are considered to be engineering
issues, which are not allowed to interfere with the linguistic issues,
as addressed by the SMT models. From a commercial point of view, clients
can have third parties focus on linguistic quality, while the framework
can still take care of immediate production use.
A pipeline architecture
If we take a closer look at the evolution of Moses SMT and MT in general,
it becomes clear that investing in a consistent design, that follows the
very principles of modern MT, is well worth the effort: it alleviates the
costs related to the implementation of difficult languages and it streamlines
processing workflows.
To clarify this, we need to address a common misunderstanding among
non-academic users: Moses SMT is a toolkit, not a monolithic application
that will reach the 1.0 milestone somewhere in the near future. It
consists of data training scripts and a decoder that can handle so-called
factored input. It is designed to primarily work with phrase alignments
and n-gram models, but it can handle far more information. Up to date
however, the factored model (which may include for example syntactic
information) has not lived up to the expectations, mainly due to the data
sparseness introduced by extra information and the locality issues inherent
to n-gram models.
Still, the original concept of processing other information than phrase
alignment is useful, but it may be more fruitful to incorporate it outside
of the decoder and the training. This is exactly what most researchers
do when developing so-called “hybrid” systems. For agglutinative
languages, for example, morphological analyzers are used to transform
training and input data before they are fed to Moses SMT. In other
words: modelling is executed outside of the training scripts and the
decoder. For this reason, there is a growing consensus in the research
community that MT is more and more evolving towards a pipeline process
in which Moses SMT plays an important role. This is in contrast to
what most people picture when making a Moses SMT buy decision.
With the pipeline architecture, the stage is set for developing difficult
language pairs. In the meantime it can be used to harbour text engineering
tools, which basically also add extra information to the translation
process.
tekom-Jahrestagung 2012
261