Rome Wasn't Digitized in a Day - Council on Library and Information ...

22
In ancient manuscripts, Sanskrit is written without spaces, and from our po<strong>in</strong>t of view, this is an
important graphical specificity, because it <strong>in</strong>creases greatly the complexity of text comparison
algorithms. One may remark that Sanskrit is not the only language where spaces are miss<strong>in</strong>g <strong>in</strong>
the text: Roman epigraphy and European Middle Age manuscripts are also good examples of
that (Csernel and Patte 2009).
The solution that the authors ultimately proposed for creat<strong>in</strong>g a critical edition of a Sanskrit text
<strong>in</strong>volved the lemmatization by hand of one of the two texts, specifically, the text of the edition.
Alignments between this lemmatized text and other texts then made use of the longest common
subsequence (LCS) algorithm. The authors are still experiment<strong>in</strong>g with their methodology, but po<strong>in</strong>ted
out that the absence of a Sanskrit lexicon limited their approach.
The development of OCR tools that will process Sanskrit scripts is a highly sought-after goal. Very
little work has been done <strong>in</strong> this area, but Thomas Breuel recently reported not only on the use of
OCRopus to recognize the Devanagari script but also on its application both to primary texts <strong>in</strong>
classical languages and to secondary classical scholarship. As was discussed previously <strong>in</strong> Boschetti et
al. (2009), prelim<strong>in</strong>ary work with OCRopus produced promis<strong>in</strong>g results with Ancient Greek.
Breuel (2009) described OCRopus as an OCR system that is designed to be both omnil<strong>in</strong>gual and
omniscript and that advances the state of the art <strong>in</strong> that new text-recognition and layout-analysis
modules can be easily plugged <strong>in</strong> and that it uses an adaptive and user-extensible character recognition
module. Breuel acknowledged that there are many challenges to recogniz<strong>in</strong>g Devanagari script,
<strong>in</strong>clud<strong>in</strong>g the large number of ligatures, complicated diacritics, and the “large and unusual vocabulary
used <strong>in</strong> academic and historical texts” (Breuel 2009). In addition to Sanskrit texts, Breuel made the
important po<strong>in</strong>t that historical scholarship about Sanskrit and other classical languages is frequently
multil<strong>in</strong>gual and multiscript and can mix Devanagari and Lat<strong>in</strong> as well as Greek. Breuel thus proposed
that OCRopus has a number of potential applications <strong>in</strong> the field of classical scholarship, <strong>in</strong>clud<strong>in</strong>g the
recognition of orig<strong>in</strong>al documents (written records), orig<strong>in</strong>al primary source texts (pr<strong>in</strong>ted editions of
classical texts), and both modern and historical secondary scholarship, <strong>in</strong>clud<strong>in</strong>g commentaries and
textbooks, and reference works such as dictionaries and encyclopedias.
He also expla<strong>in</strong>ed that OCRopus uses a “strictly feed-forward system,” an important feature that
supports the plug-<strong>in</strong> of other layout-analysis and text-recognition modules. Other features <strong>in</strong>clude the
use of only a small number of data types to support reuse, “weighted f<strong>in</strong>ite state transducers” (WFSTs)
to represent the output of text l<strong>in</strong>e recognition, and f<strong>in</strong>al output <strong>in</strong> the hOCR format, which “encodes
OCR <strong>in</strong>formation <strong>in</strong> completely standards-compliant HTML files.” This open-source system can be
hosted through a web service, run from the command l<strong>in</strong>e or shell scripts, and users can customize how
it performs by script<strong>in</strong>g “the OCR eng<strong>in</strong>e <strong>in</strong> Lua.”
The basic stages <strong>in</strong> us<strong>in</strong>g OCRopus are image preprocess<strong>in</strong>g, layout analysis, text-l<strong>in</strong>e recognition, and
statistical language model<strong>in</strong>g. Each stage offers a variety of customization options that make it
particularly useful for historical languages. In terms of text-l<strong>in</strong>e recognition <strong>in</strong> historical texts, the fact
that OCRopus has both built-<strong>in</strong> text-l<strong>in</strong>e recognizers and the ability to add external text-l<strong>in</strong>e
recognizers for different scripts is very important, because as Breuel articulated:
Some historical texts may use different writ<strong>in</strong>g systems, s<strong>in</strong>ce Devanagari is not the only script
<strong>in</strong> historical use for Sanskrit. Scholarly writ<strong>in</strong>g on Sanskrit almost always uses Lat<strong>in</strong> script, and
Lat<strong>in</strong> script is also used for writ<strong>in</strong>g Sanskrit itself, <strong>in</strong>clud<strong>in</strong>g extended passages. Sanskrit
written <strong>in</strong> Devanagari and Lat<strong>in</strong> scripts also makes use of numerous diacritics that need to be