PDFlib Text Extraction Toolkit (TET) Manual

6.5 Content Analysis
PDF documents provide the semantics (Unicode mapping) of individual text characters
as well as their position on the page. However, they generally do not convey information
about words, lines, columns or other high-level text units. The fragments comprising
text on a page may contain individual characters, syllables, words, lines, or an arbitrary
mixture thereof, without any explicit marks designating the start or end of a
word, line, or column.
To make matters worse, the ordering of text fragments on the page may be different
from the logical (reading) order. There are no rules for the order in which portions of
text are placed on the page. For example, a page containing two columns of text could
be produced by creating the first line in the left column, followed by the first line of the
right column, the second line of the left column, the second line of the right column etc.
However, logical order requires all text in the left column to be processed before the
text in the right column is processed. Extracting text from such documents by simply
replaying the instructions on the PDF page generally provides undesirable results since
the logical structure of the text is lost.
TET’s content analysis engine analyzes the contents, position, and relationship of
text fragments in order to achieve the following goals:
> create words from characters, and insert separator characters between words if desired
> remove redundant text, such as duplicates which are only present to create a shadow
effect
> recombine the parts of hyphenated words which span more than one line
> identify text columns (zones)
> sort text fragments within a zone, as well as zones within a page
These operations are discussed in more detail below, as well as options which provide
some control over content processing.
Text granularity. The granularity option of TET_open_page( ) specifies the amount of
text which is returned by a single call to TET_get_text( ):
> With granularity=glyph each fragment contains the result of mapping one glyph,
which may be more than one character (e.g. for ligatures). In this mode content analysis
is disabled. TET will return the original text fragments on the page in their original
order. Although this is the fastest mode, it is only useful if the TET client intends
to do sophisticated postprocessing (or is only interested in the text position, but not
in its logical structure) since the text may be scattered all over the page.
> With granularity=word the Wordfinder algorithm will group characters into logical
words. Each fragment contains a word. Isolated punctuation characters (comma, colon,
question mark, quotes, etc.) are returned as separate fragments by default, while
multiple sequential punctuation characters are grouped as a single word (e.g. a series
of period characters which simulates a dotted line). However, punctuation treatment
can be changed (see »Word boundary detection for Western text« below).
> With granularity=line the words identified by the Wordfinder are grouped into lines.
If dehyphenation is enabled (which is the default) the parts of hyphenated words at
the end of a line are combined, and the full dehyphenated word is part of the line.
> With granularity=page all words on the page are returned in a single fragment.
84 Chapter 6: Text Extraction