PDFlib Text Extraction Toolkit (TET) Manual

Separator characters are inserted between multiple words, lines, or zones if the chosen
granularity is larger than the respective unit. For example, with granularity=word there’s
no need to insert separator characters since each call to TET_get_text( ) will return exactly
one word.
The separator characters can be specified with the wordseparator, lineseparator options
of TET_open_page( ) (use U+0000 to disable a separator), for example:
lineseparator=U+000A
By default, all content processing operations are disabled for granularity=glyph, and enabled
for all other granularity settings. However, more fine-grain control is possible via
separate options (see below).
Word boundary detection for Western text. The Wordfinder, which is enabled for all
granularity modes except glyph, creates logical words from multiple glyphs which may
be scattered all over the page in no particular order. Word boundaries for Western text
are identified by two criteria:
> A sophisticated algorithm analyzes the geometric relationship among glyphs to find
character groups which together form a word. The algorithm takes into account a variety
of properties and special cases in order to accurately identify words even in
complicated layouts and for arbitrary text ordering on the page.
> Some characters, such as space and punctuation characters (e.g. colon, comma, full
stop, parentheses) are considered a word boundary, regardless of their width and position.
If the punctuationbreaks option in TET_open_page( ) is set to false, the Wordfinder
will no longer treat punctuation characters as word boundaries:
contentanalysis={punctuationbreaks=false}
Ignoring punctuation characters for word boundary detection can, for example, be useful
for maintaining Web URLs where period and slash characters are usually considered
part of a word (see Figure 6.5).
Note Word boundary detection for text with ideographic characters works differently; see Section
6.3.2, »Word Boundaries for CJK Text«, page 79, for more information.
Fig. 6.5
The default setting punctuationbreaks=true
will separate the parts of URLs (top), while
punctuationbreaks=false will keep the parts
together (bottom).
6.5 Content Analysis 85