PDFlib Text Extraction Toolkit (TET) Manual

Dehyphenation. Hyphenated
words at the end of a line are usually
not desired for applications
which process the extracted text
on a logical level. TET will therefore
dehyphenate, or recombine
the parts of a hyphenated word.
More precisely, if a word at the
end of a line ends with a hyphen
character and the first word on
the next line starts with a lowercase
character, the hyphen will be
removed and the first part of the
word will be combined with the part on the next line, provided there is at least one
more line in the same zone. Dash characters (as opposed to hyphens) will be left unmodified.
The parts of a hyphenated word will not be modified, only the hyphen will be
removed. Dehyphenation can be disabled with the following option list for TET_open_
page( ):
contentanalysis={dehyphenate=false}
Shadow and fake bold text removal. PDF documents sometimes include redundant
text which does not contribute to the semantics of a page, but creates certain visual effects
only. Shadow text effects are usually achieved by placing two or more copies of the
actual text on top of each other, where a small displacement is applied. Applying
opaque coloring to each layer of text provides a visual appearance where the majority
of the text in lower layers is obscured, while the visible portions create a shadow effect.
Similarly,
word processing
applications
sometimes support
a feature for
creating artificial bold text. In order to create bold text appearance even if a bold font is
not available, the text is placed repeatedly on the page in the same color. Using a very
small displacement the appearance of bold text is simulated.
Shadow simulation, artificial bold text, and similar visual artifacts create severe
problems when reusing the extracted text since redundant text contents which contribute
only to the visual appearance will be processed although the text does not contribute
to the page contents.
If the wordfinder is enabled, TET will identify and remove such redundant visual artifacts
by default. Shadow removal can be disabled with the following option list for
TET_open_page( ):
contentanalysis={shadowdetect=false}
6.6 Content Analysis 73