PDFlib Text Extraction Toolkit (TET) Manual

Options for text filtering. There are several situations where TET will modify the actual
character values found on the page in order to make the results more useful. Most of
these steps can be controlled via options. The following list gives an overview of all operations
which may modify the text:
> Dehyphenation will remove hyphen characters and combine the parts of a hyphenated
word. This can be disabled with the dehyphenate suboption of the
contentanalysis option for TET_open_page( ).
> Redundant text which creates only visual artifacts such as shadow effects or artificial
bold text will be removed. This can be disabled with the shadowdetect suboption of
the contentanalysis option for TET_open_page( ).
> Very small or very large text can be ignored. The limits can be controlled with the
fontsizerange option of TET_open_page( ).
> Unicode post-processing will replace certain Unicode characters with more familiar
ones. For example, Latin ligatures will be replaced with their constituent characters,
and fullwidth ASCII variants in CJK fonts will be replaced with the corresponding
non-fullwidth characters. For details see Table 6.2, page 68.
> Invisible text (text with textrendering=3) will be extracted by default, but this can be
changed with the ignoreinvisibletext option of TET_open_page( ).
> Glyphs which cannot be mapped to Unicode will be replaced with the Unicode character
defined in the unknownchar option of TET_open_document( ). See section »Unmappable
glyphs«, page 69.
Characters outside the BMP and surrogate handling. Characters outside Unicode’s Basic
Multilingual Plane (BMP), i.e. those with Unicode values above 0xFFFF, cannot be expressed
as a single UTF-16 value, but require a pair of UTF-16 values called a surrogate
pair. Examples of characters outside the BMP include certain mathematical and musical
symbols at U+1DXXX as well as thousands of CJK extension characters starting at
U+20000.
TET interprets and maintains surrogates, and allows access to the corresponding
UTF-32 value even in programming languages where native Unicode strings support
only UTF-16. Leading (high) surrogates and trailing (low) surrogates will be maintained.
The string returned by TET_get_text( ) will return two UTF-16 values where the leading
value will be treated as a real character which carries the glyph properties (size, font,
etc.). The trailing value will be treated as an artificial character without any corresponding
glyph.
The uv field returned by TET_get_char_info( ) for the leading surrogate value will contain
the corresponding UTF-32 value. This allows direct access to the UTF-32 value of a
non-BMP character even if you are working in an UTF-16 environment without any support
for UTF-32. The uv field for the trailing surrogate value will be 0.
Unmappable glyphs. There are several reasons why text in a PDF cannot reliably be
mapped to Unicode. If the document does not contain a ToUnicode CMap with Unicode
mapping information for the codes used on the page, Unicode values can be missing for
various reasons:
> Type 1 fonts may contain unknown glyph names, and TrueType, OpenType, or CID
fonts may be addressed with glyph ids without any Unicode values in the font or
PDF. By default, TET will assign the Unicode Replacement Character U+FFFD to these
characters. You can select a different replacement character (e.g. the space character
6.5 Unicode Pipeline 69