PDFlib Text Extraction Toolkit (TET) Manual

1 Introduction
The PDFlib Text Extraction Toolkit (TET) is targeted at extracting text and images from
PDF documents, but can also be used to retrieve other information from PDF. TET can be
used as a base component for realizing the following tasks:
> search the text contents of PDF
> create a list of all words contained in a PDF (concordance)
> implement a search engine for processing large numbers of PDF files
> extract text from PDF to store, translate, or otherwise repurpose it
> convert the text contents of PDF to other formats
> process or enhance PDFs based on their contents
> compare the text contents of multiple PDF documents
> extract the raster images from PDF for repurposing
> extract metadata and other information from PDF
TET has been designed for standalone use, and does not require any third-party software.
It is robust and suitable for multi-threaded server use.
1.1 Overview of TET Features
Supported PDF input. TET has been tested against thousands of PDF test files from various
sources. It accepts PDF 1.0 up to PDF 1.7 extension level 3 (corresponding to
Acrobat 1-9) as well as encrypted documents.
Unicode support. TET includes a considerable number of algorithms and data to
achieve reliable Unicode mappings for all text. Although text in PDF documents is not
usually encoded in Unicode, TET will normalize the text from a PDF document to Unicode:
> TET converts all text contents to Unicode. In C the text will be returned in UTF-8 or
UTF-16 format; in other language bindings as native Unicode strings.
> Ligatures and other multi-character glyphs will be decomposed into a sequence of
their constituent Unicode characters.
> Vendor-specific Unicode values (Corporate Use Subarea, CUS) are identified, and will
be mapped to characters with precisely defined meanings if possible.
> Glyphs which are lacking Unicode mapping information are identified as such, and
will be mapped to a configurable replacement character.
> UTF-16 surrogate pairs for characters outside the Basic Multilingual Plane (BMP) are
properly interpreted and maintained. Surrogate pairs and UTF-32 values can be retrieved
in all language bindings.
Some PDF documents do not contain enough information for reliable Unicode mapping.
In order to successfully extract the text nevertheless TET offers various configuration
options which can be used to supply auxiliary information for proper Unicode
mappings. In order to facilitate writing the required mapping tables we make available
PDFlib FontReporter, a free plugin for Adobe Acrobat. This plugin can be used for analyzing
fonts, encodings, and glyphs in PDF.
1.1 Overview of TET Features 11