PDFlib Text Extraction Toolkit (TET) Manual

CJK support. TET includes full support for extracting Chinese, Japanese, and Korean
text:
> All predefined CJK CMaps (encodings) are recognized; CJK text will be converted to
Unicode. The CMap files for CJK encoding conversion are included in the TET distribution.
> Horizontal and vertical writing modes are supported.
> CJK font names will be normalized to Unicode.
Image extraction. TET extracts raster images from PDF. Adjacent parts of a segmented
image will be recombined to facilitate postprocessing and re-use (e.g. multi-strip images
created by some applications). Small images can be filtered in order to exclude tiny
image fragments from cluttering the output.
Images will be extracted in the common TIFF, JPEG, or JPEG 2000 formats.
Geometry. TET provides precise metrics for the text, such as the position on the page,
glyph widths, and text direction. Specific areas on the page can be excluded or included
in the text extraction process, e.g. to ignore headers and footers or margins.
For images the pixel size, physical size, and color space are available as well as position
and angle.
Word detection and content analysis. TET can be used to retrieve low-level glyph information,
but also includes advanced algorithms for high-level content analysis:
> Detect word boundaries to retrieve words instead of characters.
> Recombine the parts of hyphenated words (dehyphenation).
> Remove duplicate instances of text, e.g. shadow and fake bold text.
> Recombine paragraphs into reading order.
> Reorder text which is scattered over the page.
> Reconstruct lines of text.
> Recognize tabular structures on the page.
pCOS interface for simple access to PDF objects. TET includes pCOS (PDFlib Comprehensive
Object System) for retrieving arbitrary PDF objects. With pCOS you can retrieve
PDF metadata, interactive elements (e.g. bookmark text, contents of form fields), or any
other information from a PDF document with a simple query interface.
TET Markup Language (TETML). The information retrieved from a PDF document can
be presented in an XML format called TET Markup Language, or TETML for processing
with standard XML tools. TETML contains text, image, and metadata information and
can optionally also contain font- and geometry-related details.
What is text? While TET deals with a large class of PDF documents, not all visible text
can successfully be extracted. The text must be encoded using PDF’s text and encoding
facilities (i.e., it must be based on a font). Although the following flavors of text may be
visible on the page they cannot be extracted with TET:
> Rasterized (pixel image) text, e.g. scanned pages;
> Text which is directly represented by vector elements without any font.
Note that metadata and text in hypertext elements (such as bookmarks, form fields,
notes, or annotations) can be retrieved with the pCOS interface. On the other hand, TET
12 Chapter 1: Introduction