PDFlib Text Extraction Toolkit (TET) Manual

Geometry. The geometry features may be useful for some applications:
> The TET_get_char_info( ) interface is only required if you need the position of text on
the page, the respective font name, or other details. If you are not interested in text
coordinates calling TET_get_text( ) will be sufficient.
> If you have advance information about the layout of pages you can use the includebox
and/or excludebox options in TET_open_page( ) to get rid of headers, footers, or
similar items which are not part of the main text.
Unknown characters. If TET is unable to determine the appropriate Unicode mapping
for one or more characters it will represent it with the Unicode replacement character
U+FFFD. If your application is not concerned about unmappable characters you can
simply discard all occurrences of this character. Applications which require more finegrain
results could take the corresponding font into account, and use it to decide on
processing of unmappable characters. Use the following document option to replace all
unmapped characters with a question mark:
unknownchar=?
The special character U+0000 means »no character«. Use the following document option
to remove all unmapped characters from the output:
unknownchar=U+0000
Legal documents. When dealing with legal documents there is usually zero tolerance
for wrong Unicode mappings since they might alter the content or interpretation of a
document. In many cases the text position is not required, and the text must be extracted
word by word. Recommendations:
> Use the granularity=word option in TET_open_page( ).
> Use the password option with the appropriate document password in TET_open_
document( ) if you must process documents which require a password for opening, or
if content extraction is not allowed in the permission settings.
> For absolute text fidelity: stop processing as soon as the unknown field in the character
info structure returned by TET_get_char_info( ) is 1, or if the Unicode replacement
character U+FFFD is part of the string returned by TET_get_text( ). In TETML with one
of the text modes glyph or wordplus you can identify this situation by the following
attribute in the Glyph element:
unknown="true"
Do not set the unknownchar option to any common character since you may be unable
to distinguish it from correctly mapped characters without checking the
unknown field.
> Also to ensure text fidelity you may want to disable text extraction for text which is
not visible on the page:
ignoreinvisibletext=true
Processing documents with PDFlib+PDI. When using PDFlib+PDI to process PDF documents
on a per-page basis you can integrate TET for controlling the splitting or merging
process. For example, you could split a PDF document based on the contents of a page. If
you have control over the creation process you can insert separator pages with suitable
processing instructions in the text. The TET Cookbook contains examples for analyzing
documents with TET and then processing them with PDFlib+PDI.
5.3 Recommendations for common Scenarios 55