PDFlib Text Extraction Toolkit (TET) Manual

Geometry. The geometry features may be useful for some applications:
> The TET_get_char_info( ) interface is only required if you need the position of text on
the page, the respective font name, or other details. If you are not interested in text
coordinates calling TET_get_text( ) will be sufficient.
> If you have advance information about the layout of pages you can use the includebox
and/or excludebox options in TET_open_page( ) to get rid of headers, footers, or
similar items which are not part of the main text.
Unknown characters. If TET is unable to determine the appropriate Unicode mapping
for one or more characters it will represent it with the Unicode replacement character
U+FFFD. If your application is not concerned about unmappable characters you can
simply discard all occurrences of this character. Applications which require more finegrain
results could take the corresponding font into account, and use it to decide on
processing of unmappable characters. Use the following document option to replace all
unmapped characters with a question mark:
unknownchar=?
Use the following document option to remove all unmapped characters from the output:
fold={{[:Private_Use:] remove} {[U+FFFD] remove} default}
Complex layouts. Some classes of documents often use very elaborate page layouts.
For example, with magazines and periodicals TET may not be able to properly determine
the relationship of columns on the page. In such situations it is possible to enhance
the extracted text at the expense of processing time. Suitable options for this
purpose are summarized in Section 6.6, »Layout Analysis«, page 88. See Table 10.12, page
174, for more details on relevant options.
Legal documents. When dealing with legal documents there is usually zero tolerance
for wrong Unicode mappings since they might alter the content or interpretation of a
document. In many cases the text position is not required, and the text must be extracted
word by word. Recommendations:
> Use the granularity=word option in TET_open_page( ).
> Use the password option with the appropriate document password in TET_open_
document( ) if you must process documents which require a password for opening, or
the shrug option if content extraction is not allowed in the permission settings and
you are in a legal position to extract text from the document (see »The »shrug« feature
for protected documents«, page 60).
> For absolute text fidelity: stop processing as soon as the unknown field in the character
info structure returned by TET_get_char_info( ) is 1, or if the Unicode replacement
character U+FFFD is part of the string returned by TET_get_text( ). In TETML with one
of the text modes glyph or wordplus you can identify this situation by the following
attribute in the Glyph element:
unknown="true"
Do not set the unknownchar option to any common character since you may be unable
to distinguish it from correctly mapped characters without checking the
unknown field.
66 Chapter 5: Configuration