PDFlib Text Extraction Toolkit (TET) Manual

5.3 Recommendations for common Scenarios
TET offers a variety of options which you can use to control various aspects of operation.
In this section we provide some recommendations for typical TET application scenarios.
Please refer to Chapter 10, »TET Library API Reference«, page 121, for details on the
functions and options mentioned below.
Optimizing performance. In some situations, particularly when indexing PDF for
search engines, text extraction speed is crucial, and may play a more important role
than optimal output. The default settings of TET have been selected to achieve the best
possible output, but can be adjusted to speed up processing. Some tips for choosing options
in TET_open_page( ) to maximize text extraction throughput:
> docstyle=searchengine
Several internal parameters will be set to speed up operation by reducing the output
quality in a way which does not affect the indexing process for search engines.
> skipengines={image}
If image extraction is not required internal image processing can be skipped in order
to speed up operation.
> contentanalysis={merge=0}
This will disable the expensive strip and zone merging step, and reduces processing
times for typical files to ca. 60% compared to default settings. However, documents
where the contents are scattered across the pages in arbitrary order may result in
some text which is not extracted in logical order.
> contentanalysis={shadowdetect=false}
This will disable detection of redundant shadow and fake bold text, which can also
reduce processing times.
Words vs. line layout vs. reflowable text. Different applications will prefer different
kinds of output (hyphenated words will always be dehyphenated with these settings):
> Individual words (ignore layout): a search engine may not be interested in any layout-related
aspects, but only the words comprising the text. In this situation use
granularity=word in TET_open_page( ) to retrieve one word per call to TET_get_text( ).
> Keep line layout: use granularity=page in TET_open_page( ) for extracting the full text
contents of a page in a single call to TET_get_text( ). Text lines will be separated with a
linefeed character to retain the existing line structure.
> Reflowable text: in order to avoid line breaks and facilitate reflowing of the extracted
text use contentanalysis={lineseparator=U+0020} and granularity=page in TET_open_
page( ). The full page contents can be fetched with a single call to TET_get_text( ).
Zones will be separated with a linefeed character, and a space character will be inserted
between the lines in a zone.
Writing a search engine or indexer. Indexers are usually not interested in the position
of text on the page (unless they provide search term highlighting). In many cases they
will tolerate errors which occur in Unicode mapping, and process whatever text contents
they can get. Recommendations:
> Use granularity=word in TET_open_page( ).
> If the application knows how to process punctuation characters you can keep them
with the adjacent text by setting the following page option:
contentanalysis={punctuationbreaks=false}
54 Chapter 5: Configuration