PDFlib Text Extraction Toolkit (TET) Manual

tracts text and metadata from the PDF document and appends it to the optional usersupplied
comment which accompanies the uploaded document. The text is hidden in
an HTML comment so that it will not be visible to users when they view the document
comment. Since MediaWiki indexes the full contents of the comment (including the
hidden full text) the text contents of the PDF will also be indexed. The text for the index
is constructed as follows:
> The TET connector feeds the value of all document info fields to the index.
> The text contents of all pages are extracted and concatenated.
> If the size of the extracted text is below a limit, it will completely be fed to the index.
The advantage of this method is that search results will display the search term in
context.
> If the size of the extracted text exceeds a limit, the text is reduced to unique words
(i.e. multiple instances of the same word are reduced to a single instance of the
word).
> If the size of the reduced text is below a limit, it will be fed to the index. Otherwise it
will be truncated, i.e. some text towards the end of the document will not be indexed.
The predefined limit is 512 KB, but this can be changed in PDFIndexer.php. If one of the
size tests described above hits the limit, a warning message will be written to Media-
Wiki’s DebugLogFile if MediaWiki logging is activated.
Searching for PDF documents. Since PDF documents are treated as images by Media-
Wiki you must search them in the Image namespace. This can be achieved by activating
the Image checkbox in the list of namespaces in the Advanced search dialog (see Figure
4.2). The Image namespace will not be searched by default. However, this setting can be
enabled in the LocalSettings.php preferences file as follows:
$wgNamespacesToBeSearchedDefault = array(
NS_MAIN
=> true,
NS_IMAGE
=> true,
}
Fig. 4.2 Searching PDF documents in MediaWiki
The search results will display a list of documents which contain the search term. If the
full text has been indexed (as opposed to the abbreviated word list for long documents)
some additional terms will be displayed before and after the search term to provide
context. Since the PDF text contents are fed to the MediaWiki index in HTML form, line
numbers will be displayed in front of the text. These line numbers are not relevant for
PDF documents, and you can safely ignore them.
4.6 TET Connector for MediaWiki 47