Algorithms and Data Structures for External Memory

15.2 External Memory Compressed Data Structures 135
Eltabakh et al. [146] consider sets of variable-length strings encoded
using run-length coding in order to exploit space savings when there
are repeated characters. They adapt string B-trees [157, 158] (see Section
14.2) with the EM priority search data structure for three-sided
range searching [50] (see Section 12.4) to develop a dynamic compressed
data structure for the run-length encoded data. The data structure supports
substring matching, prefix matching, and range search queries.
The data structure uses O( ˆ N/B) blocks of space, where ˆ N is the total
length of the compressed strings. Insertions and deletions of t runlength
encoded suffixes take O � tlog B( ˆ N + t) � I/Os. Queries for a pattern
P can be performed in O � log B ˆ N + � | ˆ P | + Z � /B � I/Os, where | ˆ P |
is the size of the search pattern in run-length encoded format.
One of the major advances in text indexing in the last decade has
been the development of entropy-compressed data structures. Until
fairly recently, the best general-purpose data structures for pattern
matching queries were the suffix tree and its more space-efficient version,
the suffix array, which we studied in Section 14.3. However, the
suffix array requires several times more space than the text being
indexed. The basic reason is that suffix arrays store an array of pointers,
each requiring at least logN bits, whereas the original text being
indexed consists of N characters, each of size log |Σ| bits. For a terabyte
of ascii text (i.e., N =2 40 ), each text character occupies 8 bits.
The suffix array, on the other hand, consists of N pointers to the text,
each pointer requiring logN = 40 bits, which is five times larger. 1
For reasonable values of h, the compressed suffix array of Grossi et
al. [185] requires an amount of space in bits per character only as large
as the hth-order entropy Hh(T ) of the original text, plus lower-order
terms. In addition, the compressed suffix array is self-indexing, in that
it encodes the original text and provides random access to the characters
of the original text. Therefore, the original text does not need to be
stored, and we can delete it. The net result is a big improvement over
conventional suffix arrays: Rather than having to store both the original
1 Imagine going to a library and finding that the card catalog is stored in an annex that is
five times larger than the main library! Suffix arrays support general pattern matching,
which card catalogs do not, but it is still counterintuitive for an index to require so much
more space than the text it is indexing.