Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
6.5: Demonstration 121
Common ground
I have emphasized the difference in philosophy behind arithmetic coding and
Lempel–Ziv coding. There is common ground between them, though: in principle,
one can design adaptive probabilistic models, and thence arithmetic
codes, that are ‘universal’, that is, models that will asymptotically compress
any source in some class to within some factor (preferably 1) of its entropy.
However, for practical purposes, I think such universal models can only be
constructed if the class of sources is severely restricted. A general purpose
compressor that can discover the probability distribution of any source would
be a general purpose artificial intelligence! A general purpose artificial intelligence
does not yet exist.
6.5 Demonstration
An interactive aid for exploring arithmetic coding, dasher.tcl, is available. 2
A demonstration arithmetic-coding software package written by Radford
Neal 3 consists of encoding and decoding modules to which the user adds a
module defining the probabilistic model. It should be emphasized that there
is no single general-purpose arithmetic-coding compressor; a new model has to
be written for each type of source. Radford Neal’s package includes a simple
adaptive model similar to the Bayesian model demonstrated in section 6.2.
The results using this Laplace model should be viewed as a basic benchmark
since it is the simplest possible probabilistic model – it simply assumes the
characters in the file come independently from a fixed ensemble. The counts
{Fi} of the symbols {ai} are rescaled and rounded as the file is read such that
all the counts lie between 1 and 256.
A state-of-the-art compressor for documents containing text and images,
DjVu, uses arithmetic coding. 4 It uses a carefully designed approximate arithmetic
coder for binary alphabets called the Z-coder (Bottou et al., 1998), which
is much faster than the arithmetic coding software described above. One of
the neat tricks the Z-coder uses is this: the adaptive model adapts only occasionally
(to save on computer time), with the decision about when to adapt
being pseudo-randomly controlled by whether the arithmetic encoder emitted
a bit.
The JBIG image compression standard for binary images uses arithmetic
coding with a context-dependent model, which adapts using a rule similar to
Laplace’s rule. PPM (Teahan, 1995) is a leading method for text compression,
and it uses arithmetic coding.
There are many Lempel–Ziv-based programs. gzip is based on a version
of Lempel–Ziv called ‘LZ77’ (Ziv and Lempel, 1977). compress is based on
‘LZW’ (Welch, 1984). In my experience the best is gzip, with compress being
inferior on most files.
bzip is a block-sorting file compressor, which makes use of a neat hack
called the Burrows–Wheeler transform (Burrows and Wheeler, 1994). This
method is not based on an explicit probabilistic model, and it only works well
for files larger than several thousand characters; but in practice it is a very
effective compressor for files in which the context of a character is a good
predictor for that character. 5
2
http://www.inference.phy.cam.ac.uk/mackay/itprnn/softwareI.html
3
ftp://ftp.cs.toronto.edu/pub/radford/www/ac.software.html
4
http://www.djvuzone.org/
5
There is a lot of information about the Burrows–Wheeler transform on the net.
http://dogma.net/DataCompression/BWT.shtml