a la physique de l'information - Lisa - Université d'Angers

Author's personal copy
F. Chapeau-Blondeau, D. Rousseau / Physica A 388 (2009) 3969–3984 3971
One seeks to estimate the probability density f (x) from the N data points xn of Eq. (1). For this purpose, a histogram model
is introduced for the unknown density f (x) under the common form of an approximation by a piecewise constant function.
This histogram model is denoted M and is defined as follows. The density f (x) is modeled by K constant plateaus of value
fk, for k = 1 to K , each of these plateaus being defined in the abscissa between xmin and xmax over a regular bin of width
δx = xmax − xmin
K
= x
, (3)
K
with xmin and xmax respectively the minimum and maximum values of the xn’s over the data set x of Eq. (1). Especially,
consistency of the probability density model imposes
K
fkδx = 1. (4)
k=1
The probability P(x) of Eq. (2), based on the histogram model M for the density f (x), is expressible as
P(x) = dx N
K
k=1
f Nk k , (5)
where Nk is the number of data points xn of the data set x that fall within bin number k, verifying K
k=1 Nk = N.
3. Maximum-likelihood histogram estimation
When the number of bins K is fixed, the density model M is specified by the K parameters fk for k = 1 to K . To determine
these parameters from the data, a standard approach is the maximum-likelihood method [39] which consists in selecting
those values of the parameters fk that maximize the probability P(x) in Eq. (5) of the observed data set x. Maximizing P(x)
of Eq. (5) under the constraint of Eq. (4) is achieved by the well-known maximum-likelihood solution
fk = Nk
, k = 1, . . . K. (6)
Nδx
The maximum-likelihood solution of Eq. (6) completely specifies, for the probability density f (x), the histogram model with
a fixed number K of regular bins.
4. Minimum description length
Another point of view can be adopted to arrive at the solution of Eq. (6). Information theory stipulates that to code data
xn appearing with probability P(xn), the optimal code assigns a codeword with length − log P(xn). To code the whole data
set x of Eq. (1), the optimal code assigns a length − log P(x), which by the probability model of Eq. (5) is
Ldata = − log P(x) = − log(dx N ) −
K
Nk log(fk). (7)
k=1
The maximum-likelihood solution of Eq. (6) maximizes the likelihood P(x) of Eq. (5) and equivalently the loglikelihood
log P(x). Therefore, the solution of Eq. (6) also minimizes the code length Ldata = − log P(x) of Eq. (7). The solution of Eq. (6)
selects from the data, the K parameters fk of the probability density model M, so that the optimal code designed for the
data from this density model, achieves the minimal code length. This is the rationale of the MDL principle: to select the
parameters of the model that allow the shortest coding of the complete data. This guarantees that the selected model is the
best (within its class) at capturing the structures and regularities in the data.
We can add here, that the minimum of the description length (7) achieved by the solution of Eq. (6) can be expressed as
x
Lmin = NH({pk}) − N log(K) + N log , (8)
dx
where we have introduced the entropy
H({pk}) = −
K
pk log(pk) (9)
k=1
of the empirical probabilitiespk = fkδx = Nk/N deduced from Eq. (6).
Here, when the number of bins K of the histogram model is fixed in an a priori way, the MDL solution coincides with the
maximum-likelihood solution of Eq. (6). However, the MDL principle can be extended to also optimally select the number
of bins K of the model from the data, along with the K parameter values fk for k = 1 to K . This extension proceeds in the
152/197