Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
484 40 — Capacity of a Single Neuron
of its weights is a real number and so can convey an infinite number of bits’.
We exclude this answer by saying that the receiver is not able to examine the
weights directly, nor is the receiver allowed to probe the weights by observing
the output of the neuron for arbitrarily chosen inputs. We constrain the
receiver to observe the output of the neuron at the same fixed set of N points
{xn} that were in the training set. What matters now is how many different
distinguishable functions our neuron can produce, given that we can observe
the function only at these N points. How many different binary labellings of
N points can a linear threshold function produce? And how does this number
compare with the maximum possible number of binary labellings, 2 N ? If
nearly all of the 2 N labellings can be realized by our neuron, then it is a
communication channel that can convey all N bits (the target values {tn})
with small probability of error. We will identify the capacity of the neuron as
the maximum value that N can have such that the probability of error is very
small. [We are departing a little from the definition of capacity in Chapter 9.]
We thus examine the following scenario. The sender is given a neuron
with K inputs and a data set DN which is a labelling of N points. The
sender uses an adaptive algorithm to try to find a w that can reproduce this
labelling exactly. We will assume the algorithm finds such a w if it exists. The
receiver then evaluates the threshold function on the N input values. What
is the probability that all N bits are correctly reproduced? How large can N
become, for a given K, without this probability becoming substantially less
than one?
General position
One technical detail needs to be pinned down: what set of inputs {xn} are we
considering? Our answer might depend on this choice. We will assume that
the points are in general position.
Definition 40.1 A set of points {xn} in K-dimensional space are in general
position if any subset of size ≤ K is linearly independent, and no K + 1 of
them lie in a (K − 1)-dimensional plane.
In K = 3 dimensions, for example, a set of points are in general position if no
three points are colinear and no four points are coplanar. The intuitive idea is
that points in general position are like random points in the space, in terms of
the linear dependences between points. You don’t expect three random points
in three dimensions to lie on a straight line.
The linear threshold function
The neuron we will consider performs the function
where
y = f
f(a) =
K
k=1
wkxk
1 a > 0
0 a ≤ 0.
(40.1)
(40.2)
We will not have a bias w0; the capacity for a neuron with a bias can be
obtained by replacing K by K + 1 in the final result below, i.e., considering
one of the inputs to be fixed to 1. (These input points would not then be in
general position; the derivation still works.)