Foundations of Data Science

W 1 W 2 W 3 W 4 W 5 W 6
Figure 6.8: A deep learning fully connected network.
In fact the function
ReLU(x) =
{ x x ≥ 0
0 otherwise
where ∂ReLU(x)
∂x
=
{ 1 x ≥ 0
0 otherwise
seems to work well even though typically its derivative at x = 0 is undefined. An advantage
of ReLU over sigmoid is that ReLU does not saturate far from the origin.
Training a deep learning network of 7 or 8 levels using gradient descent can be computationally
expensive. 26 To address this issue one trains one level at a time on unlabeled
data using an idea called autoencoding. There are three levels, the input, a middle level
called the hidden level, and an output level as shown in Figure 6.9a. There are two sets
of weights. W 1 is the weights of the hidden level gates and W 2 is W1 T . Let x be the input
pattern and y be the output. The error is |x − y| 2 . One uses gradient descent to reduce
the error. Once the weights S 1 are determined they are frozen and a second hidden level
of gates is added as in Figure 6.9 b. In this network W 3 = W2
T and stochastic gradient
descent is again used this time to determine W 2 . In this way one level of weights is trained
at a time.
The output of the hidden gates is an encoding of the input. An image might be a
10 8 dimensional input and there may only be 10 5 hidden gates. However, the number of
images might be 10 7 so even though the dimension of the hidden layer is smaller than the
dimension of the input, the number of possible codes far exceeds the number of inputs
and thus the hidden layer is a compressed representation of the input.. If the hidden layer
were the same dimension as the input layer one might get the identity mapping. This
does not happen for gradient descent starting with random weights.
26 In the image recognition community, researchers work with networks of 150 levels. The levels tend
to be convolution rather than fully connected.
224