Foundations of Data Science

W 1 W 2
W 1 W 2 W 3
(a)
(b)
Figure 6.9: Autoencoder technique used to train one level at a time. In the Figure 6.9 (a)
train W 1 and W 2 . Then in Figure 6.9 (b), freeze W 1 and train W 2 and W 3 . In this way
one trains one set of weights at a time.
The output layer of a deep network typically uses a softmax procedure. Softmax is
a generalization of logistic regression where given a set of vectors {x 1 , x 2 , . . . x n } with
labels l 1 , l 2 , . . . l n , l i ∈ {0, 1} and with a weight vector w we define the probability that
the label l given x equals 0 or 1 by
and
where σ is the sigmoid function.
Prob(l = 1|x) =
1
1 + e −wT x = σ(wT x)
Prob(l = 0|x) = 1 − Prob(l = 1/x)
Define a cost function
J(w) = ∑ (
)
l i log(Prob(l = 1|x)) + (1 − l i ) log(1 − Prob(l = 1|x))
i
and compute w to minimize J(x). Then
J(w) = ∑ (
)
l i log(σ(w T x)) + (1 − l i ) log(1 − σ(w T x))
i
Since ∂σ(wT x)
∂w j
= σ(w T x)(1 − σ(w T x))x j , it follows that ∂ log(σ(wT x))
∂w j
= σ(wT x)(1−σ(w T x))x j
σ(w T x)
,
225