Foundations of Data Science

Thus
∂J
∂w j
= ∑ i
(
σ(w T x)(1 − σ(w T x))
l i x
σ(w T j − (1 − l i ) (1 − )
σ(wT x))σ(w T x)
x
x)
1 − σ(w T j
x)
)
(l i (1 − σ(w T x))x j − (1 − l i )σ(w T x)x j
= ∑ i
((l i x j − l i σ(w T x)x j − σ(w T x)x j + l i σ(w T x)x j
)
= ∑ i
= ∑ i
(
)
l i − σ(w T x) x j .
Softmax is a generalization of logistic regression to multiple classes. Thus, the labels
l i take on values {1, 2, . . . , k}. For an input x, softmax estimates the probability of each
label. The hypothesis is of the form
⎡
h w (x) = ⎢
⎣
Prob(l = 1|x, w 1 )
Prob(l = 2|x, w 2 )
.
Prob(l = k|x, w k )
where the matrix formed by the weight vectors is
⎡ ⎤
w 1
w 2
W = ⎢ ⎥
⎣ . ⎦
w k
⎤
⎡
⎥
⎦ = 1
∑ k
⎢
i=1 ewT i x ⎣
W is a matrix since for each label l i , there is a vector w i of weights.
and
Consider a set of n inputs {x 1 , x 2 , . . . , x n }. Define
{ 1 if l = k
δ(l = k) =
0 otherwise
J(W ) =
n∑
i=1
k∑
e wT j x i
δ(l i = j) log ∑ k
h=1 ewT h x i
The derivative of the cost function with respect to the weights is
∇ wi J(W ) = −
j=1
⎤
e wT 1 x
e wT 2 x
⎥
. ⎦
e wT k x
n∑ (
x j δ(lj = k) − Prob(l j = k)|x j , W )
j=1
226