Foundations of Data Science

Figure 10.3: Some subgradients for a function that is not everywhere differentiable.
Subgradients are defined as follows. A subgradient of a function g at a point x 0 , is a
vector ∇g(x 0 ) satisfying g (x 0 + ∆x) ≥ g (x 0 ) + (∇g) T ∆x for any vector ∆x . A point is
a minimum for a convex function if there is a subgradient at that point with slope zero.
Consider the function ‖x‖ 1
, where x is a real variable. For x < 0, the subgradient
equals the gradient and has value -1. For x > 0, the subgradient equals the gradient and
has value 1. At x = 0, the subgradient can be any value in the range [-1,1]. The following
proposition generalizes this example to the 1-norm function in d-space.
Proposition 10.2 A vector v is a subgradient of the 1-norm function ||x|| 1 at x if and
only if it satisfies the three conditions below:
1. v i = −1 for all i in I 1 where, I 1 = {i|x i < 0},
2. v i = 1 for all i in I 2 where, I 2 = {i|x i > 0}, and
3. v i in [−1, 1] for all i in I 3 where, I 3 = {i|x i = 0}.
Proof: It is easy to see that for any vector y,
||x + y|| 1 − ||x|| 1 ≥ − ∑ i∈I 1
y i + ∑ i∈I 2
y i + ∑ i∈I 3
|y i |.
If i is in I 1 , x i is negative. If y i is also negative, then ||x i + y i || 1 = ||x i || 1 + ||y i || 1
and thus ||x i + y i || 1 − ||x i || 1 = ||y i || 1 = −y i . If y i is positive and less than ||x i || 1 , then
||x i +y i || 1 = ||x i ||−y i and thus ||x i +y i || 1 −||x i || = −y i . If y i is positive and greater than or
equal to ||x i || 1 , then ||x i +y i || 1 = y i −||x i || 1 and thus ||x i +y i || 1 −||x i || 1 = y i −2||x i || 1 ≥ −y i .
Similar reasoning establishes the case for i in I 2 or I 3 .
If v satisfies the conditions in the proposition, then ||x+y|| 1 ≥ ||x|| 1 +v T y as required.
Now for the converse, suppose that v is a subgradient. Consider a vector y that is zero
in all components except the first and y 1 is nonzero with y 1 = ±ε for a small ε > 0. If
1 ∈ I 1 , then ||x + y|| 1 − ||x|| 1 = −y 1 which implies that −y 1 ≥ v 1 y 1 . Choosing y 1 = ε,
335