Boyd Convex Optimization book - SFU Wiki

304 6 Approximation and fitting
point in the affine set with minimum distance to 0, i.e., it determines the projection
of the point 0 on the affine set {x | Ax = b}.
Least-squares solution of linear equations
The most common least-norm problem involves the Euclidean or l 2 -norm.
squaring the objective we obtain the equivalent problem
By
minimize ‖x‖ 2 2
subject to Ax = b,
the unique solution of which is called the least-squares solution of the equations
Ax = b. Like the least-squares approximation problem, this problem can be solved
analytically. Introducing the dual variable ν ∈ R m , the optimality conditions are
2x ⋆ + A T ν ⋆ = 0, Ax ⋆ = b,
which is a pair of linear equations, and readily solved. From the first equation
we obtain x ⋆ = −(1/2)A T ν ⋆ ; substituting this into the second equation we obtain
−(1/2)AA T ν ⋆ = b, and conclude
ν ⋆ = −2(AA T ) −1 b, x ⋆ = A T (AA T ) −1 b.
(Since rank A = m < n, the matrix AA T is invertible.)
Least-penalty problems
A useful variation on the least-norm problem (6.5) is the least-penalty problem
minimize φ(x 1 ) + · · · + φ(x n )
subject to Ax = b,
(6.6)
where φ : R → R is convex, nonnegative, and satisfies φ(0) = 0. The penalty
function value φ(u) quantifies our dislike of a component of x having value u;
the least-penalty problem then finds x that has least total penalty, subject to the
constraint Ax = b.
All of the discussion and interpretation of penalty functions in penalty function
approximation can be transposed to the least-penalty problem, by substituting
the amplitude distribution of x (in the least-penalty problem) for the amplitude
distribution of the residual r (in the penalty approximation problem).
Sparse solutions via least l 1 -norm
Recall from the discussion on page 300 that l 1 -norm approximation gives relatively
large weight to small residuals, and therefore results in many optimal residuals
small, or even zero. A similar effect occurs in the least-norm context. The least
l 1 -norm problem,
minimize ‖x‖ 1
subject to Ax = b,
tends to produce a solution x with a large number of components equal to zero.
In other words, the least l 1 -norm problem tends to produce sparse solutions of
Ax = b, often with m nonzero components.