v2007.09.17 - Convex Optimization

584 APPENDIX E. PROJECTIONFor matrix A of arbitrary rank and shape, on the other hand, A T A mightnot be invertible. Yet the normal equation can always be solved exactly by:(1638)x ⋆ = lim A + t I) −1 A T b = A † b (1651)t→0 +(ATinvertible for any positive value of t by (1255). The exact inversion (1650)and this pseudoinverse solution (1651) each solvelim minimizet→0 + x‖Ax − b‖ 2 + t ‖x‖ 2 (1652)simultaneously providing least squares solution to (1648) and the classicalleast norm solution E.1 [249, App.A.4] (conferE.5.0.0.5)arg minimize ‖x‖ 2xsubject to Ax = AA † bwhere AA † b is the orthogonal projection of vector b on R(A).(1653)E.1 Idempotent matricesProjection matrices are square and defined by idempotence, P 2 =P ;[249,2.6] [152,1.3] equivalent to the condition, P be diagonalizable[150,3.3, prob.3] with eigenvalues φ i ∈ {0, 1}. [301,4.1, thm.4.1]Idempotent matrices are not necessarily symmetric. The transpose of anidempotent matrix remains idempotent; P T P T = P T . Solely exceptingP = I , all projection matrices are neither orthogonal (B.5) or invertible.[249,3.4] The collection of all projection matrices of particular dimensiondoes not form a convex set.Suppose we wish to project nonorthogonally (obliquely) on the rangeof any particular matrix A∈ R m×n . All idempotent matrices projectingnonorthogonally on R(A) may be expressed:P = A(A † + BZ T ) ∈ R m×m (1654)E.1 This means: optimal solutions of lesser norm than the so-called least norm solution(1653) can be obtained (at expense of approximation Ax ≈ b hence, of perpendicularity)by ignoring the limiting operation and introducing finite positive values of t into (1652).