¡¡¡¡£¡¡¢£¡¡¡¡¡¡¡¡¡££¡¢¡¡1¡ ¡¡28 Jonathan Richard Shewchuk0¡0¡0¡ 10¡ 11¡1¡1¡1¡1¡1¡1¡0¡1¡0¡(a)(b)0¡0¡ 10¡ 10¡1¡1¡1¡0¡ 20¡ 21¡2¡1¡¡¡2¡0¡0¡(d)(c)Figure 27: Optimality of <strong>the</strong> <strong>Method</strong> of <strong>Conjugate</strong> Directions. (a) A two-dimensional problem. Lines thatappear perpendicular are orthogonal. (b) The same problem in a “stretched” space. Lines that appearperpendicular are § -orthogonal. (c) A three-dimensional problem. Two concentric ellipsoids are shown; £¡is at <strong>the</strong> center of both. The 0£ linetangent <strong>to</strong> <strong>the</strong> inner ellipsoid £ ¡at £¡¡1 is tangent <strong>to</strong> <strong>the</strong> outer 1£ ellipsoid at . The 0£ plane £ £2£ . (d) Stretched view of <strong>the</strong> three-dimensional problem.2 is
¡¡¡¡¡£¡¡ ¡¥¡¦0¡ 1¡ 1¡0¡ ¡1¡ ¡1¡ 2¡ 0¡ ¡2¡ ¡1¡ ¢§ ¡0¡1¡ 1¡1¡The <strong>Method</strong> of <strong>Conjugate</strong> Directions 29In Figure 27(b), and appear perpendicular because <strong>the</strong>y are -orthogonal. It is clear thatmust point <strong>to</strong> <strong>the</strong> solution , because is tangent at <strong>to</strong> a circle whose center is . However, athree-dimensional example is more revealing. Figures 27(c) and 27(d) each show two concentric ellipsoids.The point lies on <strong>the</strong> outer ellipsoid, and lies on <strong>the</strong> inner ellipsoid. Look carefully at <strong>the</strong>se figures:<strong>the</strong> plane 2 slices through <strong>the</strong> larger ellipsoid, and is tangent <strong>to</strong> <strong>the</strong> smaller ellipsoid at . Thepoint is at <strong>the</strong> center of <strong>the</strong> ellipsoids, underneath <strong>the</strong> plane.Looking at Figure 27(c), we can rephrase our question. Suppose you and I are standing at , andwant <strong>to</strong> walk <strong>to</strong> <strong>the</strong> point that minimizes on 2; but we are constrained <strong>to</strong> walk along <strong>the</strong> searchdirection . If points <strong>to</strong> <strong>the</strong> minimum point, we will succeed. Is <strong>the</strong>re any reason <strong>to</strong> expect thatwill point <strong>the</strong> right way?Figure 27(d) supplies an answer. Because 0¡ is -orthogonal <strong>to</strong> , <strong>the</strong>y are perpendicular in this1¡diagram. Now, suppose you were staring down 0¡ at <strong>the</strong> plane 2 as if it were a sheet of paper; <strong>the</strong>sight you’d see would be identical <strong>to</strong> Figure 27(b). 2¡ The point would be at <strong>the</strong> center of <strong>the</strong> paper,and ¡<strong>the</strong> point would lie underneath <strong>the</strong> paper, directly under <strong>the</strong> 2¡ point . 0¡ Because 1¡ and are¡perpendicular, points directly <strong>to</strong> 2¡ , which is <strong>the</strong> point in ¡0¡ 2 closest <strong>to</strong> . The plane ¡ ¡0¡ 21¡is tangent <strong>to</strong> <strong>the</strong> sphere on ¡ ¡which lies. If you <strong>to</strong>ok a third step, it would be straight down from 2¡ <strong>to</strong>2¡, in a direction -orthogonal <strong>to</strong> ¡2.¡¡<strong>An</strong>o<strong>the</strong>r way <strong>to</strong> understand what is happening in Figure 27(d) is <strong>to</strong> imagine yourself standing at <strong>the</strong>solution point , pulling a string connected <strong>to</strong> a bead that is constrained <strong>to</strong> lie in ¡0¡ . Each time <strong>the</strong>expanding subspace is enlarged by a dimension, <strong>the</strong> bead becomes free <strong>to</strong> move a little closer <strong>to</strong> you.Now if you stretch <strong>the</strong> space so it looks like Figure 27(c), you have <strong>the</strong> <strong>Method</strong> of <strong>Conjugate</strong> Directions.¡<strong>An</strong>o<strong>the</strong>r important property of <strong>Conjugate</strong> Directions is visible in <strong>the</strong>se illustrations. We have seen that,at each step, <strong>the</strong> hyperplane 0¡ is tangent <strong>to</strong> <strong>the</strong> ellipsoid on which ¡ ¡ lies. Recall from Section 4that <strong>the</strong> residual at any point is orthogonal <strong>to</strong> <strong>the</strong> ellipsoidal surface at that point. It follows that £¡ isorthogonal <strong>to</strong> as well. To show this fact ma<strong>the</strong>matically, premultiply Equation 35 by ¡ ¡ :¡¡ ¢ ¦¡ ¢ ¦¡¤ 1 ¥ £¡ (38)¦ 0 ¥ £¢(by -orthogonality of -vec<strong>to</strong>rs). (39)We could have derived this identity by ano<strong>the</strong>r tack. Recall that once we take a step in a search direction,we need never step in that direction again; <strong>the</strong> error term is evermore -orthogonal <strong>to</strong> all <strong>the</strong> old searchdirections. £¡ Because , <strong>the</strong> residual is evermore orthogonal <strong>to</strong> all <strong>the</strong> old search directions.¡ £ ¦£ ¡¢isBecause <strong>the</strong> search directions are constructed from <strong>the</strong> vec<strong>to</strong>rs, <strong>the</strong> subspace spannedby 0 1, and <strong>the</strong> £residual is orthogonal <strong>to</strong> <strong>the</strong>se previous vec<strong>to</strong>rs as well (see Figure 28). This is ¡ provenby taking <strong>the</strong> inner product of Equation £ ¦¡ 36 and : £¦¡0 £ £ ¦£ £¦ 1¡ ¡ £¦0¡ (40)¥ £¢(by Equation 39) (41)¡There is one more identity we will use later. From Equation 40 (and Figure 28), £¡ £ £(42)¡
Change language