A generalized Newton algorithm using higher-order derivatives

JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS: Vol. 39, No. 1, JANUARY 1983 

A Generalized Newton Algorithm Using 

Higher-Order Derivatives 

R. KALABA 1 AND A. TISHLER 2 

Abstract. This paper presents a general optimization algorithm using 

first-order to rth order derivatives to find the optimum of an r- 

continuously differentiable function of many variables. This algorithm 

collapses to the Newton-Raphson algorithm when only first- and 

second-order derivatives are used. The computation of the required 

higher-order derivatives are readily available through the table 

algorithm. The generalized CES production function is used as an 

example. 

Key Words. Optimization algorithms, generalized Newton method, 

higher-order derivatives, table method. 

1. Introduction 

The choice of optimization algorithms for nonlinear functions is the 

subject of numerous studies in recent years [see Dennis and Mor6 (Ref. 

1), Dennis and Mei (Ref. 2), Goldfeld and Quandt (Ref. 3), Pierre (Ref. 

4), and Belsley (Ref. 5)]. Most of the efforts in that area were directed 

toward developing algorithms which use only first-order analytical deriva- 

tives. The main reason for that direction in research was the difficulty in 

obtaining the analytical expressions for higher-order derivatives (see Ref. 

5). Recently, Kalaba, Rasakhoo, and Tishler (Ref. 6) and Kalaba and 

Tishler (Ref. 7) developed and applied the table algorithm to calculate 

exact higher-order derivatives of functions with many variables without 

explicitly using analytical expressions for those derivatives [see also 

Wengert (Ref. 8) and Bellman, Kagiwada, and Kalaba (Ref. 9) on this 

issue]. The current available software of the table algorithm makes the 

calculation of derivatives up to order 3 a very simple task. Thus, there is 

Professor of Economics and Biomedical Engineering, Department of Economics, University 

of Southern California, Los Angeles, California. 

2 Visiting Associate Professor of Economics, Department of Economics, University of 

Southern California, Los Angeles, California; on leave from the Faculty of Management, 

Tel Aviv University, Tel Aviv, Israel. 

1 

0022-3239/83/0100-0001503.00/0 ~) 1983 Plenum l~aMishing Corporation

2 JOTA: VOL. 39, NO. 1, JANUARY 1983 

no technological reason to avoid the use of higher-order derivatives in 

optimization algorithms. It is well known that, in important applications, 

the Newton-Raphson method is superior, in terms of computer costs, to 

lower-order algorithms [see, for example, Belsley (Ref. 5) on nonlinear 

maximum-likelihood estimation in economics]. Thus, it is reasonable that 

an algorithm which uses third or higher-order derivatives could be superior 

to the Newton-Raphson and lower-order algorithms for various applica- 

tions. 

This paper presents a general optimization algorithm using first-order 

up to rth order derivatives to find the optimum of an r-continuously 

differentiable function of many variables. 

This algorithm collapses to the well-known Newton-Raphson 

algorithm when only first-order and second-order derivatives are used. 

Thus, we denote the algorithm a generalized Newton algorithm of order 

r, GN(r), when the highest derivatives to be used in the optimization process 

are of order r + 1. 

This paper does not intend to solve the question of what is the best 

choice of optimization algorithm. Rather, since higher-order derivatives 

are readily available through the table algorithm, we suggest a simple 

algorithm for optimization using higher-order derivatives. 

This paper proceeds as follows. Section 2 presents and proves the 

main theorem. In Section 3, we develop the optimization algorithm. The 

generalized CES production function is used to test the optimization 

algorithm with second-order and third-order derivatives in Section 4. In 

Section 5, we discuss and develop some extensions of the optimization 

algorithm. 

2. Main Theorem 

The optimization problem is given by 

min F(x), (1) 

xER t( 

where F(x) is r-continuously differentiable. 

A necessary condition for a vector x* to be a local minimum for F(x) 

is 

Y -- (Y~, Y2,. • •, y~c)' -- ~F(x*)/ax = 0. (2)

JOTA: VOL. 39, NO. 1, JANUARY 1983 3 

Let the inverse functions of (2), assumed to exist, in a neighborhood 

of x*, be 

xx = hl(yl, yz, . . . , yK), 

x~: = hK(yl, Y2 ..... yK), 

where Yl, i = 1 ..... K, stand for yl evaluated at x. 

Theorem 2.1. If hi, i = 1 ..... K, are polynomials of degree r, the 

optimal x for problem (1) is given by 

where 

(3) 

x* =x°+8, (4) 

o (x0,. xO), 

X ~ .,, 

is the initial value for x in a neighborhood of x*, and 8 includes the second 

up to the K + 1 elements in the first row of the matrix A -1 (if it exists) 

evaluated at x = x °, where 

A [(1);(yl), .,(Yr¢); 2 

= .. (yl), (YlY2) .... , (y2);... ; (Yl), (Y~-ly2) ..... (y~)] 

(5) 

and the expressions in the parentheses are the following column vectors: 

(z)=[z; Oz/Oxl, .. ., Oz/OxK,"2 • o z/oxl, ,~ 2 O2z/OxlOx2 ..... 

O2z/Ox2;... ; OrZ/oXr~, O~Z/OX~ -~" aXE ..... O~Z/OX~r]'. (6) 

Proof. We first prove the theorem for r = 2 and later show how to 

extend it to the general case. 

Write hi as a second-degree polynomial, that is, 

K K K 

xl=h1(y)=a~ + 2 a~yi+ 2 2 b#y~y 1 i. (7) 

i=1 i=1 j=i 

Next, differentiate (7) on both sides sequentially with respect to xl, 

x2,..., xK, to get 

K K K 

Ohl(y)/Oxl= 1= ~2 a~(OyffOxl)+ ~ E b~.[O(y,yj)/axl], 

i=l i=t ]=i 

K K K 

Ohl(y)/Ox2 = 0 = ~ a~(OyffOx~)+ ~ ~ b~.[O(y~yi)/Ox2], 

i=1 i=1 ]=i 

K K K 

Ohl(y)/Ox~z = O= ~. a~(OyffOxK,)+ ~ ~, b~[O(yiyi)/OxK]. 

i=l i=t ]=i 

(8)


There are K equations [first-order derivatives of hi(y) with respect to 

xl .... , XK] in (8). The second-order derivatives of (7) with respect to 

xl, x2 ..... xr are given by 

K K K 

O2h~(y)/Ox2=O = Y~ a~(O2yi/oxZ)+ ~ ~ b])[O2(y~yj)/Ox2], 

i=1 i=1 j=l 

K K K 

O2hl(Y)/OxlOx2 =0= 2 a~(O2YJOXlOX2) + E Z bii[O 1 2 (yiyi)/OXl 3X2], 

i=1 i=1 j=i 

K K K 

02hl(y)/Oxl Oxg = 0 = ~ a~(O2yl/Oxl OXK)+ ~ ~', b~[O2(yiyi)/Oxl OXK], 

i=1 i=1 j=i 

K K K 

2 2 

0 hl(y)/Ox2=O= E ali(O2Yi/Ox~)+ ~ ~ I z 2 

i=1 i=1 j=i 

b~j[o (y~y~)/Ox2], 

K K K 

02hl(y)/Ox20XK = 0 = ~ a~ (OZyl/OXl Oxr,:)+ ~ Y, b~[O2(yiyi)/Ox20XK], 

i=I i=1 j=i 

K K K 

02hl(y)/Ox2=O= ~ a~(O2yl/Ox~)+ y~ y~ bij[O 1 2 (ylyi)/Oxlc]. 2 

i~l i=1 ]=i 

There are K(K + 1)/2 equations [distinct second-order derivatives of hKy) 

with respect to xl,..., XK] in (9). Thus, the system of equations (7)-(9) 

includes 1 +K +K(K + 1)/2 equations in the same number of unknowns 

(aol;a~,.. 1 a 1 b 1 b 1 b~cr).Rewritingsystem (7)-(9) 

.,aK;bn,...,blK, 22, 23," .-, 

in matrix form gives 

where 

d~ = (x °, 1, 0 ..... 0)', 

(9) 

A01 = dl, (10) 

e, -- (a0~; al,.. .,aLbh, b ~12, • • • , bb,)',

and 

A= 

-1 yl 

0 Oyl/Oxl 

: 

0 Oyl/OxK 

0 OZy~/ox~ 

0 02yl/OXl OX2 

0 o2y~/Ox~ 


2 2 

YK Yl YlY2 " " " YK 

OyK/Oxl O(y~)/axl o(y~yz)/ax~ ., • O(y~)/ax~ 

: : : : 

Oy~/OxK o(y~)/~xr o(y~y2)/oxK .,. O(y~)/Oxr, 

02 yr/Ox21 02(y2)/Ox~ O2(yly2)/OX 2 ..o 02(y2)/t~X21 

82yK/aXl OX2 a2(y21)/OXl aX2 a2(ylY2)/OXl~X2 "'" 02(y2)/OXl OX2 

If A is nonsingular, the solution for the unknown 01 is given by 

0i =A-ldl. (1t) 

Repeating the derivation of Eqs. (7)-(9) for h2, we get a similar set 

of equations; that is, 

where 

d2=( x 2,0, 0 1, 0,..., 0)', 

A02 = d2~ (12) 

02=(ao 2"a~, 2 , b 2 

..... aK; b~l, 22 . . . . . b2Ky, 

and A is given in (10). Clearly, nonsingularity of A implies 

02 = A-ld2. (13) 

Applying the same derivation for any h~, i = 1 ..... K, yields 

where 

and 

AO~ = du (14) 

di= (x o, 0, .... 0, 1, 0 ..... 0)', 

i-1 

i i i i i 

0i = (a~; as .... , aK; bm blz .... , bKK), 

Oi = A -ldi, (15) 

where A is defined in (10). Thus, (11) gives the solution for all a~ and b~ 

in (7), and (15) yields the unknown parameters for any h~, i = 1 ..... K.


Choose 

It follows from (7) that 

or, in general [see (15)], 

i 

x* =ao, i=1 ..... K. 

K K K 

i=1 i=1 j=i 

K K K 

Z a ~y~ + Z Z b ~:y~yj = 0, 

i=1 i=1 j=i 

b qy~yj" = 0 

for n = 1,...,K. 

If A in (10) is nonsingular, the K equations (17) hold if and only if 

From (2), 

hence, 

This implies that 

yl=y2 ..... yK=0. 

y=0~ atx =x*; 

x*=(at, ag .... ,a~)'. 

i 

X} g ---- a0 

(16) 

(17) 

is the first component of the vector 0~ in (14). This component is the product 

of the first row of A -~ in (10) and d, in (15); thus, 

i --1 0 

x* =ao=(A )taxi +(A-1)a.i+1, (18) 

where (A-1)i,i is the iith element of A -1. The first column in A is 

this implies that 

(1, 0, 0 ..... 0)'; 

(A-1)1.1 = 1, 

which gives the desired result in (4). 

When r > 2, we rewrite (7) as follows: 

xl=hl(y)=a~+ E ' 

K K K 

aiyi+ E E b~.yiyj 

i=l i=l .i=i 

K /,2 K 

+'"+22" "" Z qij 1 ..... YlYj "'" y,~yr. (19) 

i=1 l=i r=rn 

We compute all the first-order up to rth order derivatives of (19) with

JOTA: VOL. 39, NO. 1, JANUARY !983 7 

respect to xl,..., xx to get the same number of equations as there are 

unknowns 

(a 's, b's ..... q's). 

We form (10) again, where A is given now by (5). Moreover, 

and 

01=(a~.al ' 1 1 

..... aK; bm b12 ..... blK;... ; 

1 1 1 

qll.,.ll; qll...12 ..... qKK...KK) 

d~ = (xl, 1, 0 ..... 0)', 

where the number of elements in dl is equal to the number of parameters 

in 01. The vector 01 is given by (11) and, in general, 0i is given by (15). 

Again, choose 

x* =a?, i=1 ..... K, 

which implies that (19) holds if and only if 

yl=y2 ..... yK =0. 

The rest of the proof is exactly similar to the case where r = 2. 

In the next section, we shall use the results of Theorem 2.i, combined 

with the table algorithm given in KaJaba and Tishler (Ref. 7), to present 

a simple algorithm for solving problem (1). 

3. Generalized Newton Algorithm and Table Algorithm 

The theorem in Section 2 states that, if the functions hl (y), i = 1 ..... K, 

are polynomials of rth degree, using Eq. (4) will result in finding the optimal 

x*, starting at x ° close to x*, in one step. 

In general, one does not know hi(y), and we propose to approximate 

them by a polynomial of degree r. A GN(r) algorithm for finding the optimal 

x in problem (1) is defined as follows: 

Step a.O. Choose x ° and form A according to Eq. (5). 

Step a.i. Set i = 1, 2, 3 ..... 

Step a.i.1. Calculate x; =x* according to Eq. (4). Go to Step a.i.2. 

Step a.i.2. If I]x ~-xi-lll0 being some tolerance, go to Step 

a.i.3.; otherwise, compute A using Eq. (5) at x =x i and go to Step a.i.1. 

Step a.i.3, x ~ is a local minimum for F(x) if y (x i) = 0 and the Hessian 

of F is positive definite.


Note that, if r = 1, the above algorithm is the Newton-Raphson 

algorithm. The GN(r) algorithm requires the use of the values of higher- 

order derivatives in the computation of the matrix A in Step a.i.2. Analytical 

derivatives of order 2 or higher are, in general, very difficult to obtain; 

thus, we shall use the table algorithm to compute the exact values of these 

derivatives. 

The precise description and properties of the table algorithm are given 

in Kalaba et al. (Ref. 6) and Kalaba and Tishler (Ref. 7). In this paper, we 

shall only briefly describe the table algorithm. 

In principle, a table algorithm can be constructed to calculate the value 

and partial derivatives through order r of any admissible (r, n) function, 

defined as follows. 

Definition 3.1. A function F:D -~ R, D C R n, n - 1, will be referred 

to as an admissible (r, n) function if, given any x = (xl ..... xn) in D, the 

value F(x) can be sequentially calculated in a finite number of steps by 

means of the n initial conditions 

sl =xl ..... s. =xn, (20) 

the two-variable algebraic special functions f : R 2 --> R, given by 

w=u+v, w=u-v, w=uv, w=u/v, w=u v, (21) 

and arbitrary, one-variable, rth-order continuously differentiable special 

functions of the form 

h :M~,R, McR. (22) 

It is quite easy to observe that, if we can write a function F and its 

derivatives, up to order r, analytically, then F is an admissible (r, n) function. 

Given any admissible (r, n) function F with domain point x, the general 

table algorithm for F(x) is constructed as follows: 

Step I. Form the admissible list for F(x), i.e., a list of initial conditions 

(20) and special functions (21) and (22) whose sequential evaluation yields 

F(x). 

Step 2. Replace (using a calculus subroutine) the initial conditions 

sl = xl in the admissible list for F(x) by a one-dimensional array (vector) 

containing the value and partial derivatives through order r of the function 

g : R" ~ R defined by 

g(xl .... , x°)-=xl. 

Specify analogous arrays (vectors) for the functions 

g(xl,...,x,)--xl, i---2 ..... n,


to replace the remaining initial conditions 

in the admissible list for F(x). 

si = xi, i = 2,..., n, 

Step 3. Replace each special function (21) and (22) in the admissible 

list for F(x) by a calculus subroutine for outputting the value of the special 

function, together with all of its partial derivatives with respect to x through 

order r. For any two-variable special function w = f(u, v), the input to the 

calculus subroutine will be the previously calculated one-dimensional arrays 

and 

U ~ (u, u~,~, usa,...) 

V =- (v, v,1, vx~ .... ), 

and the output will be a one-dimensional array 

For any one-variable special function 

W =- (w, w~ l, w~ .... ). 

z = h(y), 

the input to the calculus subroutine wilt be the previously calculated 

one-dimensional array 

Y - (Y, Yxl, Y~, 

and the output will be a one-dimensional array 

Z - (z, z~,, zx~,...). 

The list of calculus subroutines resulting from these three steps is 

called the table algorithm for F at x. The proof that the table algorithm 

for F at x correctly calculates the value and all partial derivatives through 

order r of F at x follows by a straightforward complete induction argument. 

The value of the function F and its derivatives (up to order r) are 

exact, provided that the sequential evaluation given in steps (1) to (3) above 

is correct. In practice, as in the case with the use of explicit analytical 

derivatives, roundoff and truncation errors wilt be introduced. 

Currently we have a program that evaluates the derivatives up to order 

3 of functions with arbitrary number of variables. 

It is obvious that to construct the matrix A in (5) for r = r °, one needs 

the values of all the derivatives of F(x) up to order r°+ 1. However, note 

that the construction of A requires the construction of the functions and 

all the derivatives (up to order r) of Yt,..-, YK, yl z, yly2, Yly3,--., as is


evident in the matrix A in (10). These vectors are already available from 

the computation of F(x) using the table algorithm; thus, the construction 

of A in (5) or (10) is straightforward. 

4. Application: CES Production Function 

In this section, we apply the table algorithm to the generalized CES 

production function given by [see Arrow et aI. (Ref. 10)] 

O=f(K,L;x)=~/[aKO+(1-a)Le] ~/~, 0

Table 1. Results of estimation, T ~- 25, GN(1) Algorithm. 

< 

© 

Iteration 

number t3 v ~ G F~ G~ Ge 

9 

Z 

0 

y- 

0 0.65000 -0.45000 1,45000 0.85000 349.36 29,77 0,15 x 104 0.11 x 103 

1 0.6476 -0.56873 1.45135 0.81543 72.68 6.13 0.89 X i03 0.78X 102 

2 0.61477 -0.54893 1.45128 0.80779 7.22 0.46 0.70 x 103 0,67 x 102 

5 0.59530 -0,45344 1,45364 0.80687 -2,42 -0.22 0.67 X 10 3 0,65 x 10 ~ 

7 0,59985 -0.49852 1,49851 0.80021. -0.11 -0.93 x 10 -2 0.67 x 103 0.64 x 102 

9 0,60000 -0.50000 1.50000 0.80000 -0,75 x 10 -6 -0.67 x 10 .7 0.6 7 x 103 0.64 x 102 

Iteration 0 gives the initial conditions. Computations done in double precision on IBM 370/168 at the University of Southern California. 

,< 

oo

Fo 

Table 2. Results of estimation, T = 25, GN(2) Algorithm. 

< 

O 

Iteration 

number a f3 T ~ F,~ F~ F,~ F~a F.,~o, 

ho 

P 

9 

Z 

0 0.65000 -0.45000 1.45000 0.85000 349.36 29.77 0.15 X 10 4 0.11 X 103 0.32 × 102 

1 0.63532 --0.61571 1.45240 0,80926 25.96 2.00 0,76 X 103 0.70 X 102 0.19 X 103 

2 0.59309 --0.43161 1.43542 0.81037 -0.5X 10 -2 0.3X10 -2 0,68X103 0.66X 102 0.19X 103 

3 0.59998 --0.49977 1.49977 0.80002 --0.08 --0.07 0.67 X 10 3 0.64 × 102 0.19 × 10 3 

4 0.60000 -0,50000 1.50000 0,80000 -0.71 X t0 -6 -0.67 X 10 -7 0.67 X 10 3 0.64 × 102 0.19 × 103 

See notes of Table 1. 

Oo

JOTA: VOL. 39, NO. 1, JANUARY 1983 !3 

The values for (Kt, Lt) are lid normal variates; that is, 

~N t = 1, T. (26) 

Lt 60 '\100 350]J' "'" 

The true parameters are 

x*- (a,/3, r, ~) = (0.6, -0.5, 1.5, 0.8). 

The Q,'s were generated according to (23) given the values of x, K~, and 

L,. The starting point for optimization is 

x ° = (0.65, -0.45, 1.45, 0.85). 

Table 1 presents the results for the GN(1) algorithm (which is the 

Newton-Raphson method). 

The convergence to the optimum using GN(1) is fairly fast. Note that 

the function F is very steep further away from the optimum, as can be 

seen from the values of F~ and F~ in iterations 0, 1, 2. It is very fiat close 

to the optimum (the Hessian is almost singular), but we did not encounter 

any problems of convergence once we get close to the optimum. In other 

experiments, starting further away from x* (the true value), the algorithm 

sometimes failed, as is expected, to achieve convergence. 

In Table 2, we present the results of using GN(2); that is, the GN 

algorithm, using derivatives of F up to order 3. The data and initial 

conditions are equal to those in Table 1. The number of iterations needed 

for convergence is, however, much smaller than in the Newton-Raphson, 

GN(1), algorithm. As was the case for GN(1), when we started further 

away from x*, the algorithm sometimes failed to converge. 

For our example, however, for a large number of experiments, every- 

time that convergence was achieved, GN(2) was much faster than GN(I) 

in terms of number of iterations. For the limited number of examples (the 

CES function and other functions) that we tried, it seems that the domain 

of attraction of GN(1) may be slightly greater than that of GN(2). We shall 

further elaborate on this in the next section. 

5. Extensions and Discussion 

Section 3 derives the generalized Newton algorithm in general form. 

However, one can improve this algorithm by defining it as follows (i is the 

iteration index): 

x ~+1 = x / + a/8 i, (27) 

where 8 ~ is defined in (4) and ct~, a scalar, is the stepsize along the search


direction. For simplicity of exposition, we chose ai = 1, but one can use 

any other method to determine a~ (Ref. 5). 

Additional research is needed to determine what is the optimal order 

of derivatives to be used in different problems. In other words, what is the 

optimal GN(r)? The advantage of using larger r is obvious if the functions 

to be approximated [the h's in (3)] behave like a high-degree polynomial. 

However, there are two possible disadvantages. First, it can be costly, in 

terms of computer time, to evaluate derivatives of high order in functions 

with many variables. The limited number of experiments that we carried 

out using the table algorithm to evaluate derivatives up to order 3 needed 

negligible additional computer time. We feel that the additional computer 

costs for computing derivatives up to the fourth order for functions of up 

to 10 variables are not substantial. However, further experimentation is 

needed in that area. The second disadvantage can be described using a 

simple example. 

Let Y = F(x), where x is a scalar. Let 

and 

y = OF (x)/Ox = f(x ), 

x =h(y) 

is the inverse of f. Approximate h (y) as follows, using GN(2): 

a 2 

x=ao+aly+ 2Y +R, (28) 

where R is the remainder term. Differentiate (28) twice with respect to x 

to get 

1 = a l(oy/Ox) + a2[O(y 2)/Ox ] + OR/Ox, (29) 

0 = a 1 (02y/Ox 2) + az[O2(y 2)/0X Z] q_ 02R/ox 2. (30) 

Applying Theorem 2,1, we obtain 

i+1 i ~ tl t 

x =x -(y/y')[l+l/2(y/y )(y /y )], (31) 

where 

y' =Oy/Ox and y" =O2y/Ox 2. 

The problem is, as can be seen in (28) and (29), in neglecting OR/Ox 

and oER/Ox 2. Differentiation is a roughening operation; and, even if R in 

(28) is very small, OR/Ox can be large relative to 1 and oER/Ox 2 can be 

large relative to 0, thus, presenting a large noise to the approximation 

process (31). We expect, however, that close to the optimum, or when 

F(x) is smooth but nontrivial, higher-order methods will do better. We 

use an example to illustrate that last remark.


Consider the circle given by 

Solving for y, we have 

(y -a) 2 + (x -/3) 2 = R 2. (32) 

y = a ±,][R 2_ (x -/3)a]. (33) 

Assume that (33) is y =f(x) [see (2)], and we have to find x such that y = 0. 

The maximal value of x is 

x=a+R. 

If we choose x ° to be very close to a +R and use the negative value of 

the square root in Eq. (33) (to ensure convergence), we expect a higher 

method of optimization to be superior, since it gives a better approximation 

to y = f(x). Table 3 gives the results of the optimization for a = i, I3 = 2, 

R=3. 

As is clearly seen in Table 3, GN(2) is much faster in locating the 

correct root for x, compared to GN(1). This is the result of a second-degree 

polynomial approximating the half circle much better than a first-degree 

polynomial. 

Finally, we would like to point out two other extensions to our 

algorithm. 

(a) Equation (7) could include terms other than yi and products of 

y~, yj. For example, one can specify (7) as follows: 

K K K 

1 

x~=hl(y)=a~+ E a~y~+ E E bo~/(Y,Yj) (34) 

i=1 i=lj=i 

Table 3. Results for Example (33) using GN(1) and GN(2). 

GN(1) GN(2) 

Iteration 

number x 2 x y 

0 3.999990 1.992254 

1 3.994846 1.824223 

2 3.887776 1.187138 

3 3.553617 0.425503 

4 3.291262 0.063478 

5 3.237612 0.001728 

6 3.236069 0.000001 

7 3.236068 0.000000 

3.999990 1.992254 

3.333327 0.114374 

3.236264 0.000220 

3.236068 0.000000 

Iteration 0 gives the initial conditions when x = 3.999990 and ~r = 1,/3 = 2, R = 3.


or 

K K K 

xl=hi(y) = ao~+ ~ a~x/(yi)+ ~ ~ 

i=1 i=l]=i 

bii 1 sm(yiyi). • 

(35) 

Other possibilities are obvious. The use of the algorithm is unaffected by 

these specifications, which may improve the optimization process. 

(b) The basic general equation to compute x~ +I is given by (10) as 

follows: 

AOl = dl. (36) 

One could get x ~+~ as the first component in 01, using 

O1 =A-*d1, (37) 

which solves for all the parameters in 01 [see (10)]. It is apparent that we 

need to solve only for a01. Thus, our problem can be redefined as follows: 

Compute c'z, when AO1 = dl, where 

Then, 

c=(a~, ,a 1 b~l,b 1 b ~:K)', 

z = (Yl ..... YK; Y~,YlY2,-.-,Y2)'. 

an =xl-c'z, 

and there is no need to solve for all the vector of unknowns 01. Solving 

(38), using only forward Gaussian elimination, is a more efficient technique, 

compared to matrix inversion. 

References 

1. DENNIS, J. E., and MORt~, J., Quasi-Newton Methods: Motivation and Theory, 

SIAM Review, Vol. 19, pp. 48-89, 1977. 

2. DENNIS, J. E., and MEI, H. H. W, Two New Unconstrained Optimization 

Algorithms Which Use Function and Gradient Values, Journal of Optimization 

Theory and Applications, Vol. 28, pp. 453-482, 1979. 

3. GOLDFELD, S. M., and QUANDT, R. E., Nonlinear Methods in Econometrics, 

North-Holland, Amsterdam, Holland, 1972. 

4. PIERRE, D. A., Optimization Theory with Applications, John Wiley, New York, 

New York, 1969. 

5. BELSLEY, D. A., On the Efficient Computation of the Nonlinear Full-Informa- 

tion Maximum-Likelihood Estimator, Journal of Econometrics, Vol. 14, pp. 

203-225, 1980.

JOTA: VOL 39, NO. 1, JANUARY I983 17 

6. KALABA, R., RASAKHOO, N., and TISHLER, A., Nonlinear Least Squares 

via Automatic Derivative Evaluation, Applied Mathematics and Computation 

(to appear). 

7. KALABA, R., and TISHLER, A., Optimization in Nonlinear Models via 

Automatic Derivative Evaluation (to appear). 

8. Wlb2NGERT, R,, A Simple Automatic Derivative Evaluation Program, Com- 

munications of the Association for Computing Machinery, Vol. 7, pp. 463-464, 

1964. 

9. BELLMAN, R., KAGIWADA, H., and KALABA, R., Wengert's Numerical 

Method for Partial Derivatives, Orbit Determination, and Ouasitinearization, 

Communications of the Association for Computing Machinery, Vol. 8, pp. 

231-232, 1965. 

10. ARROW, K., CHENERY, H. B., MINHAS, B., and SOLOW, R. M., Capital- 

Labor Substitution and Economic Efficiency, Review of Economics and Statis- 

tics, Vol. 43, pp. 228-232, 1961. 

[1. HENDERSON, J. M., and QUANDT, R. E., Microeconomic Theory, McGraw- 

Hill, New, York, New York, 1980,

A generalized Newton algorithm using higher-order derivatives

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template ?