Approximation of Hessian Matrix for Second-order SPSA Algorithm ...

1.6 FORMULATION OF SPSA ALGORITHM
∂L
g ( θ)
= = 0 . (1.4)
∂ θ
For the gradient-free setting, it is assumed that measurements of L (θ ) say y (θ ) are available at
various values of θ . These measurements may or may not include random noise. No direct
measurements (either with or without noise) of g (θ ) are assumed available in this setting. In
the Robbins-Monroe/stochastic gradient (SG) case [9], it is assumed that direct measurements of
g (θ) are available, usually in the presence of added noise. The basic problem is to take the
available information measurements of L (θ ) and/or g (θ ) and attempt to estimate
*
θ . This is
essentially a local unconstrained optimization problem. The SPSA algorithm is a tool for
solving optimization problems in which the cost function is analytically unavailable or difficult
to compute. The algorithm is essentially a randomized version of the Kiefer-Wolfowitz method
in which the gradient is estimated using only two measurements of the cost function at each
iteration [15][16]. SPSA is particularly efficient in problems of high dimension and where the
cost function must be estimated through expensive simulations. The convergence properties of
the algorithm have been established in [16]. Consider the problem of finding the minimum of a
real valued function L (θ ), for θ ∈D
where D is an open domain in
P
R . The function is not
assumed to be explicitly known, but noisy measurements M ( n,
θ)
of it are available:
M ( n,
θ)
= L(
θ)
+ ε ( θ)
(1.5)
n
where { ε n
} is the measurement noise process. We assume that the function L (⋅)
is at least
three-times continuously differentiable and has a unique minimize in D. The process { ε n
} is a
zero-mean process, uniformly bounded and smooth in θ in an appropriate technical sense. The
problem is to minimize L (⋅)
using only the noisy measurements M (⋅)
. The SPSA algorithm for
minimizing functions relies on the SP gradient approximation [16]. At each iteration k of the
algorithm, a random perturbation vector
∆ is taken, where the ∆
ki
forms a
T
k
= ( ∆k
1,...,
∆kp)
sequence of Bernoulli random variables taking the values ± 1. The perturbations are assumed to
be independent of the measurement noise process. In fixed gain SPSA, the step size of the
perturbation is fixed at, say, some c > 0. To compute the gradient estimate at iteration k, it is
necessary to evaluate M (⋅)
at two values of θ :
M θ ) L(
θ + c∆
) + ε ( θ + c∆
)
(1.6)
+
k
( =
k 2k−1
k
13