A note on estimating autocovariance from short-time observations

A <strong>note</strong> on estimating autocovariance from
short-time observations
Yoel Shkolnisky, Fred J. Sigworth and Amit Singer
Abstract
We revisit the classical problem of estimating the autocovariance function and power spectrum of a
stochastic process. In the typical setting for this problem, one observes a long sequence of samples of
the process, from which the autocovariance needs to be estimated. It is well known how to construct
consistent estimators of the autocovariance function in this case, and how to trade bias for variance. In
physical settings such as cryo-electron microscopy (EM) we are required to estimate the response of the
physical instrument through the observation of many short noise sequences, each with a different mean.
In this setting, the known estimators are significantly biased, and unlike the typical case, this bias does
not disappear as the number of observations increases. The bias originates from replacing the unknown
true mean by the sample average. We analyze and demonstrate this bias, derive an unbiased estimator,
and examine its performance for various noise processes.
EDICS Category: SSP-SSAN Statistical signal analysis
I. INTRODUCTION
The autocovariance function (ACF) of a discrete stationary ergodic time series {Xt} ∞ t=−∞
value E[Xt] = µ at lag h is defined as
1
with expected
C(h) = E[(Xt − µ)(Xt+h − µ)]. (1)
Yoel Shkolnisky is with the Department of Mathematics, Program in Applied Mathematics, Yale University, 10 Hillhouse
Ave. PO Box 208283, New Haven, CT 06520-8283 USA.
Fred J. Sigworth is with the Department of Cellular and Molecular Physiology, Yale University School of Medicine, 333
Cedar Street, New Haven, CT 06520 USA.
Amit Singer is with the Department of Mathematics and PACM, Princeton University, Fine Hall, Washington Road, Princeton
NJ 08544-1000 USA.
emails: yoel.shkolnisky@yale.edu, fred.sigworth@yale.edu, amits@math.princeton.edu
July 14, 2008 DRAFT

The ACF can be estimated from a length N finite sample X1,X2,... ,XN as
Ĉ0(h) = 1
N−h
(Xi − µ)(Xi+h − µ), h = 0,... ,N − 1. (2)
N − h
i=1
The estimator Ĉ0(h) is an unbiased estimator of the ACF, i.e., E[ Ĉ0(h)] = C(h), but is known to have
a larger variance than the biased estimator [1], [2]
Ĉ1(h) = 1
N−h
(Xi − µ)(Xi+h − µ), h = 0,... ,N − 1. (3)
N
i=1
When the true mean µ is unknown to the data analyst, as is often the case in practice, different estimators
of the ACF have to be used. Many textbooks suggest replacing the unknown true mean by the sample
mean
in (2)–(3), leading to the estimators
and
¯X = 1
N
N
i=1
Xi
˜C0(h) = 1
N−h
(Xi −
N − h
¯ X)(Xi+h − ¯ X), (5)
i=1
˜C1(h) = 1
N−h
(Xi −
N
¯ X)(Xi+h − ¯ X). (6)
i=1
Unfortunately, both ˜ C0(h) and ˜ C1(h) are biased estimators of C(h), as we see below. The negative bias
caused by replacing the true mean by the sample mean was <strong>note</strong>d long ago and was shown to be of the
order of N −1 [3]–[6]. For long time series, the 1/N bias is not so harmful, but for short time series it
may cause a non-negligible discrepancy. The bias can be further reduced to order N −2 by a procedure
similar to the Richardson extrapolation in numerical analysis or bootstrapping [7], [8]. In this method,
the time series is chopped into two halves of length N/2. The ACF is estimated separately for each half
based on its own sample mean, yielding two different estimates which are averaged and subtracted from
twice the ACF estimator for the entire series. Though the resulting bias is asymptotically smaller, its
range is only N/2, and the question whether or not it is possible to construct an unbiased estimator for
the ACF remains.
An unbiased estimator of the ACF is useful in cases where the data analyst is faced with many
independent short time series sharing the same ACF (up to a multiplicative constant) but different unknown
true means. An unbiased estimator would be useful even at the expense of variance inflation with respect
to other biased estimators, since averaging over many different series would reduce the variance error
term. This is exactly the case one encounters in analyzing the noise characteristics of single-particle
July 14, 2008 DRAFT
2
(4)

cryo-EM images [9] which are small, often with fewer than 100 pixels on a side. Every image is a
superposition of signal and noise filtered by the contrast transfer function of the microscope. The signal
part differs from image to image as it corresponds to projections of the molecule at different unknown
view angles. After acquisition, the images are normalized to have zero mean and unit variance (ℓ2 energy).
Note that further discrepancies are introduced by the acquisition process itself, as different images may
have different acquisition characteristics (variations in acquisition times, etc.).
In this paper we explain the origin of the bias in (5) and (6) and derive a simple unbiased estimator
for the ACF. We investigate the effect of the bias on the power spectrum, and demonstrate the bias and
its elimination in numerical examples for different types of colored noise.
II. ORIGIN OF THE BIAS
To evaluate the estimation bias of ˜ C0 of (5), we calculate its expected value
We <strong>note</strong> that
E[ ˜ C0(h)] =
=
From (8) it also follows that
Plugging (8) and (9) in (7) gives
where
N−h
1
E
N − h
(Xi − ¯ X)(Xi+h − ¯ X)
i=1
N−h
1
E
N − h
(Xi − µ) − ( ¯X − µ) (Xi+h − µ) − ( ¯X − µ) .
i=1
E[(Xi − µ)( ¯ ⎡ ⎛
X − µ)] = E ⎣(Xi − µ) ⎝ 1
N
= 1
N
= 1
N
E ( ¯ X − µ)( ¯ X − µ) = 1
N
B(h) = 1
N−h 1
N − h N
i=1
⎞⎤
N
(Xj − µ) ⎠⎦
j=1
N
E [(Xi − µ)(Xj − µ)]
j=1
3
(7)
N
C(i − j). (8)
j=1
= 1
N 2
N
E (Xk − µ)( ¯ X − µ)
k=1
N
j,k=1
C(k − j).
E[ ˜C0(h)] = C(h) − B(h), (10)
N
j=1
[C(i − j) + C(i + h − j)] + 1
N 2
N
j,k=1
(9)
C(j − k) (11)
July 14, 2008 DRAFT

is the bias term, which is expressed as a linear combination of the true ACF.
In applications, the autocovariance function is often used to estimate the power spectrum of the noise,
a method known as the correlogram [10]. The power spectrum is estimated from the ACF as
S(ω) =
N−1
h=−N+1
C(h)w(h)e −2πiωh/(2N−1) , (12)
with some window function w(h), where we used the Hermitian symmetry of the ACF C(h) = C ∗ (−h).
Therefore, estimating the power spectrum as the Fourier transform of the estimated ACF (5) is a biased
estimator due to the bias in C(h). Further bias is introduced due to the windowing and the finite lag
extent of the autocovariance. The window in (12) is used also to reduce the variance of the estimate
[10]. In our setting, windowing for that purpose is unnecessary, as we can always reduce the variance
by averaging over many K different sequences.
For example, for white noise (WN) the true ACF is
C WN ⎧
⎨ σ
(h) =
⎩
2 h = 0
0 h = 0,
and equation (10) becomes
E[ ˜ C WN
0 (h)] = C WN (h) − σ2
, (13)
N
rendering the order N −1 negative bias of ˜ C0. In this case the bias is only a constant shift of σ2
N
independently of h. For general noise processes, however, the bias depends of h, as we see in Section
IV.
III. ELIMINATING THE BIAS
We would like the power spectrum estimator to be unbiased for all frequencies with the exception of
the DC component. As the DC component usually plays no role in applications, we gain a degree of
freedom that allows us to estimate the autocovariance up to an additive constant.
The first step in deriving an unbiased estimator for the power spectrum is noting that (1) can be
rewritten as
and in particular
Subtracting (15) from (14) gives
C(h) = E[XtXt+h] − µ 2 , (14)
C(0) = E[X 2 t ] − µ 2 . (15)
c(h) ≡ C(h) − C(0) = E[XtXt+h] − E[X 2 t ]. (16)
July 14, 2008 DRAFT
4

The representation (16) does not involve the unknown parameter µ and thus can be estimated directly
from the data as
ĉ(h) = 1
N−h
XiXi+h −
N − h
1
N
i=1
N
i=1
X 2 i . (17)
The estimator ĉ(h) given by (17) is an unbiased estimator of c(h) = C(h) − C(0). Note that the power
spectrum estimator
ˆS(ω) =
N−1
h=−N+1
ĉ(h)w(h)e −2πiωh/(2N−1)
is still a biased estimator of S(ω) due to the finite lag extent. However, we observe that
E[ ˆ S(ω)] = E[S(ω)] − C(0)W(ω),
where W(ω) is the discrete-time Fourier transform of the window function, so if a fast-decaying window
is used, only the very low frequencies are affected compared to the case of a known µ.
IV. NUMERICAL EXAMPLES
We start by demonstrating the bias of the estimate (5) as well as the performance of the unbiased
estimator (17) for the case of standard Gaussian noise in Figs. 1a and 1b. Figure 1a was generated
follows. We generate K = 10 4 sequences of length N = 5 samples of a standard Gaussian random
variable. For each of the K sequences, we use (5) to estimate an N-term autocovariance function,
followed by averaging all K estimates. Figure 1a shows the true autocovariance function (a delta function
in this case), its biased estimate (5), and its unbiased estimate (17). Since the unbiased estimate (17)
is determined up an additive constant, we shift it such that the autocovariance at distance zero is one.
Figure 1b is generated in exactly the same way but with N = 50. We see that the bias agrees with (13).
The smaller N, the bigger the bias.
We next take a stochastic process whose autocovariance is C(h) = a |h| , where a is a constant chosen
such that the autocovariance at distance h = 50 is 10 −8 . As before, we generate K noise sequences
of length N (using a simple autoregressive model), estimate the autocovariance of each sequence, and
average all K estimates. This is shown in Figs. 2a and 2b for K = 10 4 and K = 10 6 , respectively, with
N = 50. In each of the figures we show the true autocovariance, the biased estimate and the unbiased
one. It is clear from Figs. 2a and 2b that increasing K reduces the variance of the estimate.
It is also apparent that unlike the Gaussian case above, the bias is not constant and varies with h. To
see how the bias behaves as a function of h, we plot in Figs. 3 the theoretical bias, computed using (11),
July 14, 2008 DRAFT
5
(18)

correlation
correlation
1
0.8
0.6
0.4
0.2
0
−0.2
true
biased
unbiased (shifted)
−0.4
1 1.5 2 2.5 3
correlation lag
3.5 4 4.5 5
1.2
1
0.8
0.6
0.4
0.2
0
(a) N = 5, K = 10 4
correlation
1.2
1
0.8
0.6
0.4
0.2
0
true
biased
unbiased (shifted)
−0.2
0 5 10 15 20 25
correlation lag
30 35 40 45 50
(b) N = 50, K = 10 4
Fig. 1: Autocovariance estimates from K realizations of N samples of Gaussian noise.
true
biased
unbiased (shifted)
−0.2
0 5 10 15 20 25
correlation lag
30 35 40 45 50
(a) K = 10 4 , N = 50
correlation
1.2
1
0.8
0.6
0.4
0.2
0
−0.2
0 5 10 15 20 25
correlation lag
30 35 40 45 50
(b) K = 10 6 , N = 50
true
biased
unbiased (shifted)
Fig. 2: Autocovariance estimates from K realizations of N samples of noise with autocovariance a |h| .
together with the measured bias, computed by subtracting a |h| from the autocovariance estimated using
(5). Figures 3a and 3b show the behavior of the bias for K = 10 4 and K = 10 6 , respectively.
We next show how the biased estimate of the autocovariance function affects the estimate of the power
spectrum. As in previous figures, we estimate the autocovariance from K noise sequences of length N.
We then compute the power spectrum using (12) with w(h) = 1 for both the biased and the unbiased
July 14, 2008 DRAFT
6

correlation
−0.02
−0.025
−0.03
−0.035
−0.04
−0.045
−0.05
−0.055
measured bias
theoretical bias
−0.06
0 5 10 15 20 25
correlation lag
30 35 40 45 50
(a) K = 10 4 , N = 50
correlation
−0.015
−0.02
−0.025
−0.03
−0.035
−0.04
−0.045
−0.05
−0.055
measured bias
theoretical bias
−0.06
0 5 10 15 20 25
correlation lag
30 35 40 45 50
(b) K = 10 6 , N = 50
Fig. 3: Measured and theoretical bias for a process with autocovariance C(h) = a |h| .
estimates. The result is shown in Figs. 4a and 4b for K = 10 4 and K = 10 6 , respectively. We show only
positive frequencies, as the power spectrum is symmetric around the origin and the value at the DC is
irrelevant. It is important to <strong>note</strong> that the wiggles in the biased estimate are not due improper windowing.
In fact, no window is needed in this case. The constant a was chosen such that the autocovariance is
essentially periodic. Therefore, there is no spectral leakage due to discontinuities at the boundaries. Also
<strong>note</strong> that the power spectrum is sampled at the FFT points, in which the interaction of a given frequency
sample with adjacent samples is zero - these points are exactly the zeros of the Dirichlet kernel (the
Fourier transform of the rectangular window). The relative error in the power spectrum estimate is shown
in Figs. 5a and 5b.
We next show a similar experiment for a different autocovariance function. In Figs. 6a–6d we take the
power spectrum
S(ω) = 1
√ 2π e −ω2 /2 (1 + 0.1cos (10ω)) (19)
which is essentially bandlimited, and use the sampled power spectrum to generate noise samples with
the given sample spectrum. We again take K noise sequences of length N, from which we estimate the
autocovariance function. In Figs. 6a and 6b we see the true, biased, and unbiased autocovariance estimate
for N = 25 and N = 100, respectively. Since the autocovariance in this case has no closed form, we
compute the “true” autocovariance simply by not removing the sample mean in (5), essentially using the
prior knowledge that the process has zero mean. The unbiased estimate in both cases is shifted such that
July 14, 2008 DRAFT
7

3
2.5
2
1.5
1
0.5
true
biased
unbiased
0
0 5 10 15 20 25 30 35 40 45 50
frequency (FFT bin)
0.08
0.06
0.04
0.02
0
−0.02
−0.04
−0.06
−0.08
(a) K = 10 4 , N = 50
3
2.5
2
1.5
1
0.5
true
biased
unbiased
0
0 5 10 15 20 25 30 35 40 45 50
frequency (FFT bin)
(b) K = 10 6 , N = 50
Fig. 4: Power spectrum estimate from autocovariance functions of Fig. 2.
−0.1
0 5 10 15 20 25 30 35 40 45 50
frequency (FFT bin)
(a) K = 10 4 , N = 50
−0.02
−0.04
−0.06
−0.08
Fig. 5: Relative error in power spectrum estimation.
0.08
0.06
0.04
0.02
0
−0.1
−0.12
0 5 10 15 20 25 30 35 40 45 50
frequency (FFT bin)
(b) K = 10 6 , N = 50
it agrees with the true autocovariance for h = 0. The correlation lags in Figs. 6a and 6b are given in
time steps determined by the sampling rate of the power spectrum (19). In Figs. 6c and 6d we see the
corresponding estimated power spectra.
July 14, 2008 DRAFT
8

correlation
0.12
0.1
0.08
0.06
0.04
0.02
0
true
biased
unbiased (shifted)
−0.02
0 5 10 15 20 25
correlation lag
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
(a) K = 10 5 , N = 25
−0.05
0 5 10 15 20 25
frequency (FFT bin)
(c) K = 10 5 , N = 25
true
biased
unbiased
correlation
0.45
0.12
0.1
0.08
0.06
0.04
0.02
0
true
biased
unbiased (shifted)
−0.02
0 10 20 30 40 50
correlation lag
60 70 80 90 100
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
(b) K = 10 5 , N = 100
−0.05
0 10 20 30 40 50 60 70 80 90 100
frequency (FFT bin)
(d) K = 10 5 , N = 100
Fig. 6: Estimated autocovariance and power spectrum for the power spectrum in (19).
V. ACKNOWLEDGMENTS
We would like to thank Mark Tygert for interesting discussions.
REFERENCES
[1] A. Papoulis, Signal Analysis, McGraw-Hill Companies, 431 pages, 1977.
true
biased
unbiased
[2] A. V. Oppenheim, R. W. Schafer, and J. R. Buck, Discrete-Time Signal Processing, 2nd Ed., Prentice-Hall Signal Processing
Series, 870 pages, 1999.
[3] J. A. Pope, and F. H. C. Marriott, “Errors in the estimation of serial correlation”, Nature, 172, pp. 778, 1953.
[4] J. A. Pope, and F. H. C. Marriott, “Bias in the estimation of autocorrelation”, Biometrika 41 (3-4), pp. 390–402, 1954.
July 14, 2008 DRAFT
9

[5] M. G. Kendall, “Note on bias in the estimation of autocorrelation”, Biometrika 41 (3-4), pp. 403–404, 1954.
[6] D. B. Percival, “Three Curious Properties of the Sample Variance and Autocovariance for Stationary Processes with
Unknown Mean”, The American Statistician, 47 (4), pp. 274-276, 1993.
[7] M. H. Quenouille, “Notes on bias in estimation”, Biometrika 43 (3-4), pp. 353–360, 1956.
[8] M. H. Quenouille, “Approximate Tests of Correlation in Time-Series”, Journal of the Royal Statistical Society. Series B
(Methodological), 11 (1), pp. 68–84, 1949.
[9] J. Frank, Three-Dimensional Electron Microscopy of Macromolecular Assemblies: Visualization of Biological Molecules
in Their Native State, 2nd Ed., Oxford University Press, USA, 432 pages, 2006.
[10] S. Lawrence Marple, Digital Spectral Analysis: With Applications, Prentice Hall Signal Processing Series, 492 pages, 1987.
July 14, 2008 DRAFT
10