Spectral Analysis and Time Series - max planck research school ...

A. Lagg – **Spectral** **Analysis**

**Spectral** **Analysis** **and** **Time** **Series**

Andreas Lagg

Part I: fundamentals

on time series

classification

prob. density func.

autocorrelation

power spectral density

crosscorrelation

applications

preprocessing

sampling

trend removal

Part II: Fourier series

definition

method

properties

convolution

correlations

leakage / windowing

irregular grid

noise removal

Part III: Wavelets

why wavelet

transforms?

fundamentals:

FT, STFT **and**

resolution problems

multiresolution

analysis: CWT

DWT

Exercises

d

e

t

e

r

m

A. Lagg – **Spectral** **Analysis**

Basic description of physical data

deterministic: described by explicit mathematical relation

xt=X cos

k t t

n

non o deterministic: no way to predict an exact value at a future instant of time

n

A. Lagg – **Spectral** **Analysis**

Classifications of deterministic data

Deterministic

Periodic

Nonperiodic

Sinusoidal

Complex Periodic

Almost periodic

Transient

A. Lagg – **Spectral** **Analysis**

Sinusoidal data

xt=X sin2 f 0

t

T =1/ f 0

time history

frequency spectrogram

A. Lagg – **Spectral** **Analysis**

Complex periodic data

xt=xt±nT

n=1,2,3,...

xt = a 0

2 ∑ a n

cos 2n f 1

t b n

sin 2 n f 1

t

(T = fundamental period)

time history

frequency spectrogram

A. Lagg – **Spectral** **Analysis**

Almost periodic data

xt=X 1

sin2 t 1

X 2

sin3t 2

X 3

sin50 t 3

no highest common divisor > infinitely long period T

time history

frequency spectrogram

A. Lagg – **Spectral** **Analysis**

Transient nonperiodic data

all nonperiodic data other than almost periodic data

xt={ Ae−at t≥0

0 t0

xt={ Ae−at cos bt t≥0

0 t0

xt={ A c≥t≥0

0 ct0

A. Lagg – **Spectral** **Analysis**

Classification of r**and**om data

R**and**om Data

Stationary

Nonstationary

Ergodic

Nonergodic

Special classifications

of nonstationarity

A. Lagg – **Spectral** **Analysis**

stationary / non stationary

collection of sample functions = ensemble

data can be (hypothetically) described by

computing ensemble averages (averaging over

multiple measurements / sample functions)

mean value (first moment):

x

t 1

= lim

N ∞

1

∑ N

N k=1

autocorrelation function (joint moment):

R x

t 1,

t 1

= lim

N ∞

x k

t 1

1

∑ N

N k=1

x k

t 1

x k

t 1

stationary: x

t 1

= x

, R x

t 1,

t 1

=R x

weakly stationary: x

t 1

= x

, R x

t 1,

t 1

=R x

A. Lagg – **Spectral** **Analysis**

ergodic / non ergodic

Ergodic r**and**om process:

properties of a stationary r**and**om process

described by computing averages over only one

single sample function in the ensemble

mean value of kth sample function:

x

k=lim

T ∞

autocorrelation function (joint moment):

T

1

∫ T

0

x k

tdt

R x

, k=lim

T ∞

T

1

∫ T

0

x k

t x k

tdt

ergodic:

x

k= x

, R x

, k=R x

Basic descriptive properties of r**and**om data

mean square values

propability density function

autocorrelation functions

power spectral density functions

(from now on: assume r**and**om data to be stationary **and** ergodic)

A. Lagg – **Spectral** **Analysis**

A. Lagg – **Spectral** **Analysis**

Mean square values

(mean values **and** variances)

describes general intensity of r**and**om data: 2 x

= lim

T ∞

rout mean square value:

x

rms

= x

2

T

1

∫ T

0

x 2 tdt

often convenient:

static component described by mean value:

x

= lim

T ∞

T

1

∫ T

0

xtdt

dynamic component described by variance:

x

2

= lim

T ∞

1

∫ T

T

0

[ xt− x

] 2 dt

2

st**and**ard deviation: x

= x

= x 2 − x

2

A. Lagg – **Spectral** **Analysis**

Probability density functions

describes the probability that the data will assume a value within some defined

range at any instant of time

for small x :

Prob[ xxt≤x x] = lim

T ∞

Prob[ xxt≤x x] ≈ p x x

probability density function

px =

lim

x 0

probability distribution function

P x

= Prob[xt≤x]

x

= ∫ pd

−∞

Prob[ xxt≤x x]

x

T x

= lim

x 0

k

T , T x = ∑

i=1

1

x [ lim

T ∞

t i

T x

T ]

Illustration: probability density function

sample time histories:

sine wave (a)

sine wave + r**and**om noise

narrowb**and** r**and**om noise

wideb**and** r**and**om noise

all 4 cases: mean value x

=0

A. Lagg – **Spectral** **Analysis**

A. Lagg – **Spectral** **Analysis**

Illustration: probability density function

probability density function

A. Lagg – **Spectral** **Analysis**

Autocorrelation functions

describes the general dependence of the data values at one time on the

values at another time.

R x

= lim

T ∞

T

1

∫ T

0

xt xtdt

x

= R x

∞ 2 x

= R x

0 (not for special cases like sine waves)

A. Lagg – **Spectral** **Analysis**

Illustration: ACF

autocorrelation functions (autocorrelogram)

A. Lagg – **Spectral** **Analysis**

Illustrations

autocorrelation function of a rectangular pulse

xt xt−

autocorrelation function

A. Lagg – **Spectral** **Analysis**

Power spectral density functions

(also called autospectral density functions)

describe the general frequency composition of the data in terms

of the spectral density of its mean square value

f ,

f f

mean square value in frequency range :

x 2 f , f = lim

T ∞

T

1

∫ T

0

definition of power spectral density function:

G x

f =

lim

f 0

x 2 f , f

f

xt , f , f

2 dt

portion of x(t) in (f,f+∆f)

f)

= lim

f 0

x 2 f , f ≈ G x

f f

1

f [ lim

T ∞

T

1

∫ T

0

important property: spectral density function is related to the

autocorrelation function by a Fourier transform:

G x

f = 2∫

−∞

∞

R x

e −i 2 f d = 4∫

0

∞

xt , f , f 2 dt]

R x

cos 2 f d

A. Lagg – **Spectral** **Analysis**

Illustration: PSD

power spectral density functions

Dirac delta function at f=f 0

sine wave

sine wave

+ r**and**om noise

narrowb**and**

noise

“white” noise:

spectrum is uniform over

all frequencies

broadb**and**

noise

A. Lagg – **Spectral** **Analysis**

Joint properties of r**and**om data

until now: described properties of an

individual r**and**om process

Joint probability density functions

joint properties in the amplitude domain

Crosscorrelation functions

joint properties in the time domain

joint probability measurement

Crossspectral density functions

joint properties in the frequency domain

A. Lagg – **Spectral** **Analysis**

Crosscorrelation function

describes the general dependence of

one data set to another

R xy

= lim

T ∞

T

1

∫ T

0

xt ytdt

similar to autocorrelation function

R xy

=0

functions are

uncorrelated

crosscorrelation measurement

typical crosscorrelation plot (crosscorrelogram):

sharp peaks indicate the existence of a correlation

between x(t) **and** y(t) for specific time displacements

A. Lagg – **Spectral** **Analysis**

Applications

Measurement of time delays

2 signals:

different offset

different S/N

time delay 4s

often used:

'discrete' cross correlation coefficient

lag =l, for l >=0:

R xy

l =

N −l

∑

k=1

∑ N

k=1

x k

−xy kl

−y

N

x k

−x 2 ∑ y k

−y 2

k=1

A. Lagg – **Spectral** **Analysis**

Applications

Detection **and** recovery from

signals in noise

3 signals:

noise free replica of the signal

(e.g. model)

2 noisy signals

cross correlation can be used to

determine if theoretical signal is

present in data

A. Lagg – **Spectral** **Analysis**

Preprocessing Operations

sampling considerations

trend removal

filtering methods

sampling

cutoff frequency (=Nyquist

frequency or folding frequency)

f c

= 1

2 h

A. Lagg – **Spectral** **Analysis**

Trend removal

often desirable before performing a

spectral analysis

Leastsquare method:

time series: ut

desired fit

(e.g. polynomial):

K

u=∑

k=0

N

LsqFit: minimize Qb=∑

set partial

derivatives to 0:

∂Q

n=1

∂ b l

=∑

n=1

K

K+1 equations: ∑

k=0

N

bk nh k

u n

−u n

2

n=1,2,... , N

2u n

−u n

[−nh l ]

N

b k ∑ nh kl

n=1

N

= ∑

n=1

u n

nh l

Digital filtering

A. Lagg – **Spectral** **Analysis**

end of part I ...

A. Lagg – **Spectral** **Analysis**

A. Lagg – **Spectral** **Analysis**

**Spectral** **Analysis** **and** **Time** **Series**

Andreas Lagg

Part I: fundamentals

on time series

classification

prob. density func.

autocorrelation

power spectral density

crosscorrelation

applications

preprocessing

sampling

trend removal

Part II: Fourier series

definition

method

properties

convolution

correlations

leakage / windowing

irregular grid

noise removal

Part III: Wavelets

why wavelet

transforms?

fundamentals:

FT, STFT **and**

resolution problems

multiresolution

analysis: CWT

DWT

Exercises

A. Lagg – **Spectral** **Analysis**

Fourier **Series** **and** Fast Fourier Transforms

St**and**ard Fourier series procedure:

if a transformed sample record x(t) is periodic with a period T p

(fundamental

frequency f 1

=1/T p

), then x(t) can be represented by the Fourier series:

where

∞

xt = a 0

2 ∑ a q

cos 2q f 1

t b q

sin 2q f 1

t

T

q=1

a q

= 2 ∫ xtcos 2q f

T 1

t dt q=0,1,2,...

0

T

b q

= 2 T ∫ 0

xtsin 2q f 1

t dt

q=1,2,3,...

A. Lagg – **Spectral** **Analysis**

Fourier series procedure method

sample record of finite length, equally spaced sampled:

x n

=xnh

n=1,2,... , N

Fourier series passing through these N data values:

xt = A 0

N /2

∑

q=1

A q

cos 2q t

T p

N /2−1

∑

q=1

B q

sin 2qt

T p

Fill in particular points: t=nh , n=1,2,... , N , T p

=Nh , x n

=xnh = ...

coefficients A q

**and** B q

: A 0

= 1 ∑ N n=1

N

x n

= x A N /2

= 1 N ∑ n=1

N

x n

cos n

N

A q

= 2 N ∑ n=1

N

B q

= 2 N ∑ n=1

x n

cos 2q n

N

x n

sin 2q n

N

inefficient & slow => Fast Fourier Trafos developed

q=1,2,... , N 2 −1

q=1,2,... , N 2 −1

A. Lagg – **Spectral** **Analysis**

Fourier Transforms Properties

Linearity

DFT

{x n

} ⇔

DFT

{y n

} ⇔

a{x n

}b{y n

}

DFT

⇔

{X k

}

{Y k

}

a{X k

}b{Y k

}

Symmetry {X k

} = {X −k

R{X k

} is even

∗ }

I{X k

} is odd

Circular time shift

DFT

{x n−n0

} ⇔

{e i k n DFT

0

y n

} ⇔

{e −i k n 0

X k

}

{Y k−k0

}

A. Lagg – **Spectral** **Analysis**

Using FFT for Convolution

r * s ≡ ∫

−∞

∞

r st−d

Convolution Theorem:

r * s ⇔ FT R f S f

Fourier transform of the convolution

is product of the individual Fourier

transforms

discrete case:

N /2

r * s j

≡ ∑

k=−N /21

Convolution Theorem:

N /2

∑

k=−N /21

s j−k

r k

s j−k

r k

⇔ FT R n

S n

(note how the response function for negative times is wrapped

around **and** stored at the extreme right end of the array)

constraints:

duration of r **and** s are not the same

signal is not periodic

original data

response function

convolved data

A. Lagg – **Spectral** **Analysis**

Treatment of end effects by zero padding

constraint 1:

constraint 2:

simply exp**and** response function to length N by padding it with zeros

extend data at one end with a number of zeros equal to the **max**.

positive / negative duration of r (whichever is larger)

A. Lagg – **Spectral** **Analysis**

FFT for Convolution

1. zeropad data

2. zeropad response function

(> data **and** response function have N elements)

3. calculate FFT of data **and** response function

4. multiply FFT of data with FFT of response function

5. calculate inverse FFT for this product

Deconvolution

> undo smearing caused by a response function

use steps (13), **and** then:

4. divide FFT of convolved data with FFT of response

function

5. calculate inverse FFT for this product

A. Lagg – **Spectral** **Analysis**

Correlation / Autocorrelation with FFT

definition of correlation / autocorrelation see first lecture

Corr g , h = g* h = ∫

−∞

∞

Correlation Theorem:

Corr g , h ⇔ FT

AutoCorrelation:

G f H * f

Corr g , g ⇔ FT ∣G f ∣ 2

discrete correlation theorem:

Corr g , h j

≡

N −1

∑

k=0

⇔ FT G k

H k

*

g jk

h k

gthd

A. Lagg – **Spectral** **Analysis**

Fourier Transform problems

spectral leakage

T =nT p

T ≠nT p

A. Lagg – **Spectral** **Analysis**

reducing leakage by windowing (1)

Applying windowing (apodizing) function to data record:

xt = xtwt (original data record x windowing function)

x n

= x n

w n

No Window:

Bartlett Window:

Hamming Window:

wt=1−

∣t−T /2∣

T /2

wt=0.540.46∗cost

Blackman Window:

Hann Window:

Gaussian Window:

wt =

0.420.5∗cost

0.08 cos2 t

wt=cos 2 t

2 wt=exp−0.5a t 2

A. Lagg – **Spectral** **Analysis**

reducing leakage by windowing (2)

without windowing

with Gaussian windowing

A. Lagg – **Spectral** **Analysis**

No constant sampling frequency

Fourier transformation requires constant sampling (data points at

equal distances)

> not the case for most physical data

Solution: Interpolation

linear:

inear interpolation between y **and** y k k+1

IDL> idata=interpol(data,t,t_reg)

quadratic:

quadratic interpolation using y k1 , y k

**and** y k+1

IDL> idata=interpol(data,t,t_reg,/quadratic)

leastsquare quadratic

leastsquare quadratic fit using y k1 , y k

, y k+1 **and** y k+2

IDL> idata=interpol(data,t,t_reg,/lsq)

spline

IDL> idata=interpol(data,t,t_reg,/spline)

IDL> idata=spline(t,data,t_reg[,tension])

important:

interpolation changes

sampling rate!

> careful choice of

new (regular) time grid

necessary!

Fourier Transorm on irregular gridded data Interpolation

original data: sine wave + noise

FT of original data

irregular sampling of data (measurement)

interpolation: linear, lsq, spline, quadratic

'resampling'

FT of interpolated data

A. Lagg – **Spectral** **Analysis**

A. Lagg – **Spectral** **Analysis**

Noise removal

Frequency threshold (lowpass)

make FT of data

set high frequencies to 0

transorm back to time

domain

A. Lagg – **Spectral** **Analysis**

Noise removal

signal threshold for

weak frequencies (dBthreshold)

make FT of data

set frequencies with amplitudes

below a given threshold to 0

transorm back to time domain

A. Lagg – **Spectral** **Analysis**

Optimal Filtering with FFT

normal situation with measured data:

underlying, uncorrupted signal u(t)

+ response function of measurement r(t)

= smeared signal s(t)

+ noise n(t)

= smeared, noisy signal c(t)

st = ∫

−∞

∞

r t−ud

ct = stnt

estimate true signal u(t) with:

U f = C f f

R f

f ,t = optimal filter

(Wiener filter)

A. Lagg – **Spectral** **Analysis**

Calculation of optimal filter

reconstructed signal **and** uncorrupted signal should be close in leastsquare sense:

> minimize

> minimize

∫−∞

∞

∣ ut−ut∣ 2 dt = ∫

−∞

∞

∣ U f −U f ∣ 2 df

⇒

∂

∣ ∂ f

⇒ f =

[S f N f ] f

R f

∣S f ∣ 2

∣S f ∣ 2 ∣N f ∣ 2

− S f

R f ∣2

= 0

additional information:

power spectral density can often

be used to disentangle noise

function N(f) from smeared

signal S(f)

A. Lagg – **Spectral** **Analysis**

Using FFT for Power Spectrum Estimation

discrete Fourier transform of c(t)

> Fourier coefficients:

N−1

C k

=∑

j=0

c j

e 2 i j k / N

k=0,... , N−1

> periodogram estimate of

power spectrum:

P0 = P f 0

= 1 N 2 ∣C 0 ∣ 2

Periodogram

P f k

=

1

N 2 [∣C k ∣ 2 ∣C N−k ∣ 2 ]

P f c

= P f N /2

= 1 N 2 ∣C N /2 ∣ 2

end of FT

A. Lagg – **Spectral** **Analysis**

A. Lagg – **Spectral** **Analysis**

**Spectral** **Analysis** **and** **Time** **Series**

Andreas Lagg

Part I: fundamentals

on time series

classification

prob. density func.

autocorrelation

power spectral density

crosscorrelation

applications

preprocessing

sampling

trend removal

Part II: Fourier series

definition

method

properties

convolution

correlations

leakage / windowing

irregular grid

noise removal

Part III: Wavelets

why wavelet

transforms?

fundamentals:

FT, STFT **and**

resolution problems

multiresolution

analysis: CWT

DWT

Exercises

A. Lagg – **Spectral** **Analysis**

Introduction to Wavelets

why wavelet transforms?

fundamentals: FT, short term FT **and** resolution

problems

multiresolution analysis:

continous wavelet transform

multiresolution analysis:

discrete wavelet transform

A. Lagg – **Spectral** **Analysis**

Fourier: lost time information

6 Hz, 4 Hz, 2 Hz, 1 Hz 6 Hz + 4 Hz + 2 Hz + 1 Hz

Solution: Short **Time** Fourier Transform

(STFT)

perform FT on 'windowed' function:

example: rectangular window

move window in small steps over data

perform FT for every time step

STFT f ,t ' = ∫[ xtt−t ']e −i 2 f t dt

t

A. Lagg – **Spectral** **Analysis**

A. Lagg – **Spectral** **Analysis**

Short **Time** Fourier Transform

STFT

STFTspectrogram shows both time **and**

frequency information!

A. Lagg – **Spectral** **Analysis**

Short **Time** Fourier Transform: Problem

narrow window function > good time resolution

wide window function > good frequency resolution

Gaussfunctions as

windows

A. Lagg – **Spectral** **Analysis**

Solution: Wavelet Transformation

time vs. frequency resolution is intrinsic problem (Heisenberg Uncertainty Principle)

approach: analyze the signal at different frequencies with different resolutions

> multiresolution analysis (MRA)

Continuous Wavelet Transform

similar to STFT:

signal is multiplied with a

function (the wavelet)

transform is calculated

separately for different segments

of the time domain

but:

the FT of the windowed signals are

not taken

(no negative frequencies)

The width of the window is changed

as the transform is computed for

every single spectral component

A. Lagg – **Spectral** **Analysis**

Continuous Wavelet Transform

CWT x = x , s = 1

∣s∣ ∫ t

... translation parameter , s ... scale parameter

t ... mother wavelet = small wave

*

t−

xt

s

dt

mother wavelet:

finite length (compactly

supported) >'let'

oscillatory >'wave'

functions for different

regions are derived from

this function > 'mother'

scale parameter s replaces frequency in STFT

A. Lagg – **Spectral** **Analysis**

The Scale

similar to scales used in maps:

high scale = non detailed global view (of the signal)

low scale = detailed view

in practical applications:

low scales (= high frequencies)

appear usually as short bursts or

spikes

high scales (= low frequencies) last

for entire signal

scaling dilates (stretches out) or

compresses a signal:

s > 1 > dilation

s < 1 > compression

A. Lagg – **Spectral** **Analysis**

Computation of the CWT

signal to be analyzed: x(t), mother wavelet: Morlet or Mexican Hat

start with scale s=1 (lowest scale,

highest frequency)

> most compressed wavelet

shift wavelet in time from t 0

to t 1

increase s by small value

shift dilated wavelet from t 0

to t 1

repeat steps for all scales

A. Lagg – **Spectral** **Analysis**

CWT Example

signal x(t)

axes of CWT: translation **and**

scale (not time **and** frequency)

translation > time

scale > 1/frequency

A. Lagg – **Spectral** **Analysis**

**Time** **and** Frequency Resolution

every box corresponds to a value of the wavelet

transform in the time frequency plane

all boxes have constant area

Δf Δt t = const.

low frequencies: high resolution

in f, low time resolution

high frequencies: good time

resolution

STFT: time **and** frequency

resolution is constant (all boxes

are the same)

A. Lagg – **Spectral** **Analysis**

Wavelets: Mathematical Approach

WLtransform:

CWT x = x , s = 1

∣s∣ ∫ t

*

t−

xt

s

dt

Mexican Hat wavelet:

t =

−t 2

1

s e

2 2

3

t2

2 −1

Morlet wavelet:

t = e i a t e − t2

2

inverse WLtransform:

xt =

1 2 ∫∫ x

, s 1

c s s t−

2 s

d ds

admissibility

condition:

c

={2∫

−∞∞

∣∣ 2

∣∣

1/2

d } ∞ with

FT

⇔

t

A. Lagg – **Spectral** **Analysis**

Discretization of CWT: Wavelet **Series**

> sampling the time – frequency (or scale) plane

advantage:

sampling high for high frequencies (low scales)

scale s **and** 1 rate N 1

sampling rate can be decreased for low

frequencies (high scales)

scale s 2

**and** rate N 2

N 2

= s 1

s 2

N 1

N 2

= f 2

f 1

N 1

continuous wavelet

discrete wavelet

, s

= 1 s t−

s t = s − j/2 −

j , k 0

s j 0

t−k 0

, j , k

x

j , k =∫

t

xt = c ∑

j

*

xt j , k

tdt

∑

k

x

j , k j , k

t

orthonormal

WLtransformation

reconstruction of

signal

A. Lagg – **Spectral** **Analysis**

Discrete Wavelet Transform

(DWT)

discretized continuous wavelet transform is only a sampled version of the CWT

The discrete wavelet transform (DWT) has significant advantages for

implementation in computers.

excellent tutorial:

http://users.rowan.edu/~polikar/WAVELETS/WTtutorial.html

IDLWavelet Tools:

IDL> wv_applet

Wavelet expert at MPS:

Rajat Thomas

end of Wavelets

A. Lagg – **Spectral** **Analysis**

A. Lagg – **Spectral** **Analysis**

Exercises

Part I: Fourier **Analysis**

(Andreas Lagg)

Instructions:

http://www.linmpi.mpg.de/~lagg

Part II: Wavelets

(Rajat Thomas)

Seminar room

**Time**: 15:00

A. Lagg – **Spectral** **Analysis**

Exercise: Galileo magnetic field

data set from Galileo magnetometer

(synthesized)

file: gll_data.sav, contains:

total magnetic field

radial distance

time in seconds

your tasks:

Which ions are present?

Is the time resolution of

the magnetometer

sufficient to detect

electrons or protons?

Background:

Tips:

restore,'gll_data.sav'

use IDLFFT

remember basic plasma

physics formula for the

ion cyclotron wave:

gyro

= q B

m ,

f gyro = gyro

2

If the density of ions is high enough they will excite ion cyclotron waves

during gyration around the magnetic field lines. This gyration frequency

only depends on mass per charge **and** on the magnitude of the magnetic

field.

In a lowbeta plasma the magnetic field dominates over plasma effects.

The magnetic field shows only very little influence from the plasma **and**

can be considered as a magnetic dipole.

http://www.sciencemag.org/cgi/content/full/274/5286/396

A. Lagg – **Spectral** **Analysis**

Literature

R**and**om Data: **Analysis** **and** Measurement Procedures

Bendat **and** Piersol, Wiley Interscience, 1971

The **Analysis** of **Time** **Series**: An Introduction

Chris Chatfield, Chapman **and** Hall / CRC, 2004

**Time** **Series** **Analysis** **and** Its Applications

Shumway **and** Stoffer, Springer, 2000

Numerical Recipies in C

Cambridge University Press, 19881992

http://www.nr.com/

The Wavelet Tutorial

Robi Polikar, 2001

http://users.rowan.edu/~polikar/WAVELETS/WTtutorial.html