maximum likelihood binary detection in improper complex gaussian

MAXIMUM LIKELIHOOD BINARY DETECTION IN IMPROPER COMPLEX GAUSSIAN

NOISE

Amirhosse**in** S.Aghaei, Konstant**in**os N.Plataniotis, Subbarayan Pasupathy

E-mail: {aghaei, kostas, pas} @ comm.utoronto.ca

The Edward S. Rogers Sr. Dept. of Electrical and Computer Eng**in**eer**in**g

University of Toronto

ABSTRACT

In a wide range of communication systems, **in**clud**in**g DS-CDMA

and OFDM systems, the signal-of-**in**terest might be corrupted by an

**improper** [1] (also called non circularly symmetric [2]) **in**terfer**in**g

signal. This paper studies the **maximum** **likelihood** (ML) **detection**

of **b inary** signals

noise. Propos**in**g a new measure for noncircularity of **complex**

random variables, we will derive the ML decision rule and its performance

based on this measure. It will be shown that the ML detector

performs pseudo correlation [1] as well as conventional correlation

of the observation to the signals-of-**in**terest. As an alternative solution,

we will propose a filter for convert**in**g **improper** signals to

proper ones, called circularization filter, and will utilize it together

with a conventional matched-filter (MF) to construct an ML detector.

Index Terms— Maximum **likelihood** **detection**, **improper** **complex**,

Gaussian noise, circularization, matched filters

1. INTRODUCTION

S**in**ce the first studies of **improper** **complex** signals (also called non

circularly symmetric signals) by [1, 2] there has been a grow**in**g **in**terest

on characterization of such signals **in** various applications, especially

communication systems. The issue of **improper** **in**terference

was firstly studied by [3] **in** the context of multi-access **in**terference

(MAI) **in** synchronous DS-CDMA systems us**in**g BPSK modulation.

However, **improper** **in**terferences are not restricted to this case and

can arise **in** many other applications (e.g. [4, 5, 6]).

This paper considers the simple scalar case of **b inary** transmission

over a typical additive Gaussian noise channel, where the noise

is **in**dependent of the transmitted signal. Let’s assume that r is a

scalar 1 noisy observation of the unknown scalar signal-of-**in**terest s

which might take either of the determ**in**istic constant values s 0 or s 1

with equal probabilities accord**in**g to the follow**in**g hypotheses:

j

H0 : r = s 0 + n

(1)

H 1 : r = s 1 + n

where r, s 0, s 1, n ∈ C and the noise n has the follow**in**g variance

and pseudo-variance [1]:

σ 2 n E{|r − ¯r| 2 } , γ 2 n E{(r − ¯r) 2 }. (2)

1 In this paper we use the follow**in**g notation: lowercase letters for scalar

variables (e.g. z), boldface lowercase letters for vectors (e.g. z), and boldface

uppercase letters for matrices (e.g. Z). R and C represent the real and

**complex** doma**in**s. R{z} and I{z} represent the real and imag**in**ary parts of

a **complex** variable z. σ 2 z, γ 2 z , and σ xy represent the variance of z, pseudo

variance of z, and the covariance between x and y, respectively.

The pseudo variance γn 2 is **in** fact the covariance between n and n ∗ .

When n is uncorrelated from n ∗ (i.e. γn

2 = 0), it will be called

a proper or circularly symmetric noise. This paper focuses on the

problem of detect**in**g the transmitted signal, which is determ**in**istic

but unknown at the receiver, from the noisy observation r when the

noise n is **improper**. Although [7] has studied the mean square estimation

and **detection** of **improper** signals-of-**in**terest observed **in**

proper noise, the problem of ML **detection** **in** presence of **improper**

noise has not been studied **in** general yet, except some works which

have studied the special case of DS-CDMA systems with BPSK signal**in**g

(e.g. [4, 8]). However, the results of this paper are applicable

to any **b inary** signal

(e.g. antipodal or orthogonal signal**in**g).

In Section 3, after derivation of ML decision rule, we will prove

that the ML detector makes use of the correlation exist**in**g between

real and imag**in**ary parts of the **in**terfer**in**g noise to get the optimal

**detection** performance. Based on the results of Section 3, we will

propose a circularization filter for convert**in**g the **improper** additive

noise **in**to a proper one **in** Section 4. This filter can be used as a preprocess**in**g

step for conventional tools (MFs **in** this paper) to enable

them to deal with **improper** noises.

2. IMPROPER GAUSSIAN RANDOM VARIABLES

Let z be a scalar **complex** Gaussian random variable (RV) with Cartesian

representation of the form z = x + jy, where x and y are

two real valued RVs. It has been shown **in** [9] that z can be completely

characterized by either w z [x, y] T or v z √ 1 2

[z, z ∗ ] T ,

as these two random vectors relate to each other through a unitary

l**in**ear transformation:

w z = Tv z, where T = √ 1 » –

1 1

(3)

2 −j j

Moreover, complete second-order characterization of z requires knowledge

of the follow**in**g statistics [9]:

σ 2 z = σ 2 x + σ 2 y , γ 2 z = (σ 2 x − σ 2 y) + j(2 σ xy ), (4)

or knowledge of the follow**in**g covariance matrices:

C vzv z H E{(vz − v¯z)(vz − v¯z)H } = 1 »

–

σz

2 γz

2

2 (γz) 2 ∗ σz

2 (5)

» –

C wz wz T E{(w z − w¯z )(w z − w¯z ) T σ

2

} = x σ xy

. (6)

σ xy

Note that by know**in**g one of the covariance matrices **in** (5) or (6),

the other one can be determ**in**ed s**in**ce C wz wz

T = TC vz vz H TH .

σ 2 y

F**in**ally, it should be noted that the probability density function (pdf)

of a **complex** Gaussian RV z is def**in**ed **in** [9] as follows:

p z (z) =

=

˛ ˛

˛2πC wz w ˛− 1 n

2

z

T exp

˛ ˛

˛2πC vz v ˛− 1 n

2

z

H exp

− 1 ‚

2

− 1 ‚

2

‚w z − w¯z

‚

‚C

−1

wz w T z

‚v z − v¯z

‚

‚C

−1

vz v H z

o

o

, (7)

where ‚ ‚ x ‚C x H Cx.

The latter equation **in** (4) reveals that z is proper (γz

2 = 0) if

the power of z is equally distributed between its real and imag**in**ary

parts (σx 2 = σy) 2 and there exists no correlation between these two

parts (σ xy = 0). In this case, we will get γz 2 = 0 and σz 2 = 2σx, 2 and

the covariance matrices of (5) and (6) will be reduced to: C wzw z T =

C vzv z

H = (σz/2)I, 2 where I is the identity matrix. In the follow**in**g

def**in**ition, we will propose a new **complex** valued measure for

noncircularity (or **improper**ness) of RVs, which will be utilized **in**

analyz**in**g the ML detector **in** next sections.

Def**in**ition 1 Given a **complex** RV z, the noncircularity coefficient of

z, denoted by α z, is def**in**ed as the ratio of its pseudo-variance to its

variance:

„ σ

2

x − σy

2

α z γ2 z

σ 2 z

=

σ 2 x + σ 2 y

« „ «

2σxy

+ j

. (8)

σx 2 + σy

2

We call α z the noncircularity coefficient of z, **in** that its real and

imag**in**ary parts are normalized measures of the power difference

between x and y and the correlation between x and y, respectively.

Therefore, α z is **in**dependent from the changes **in** the power of z

and only conveys **in**formation about its properness. In fact, decomposition

of the pseudo-variance as γ 2 z = σ 2 z α z provides us with the

ability to dist**in**guish between changes **in** γ 2 z caused by chang**in**g the

power of z as opposed to changes caused by chang**in**g the properness

of z.

Theorem 1 The magnitude of the pseudo-variance of a **complex**

RV is upper bounded by the value of its conventional variance.

(Proof: Appendix-A) Theorem-1 implies that the newly def**in**ed coefficient

α z lies with**in** the unit circle (i.e. 0 ≤ |α z | ≤ 1), and the

extreme case for **improper**ness of z occurs when |γ 2 z| = σ 2 z.

3. MAXIMUM LIKELIHOOD BINARY DETECTION IN

THE PRESENCE OF IMPROPER NOISE

In this paper we consider the **b inary** hypothesis model of (1) and assume

that the detector has complete knowledge of the noise statistics

(i.e. ¯n as well as either C vn vn

H or C w n wn T ); hence, the noise pdf

p N(n) is available. Without loss of generality, we will assume that

n has zero mean (For nonzero mean noises, we can use r ′ = r − ¯n).

Under the **maximum** **likelihood** paradigm, the follow**in**g decision

rule could be def**in**ed:

L(r) = p R|S 1

(r|s 1 ) ŝ=s 1

≷ 1, (9)

p R|S0 (r|s 0 ) ŝ=s 0

By condition**in**g on s j, the detector assumes that s j is transmitted;

therefore, p R|Sj (r|s j ) = p N (r − s j ) s**in**ce s j is a determ**in**istic

constant value. By substitut**in**g

p R|Sj (r|s j ) =

˛ ˛

˛2πC vnv ˛− 1 n

2

n

H exp

− 1 ‚

2

‚v r − v sj

‚

‚C

−1

vnv H n

o

**in** (9) and tak**in**g the natural logarithm of both sides, the follow**in**g

decision rule will be resulted:

vs H 1

C −1

v n

v

vn

H r − 1 2 vH s 1

C −1

v n

v

vn H s1

ŝ=s 1

≷ vs H 0

C −1

v n

v

v

ŝ=s n

H r − 1

0

2 vH s 0

C −1

v n

v

vn H s0 , (10)

where v sj = √ 1

2

[s j, s ∗ j ] T , v n = √ 1 2

[n, n ∗ ] T , v r = √ 1 2

[r, r ∗ ] T ,

and

C −1

v n v H n

=

»

2

σz(1 2 − |α n | 2 )

1 −α n

−α ∗ n 1

By substitut**in**g these values **in** (10), we will get

∗

R

j`rs1 − 1 ´ 2 |s1|2 − `rs 1 − 1 ff

´

2 s2 1 α∗ n

ŝ=s 1

∗

≷ R

j`rs 0 − 1

ŝ=s 0

2 |s 0| 2 ´ − `rs 0 − 1 ff

´

2 s2 0 α∗ n . (11)

Fig. 1(a) shows the the block diagram of this detector. The term

rs ∗ j correlates the observation r with s j, while the the term rs j

pseudo correlates [1] r with s j. After the outputs of the correlator

and pseudo correlator are adjusted by 1 |s 2 j| 2 and 1 2 s2 j, respectively,

they will be superposed accord**in**g to the noncircularity coefficient

α n , and the ML decision will be based on the real part of this result.

For proper noises, the outputs of pseudo correlators for both s 1

and s 0 are ignored (multiplied by α n = 0), and the decision only

depends on the outputs of correlators accord**in**g to

∗

R

j`rs1 − 1 ´ff 2 |s 1| 2 ŝ=s 1

≷

ŝ=s 0

–

.

∗

R

j`rs0 − 1 ´ff

2 |s 0| 2

(12)

which is the decision rule of well-known conventional ML detectors

for proper noises.

By us**in**g (3), decision rule of (10) can also be rewritten as:

ws T 1

C −1

w nw

w

n

T r − 1 2 wT s 1

C −1

w nw

w

n T s1

ŝ=s 1

≷ ws T 0

C −1

w nw

w T r − 1

ŝ=s n

0

2 wT s 0

C −1

w nw

w

n T s0 , (13)

where w sj = [R{s j }, I{s j }] T , w n = [R{n}, I{n}] T , w r =

[R{r}, I{r}] T , and

»

C −1

2 1 − R{αn } −I{α

w n

=

n }

wn

T σn(1 2 − |α n | 2 ) −I{α n } 1 + R{α n }

The ML decision rule of (13) can be **in**terpreted as follows. R{r}

and I{r} can be considered as two observations of signals R{s j }

and I{s j} which are corrupted with noises R{n} and I{n}, respectively.

When n is **improper** (α n ≠ 0), the off-diagonal elements

of C wnw n T are not zero and R{n} and I{n} will become correlated

noises.

Therefore, the term ws T 1

C −1 w

w n wn

T r (= vs H j

C −1 v

v n vn H r ) can be

viewed as a whitened matched-filter for optimal jo**in**t **detection** of

R{s j } and I{s j } from the noisy observation vector of w r , when

the elements of the jo**in**tly Gaussian noise vector w n are correlated

to each other. This relationship between ML detector for **improper**

scalar noise n and the well-known whitened MF for correlated w n

motivates us to study another alternative implementation of optimal

ML detector **in** the next section.

–

.

(a) Comb**in**ation of Correlators and Pseudo

(b) Circularization Filter followed by

Fig. 1. Equivalent implementations of **b inary**

F**in**ally, it should be mentioned that for proper

rules of (10) and (13) reduce to

vs H 1

v r − 1 ŝ=s 1

2 vH s 1

v s1 ≷ vs H 0

v

ŝ=s 0

and

ws T 1

w r − 1 ŝ=s 1

2 wT s 1

w s1 ≷ ws T 0

w

ŝ=s 0

both of which make use of conventional MFs

4. CIRCULARIZATION FILTER FOLLOWED

MATCHED-FILTER

In Section 3, it was shown that an **improper**

represented by the noise vector w n with non

sequently, by convert**in**g the elements of w n

correspond**in**g noise will become proper. Thus,

concept of whiten**in**g filters **in** order to convert

**in**to a proper one as follows.

Def**in**ition 2 Given an **improper** **complex** RV

(or proprization) filter is def**in**ed as the filter

proper RV ez, us**in**g the follow**in**g **in**put/output

w ez = Λ − 2 1 U T w z,

where U and Λ are the matrices **in**clud**in**g eigenvectors

values of C wzw T , respectively (i.e. C z w zwz T

Accord**in**g to Appendix-B, this filter firstly

system and uses the eigenvectors of C wz wz T Correlators

Matched-Filter

ML detector.

noises, decision

− 1 2 vH s 0

v s0

r − 1 2 wT s 0

w s0 ,

(vs H j

v r or ws T j

w r).

BY

scalar noise n can be

i.i.d. elements. Con-

**in**to i.i.d elements, the

we can exploit the

an **improper** noise

z, the circularization

which converts z **in**to a

relationship:

(14)

and eigen-

).

rotates the coord**in**ate

system to remove the correlation between R{z} and I{z}. Afterwards,

the new coord**in**ates should be scaled separately by the **in**verse

of the square root of correspond**in**g eigenvalues to get a proper

ez with equal power on R{ez} and I{ez}.

Apply**in**g a circularization filter to the observation r, we get

w er = Λ − 2 1 U T w r = Λ − 2 1 U T w s + Λ − 1 2 U T w n = w es + w en ,

(15)

where es and en denote the transformed signal and the circularized

noise (C wen w T = I), and U and Λ are derived from eigen decomposition

of C wn w T . As it is shown **in** Fig. 1(b), the ML **detection**

en

n

of es can now be accomplished us**in**g conventional match**in**g of er to

es ∗ , i.e.,

j

R er es ∗ 1 − 1 ff

2 |es ŝ=s1

1| 2 ≷ R

jer es ∗ 0 − 1 ff

ŝ=s 0

2 |es 0| 2 (16)

or equivalently

w T es 1

w er − 1 ŝ=s 1

2 wT es 1

w es1 ≷ w T es 0

w er − 1

ŝ=s 0

2 wT es 0

w es0 . (17)

Note that both decision rules of (16) and (17) are equivalent to the

ML decision rule of (10), **in** that

R ˘er es ∗ ¯

j = w

T

esj w er = ws T j

C −1

w n

w

wn

T r = vs H j

C −1

v n

v

vn H r

Accord**in**gly, we will call the term vs H j

C −1 v

v nvn H r a circularized MF

to dist**in**guish it from conventional MF (vs H j

v r ).

In fact, circularization is a reversible preprocess**in**g step and preserves

all the data **in**cluded **in** v r . Thus, apply**in**g the ML criterion

to the preprocessed data will yield the same result as apply**in**g the

ML criterion to the orig**in**al data (see [10] Page 289).

F**in**ally, it should be noted that for proper noises (α n = 0), we

have C wen w T = (σn/2)I. 2 Consequently, the circularization filter

en

simply scales r with factor p 2/σn. 2 Therefore, for proper noises, it

can be omitted and the optimal detector will be the conventional MF.

5. PERFORMANCE

Let r ′ = x ′ + jy ′ be the representation of the observation r =

x + jy **in** a new coord**in**ate system which has the follow**in**g axes:

the l**in**e pass**in**g through the po**in**ts s 0 and s 1, and the perpendicular

bisector of the l**in**e segment s 0s 1. Assum**in**g that the angle between

the former l**in**e and x-axis is θ, we have [x ′ , y ′ ] T = R w (−θ) [x −

R{¯s}, y − I{¯s}] T , where R w (−θ) is def**in**ed **in** Appendix-B and

¯s = 1 (s 2 0 + s 1 ). It can be shown that when x ′ -axes is parallel to one

of the eigenvectors of C wn w T , the ML detectors of Sections 3,4 will

n

become equivalent to conventional MF. For all other cases, however,

the ML detector is not a conventional MF and outperforms it. It can

be proved that the probability of **detection** error is

s

!

2|s 1 − s 0 |

P e = Q

× 1 − |α r| cos(ϕ − 2θ)

σr

2 1 − |α r | 2

s

!

|s 1 − s 0 |

= Q

4σ 2 , (18)

x

(1 − ρ ′ ′2 )

‹ q

where ρ ′ = σ x ′ y ′ σ 2 x

σ 2 ′ y

and ϕ are the correlation factor of the

′

noise **in** the x ′ -y ′ coord**in**ates and the phase of the noncircularity

coefficient, respectively. Equation (18) together with Appendix-B

r

Error Probability

10 0 ρ’ = 0.00

10 −1

ρ’ = 0.25

ρ’ = 0.50

10 −2

ρ’ = 0.75

10 −3

10 −4

10 −5

10 −6

10 −7

10 −8

0 5 10 15

SNR (dB)

Fig. 2. Error Probability for different values of ρ ′ when σ 2 x ′ = σ2 y ′

and s 1 = −s 0 ∈ R .

reveals that for a fixed power constra**in**t, the optimal signal**in**g is

antipodal signal**in**g (s 1 = −s 0) along the eigenvector correspond**in**g

to the smallest eigenvalue of C wnw n T . As mentioned before, the ML

detector for this signal**in**g will become a conventional MF.

In order to illustrate how ML detectors of Section 3 outperform

conventional MFs, we will consider an antipodal signal**in**g such that

θ ≠ (ϕ − π)/2 and θ ≠ ϕ/2 (**in** these two cases the ML detector

is MF). For simplicity, let’s assume that θ = 0 and the power of

noise is equally distributed **in** x ′ and y ′ directions. Fig. 2 presents

the performance of the ML detector **in** this case for different values

of ρ ′ . It can be seen that for **improper** noises with higher ρ ′ , this

performance improves s**in**ce the optimal detector is provided with

more side **in**formation **in** y ′ .

Fig. 2 also illustrates the improvement which can be achieved

by us**in**g ML detectors **in**stead of conventional MFs. In x ′ -y ′ coord**in**ate

system, the imag**in**ary part of s ′ is zero for both s 1 and

s 0. Therefore, a conventional MF, which is based on a proper noise

model, assumes that there is no relevant data **in** the imag**in**ary part

of the observation (y ′ ); hence, ignores it. The performance of such

detector is well-known as Q( p |s 1 − s 0 | 2 /4σ 2 x

), which does not

′

change for different values of ρ ′ and is always equivalent to the case

of ρ ′ = 0 shown **in** Figure 2. However, us**in**g the correct **improper**

model for the noise, the ML detector takes advantage of the correlation

exist**in**g between the real and imag**in**ary parts of observation

r ′ to get optimal performance. Thus, its performance improves for

**improper** noises when ρ ′ **in**creases.

6. CONCLUSION AND REMARKS

In this paper, the ML **detection** of **b inary** signals

Gaussian noise was studied. Us**in**g the noncircularity coefficient of

the noise, the ML detector comb**in**es correlation and pseudo correlation

of r with s dur**in**g the **detection** process. As a more general

solution, we proposed that **in**stead of design**in**g a new ML detector

for **improper** noises, we can equip conventional detectors, which are

designed for proper noises, with a circularization filter. F**in**ally, it

was shown that the result**in**g ML detector **in** both approaches become

a conventional MF when the noise is proper or the signal**in**g is

along eigenvectors of C wn w T n .

Due to space limitations and for simplicity, this paper was restricted

to scalar RVs and **b inary** hypotheses; however, the more

general results for random vectors and M-ary hypotheses as well as

derivation of error probability will be presented **in** future works.

A. PROOF OF THEOREM-1

Accord**in**g to (4) we have σz 4 − |γz| 4 = 4[σxσ 2 y 2 − σxy].

2 From

Schwartz **in**equality (σxσ 2 y 2 − σxy 2 ≥ 0), it follows that 0 ≤ |γz| 4 ≤

σz 4 ; hence, 0 ≤ |γz| 2 ≤ σz.

2

B. EIGEN DECOMPOSITION OF C vZ v H Z AND C w Z w T Z

Let C vz vz H = QΛ 1Q H and C wz wz T = UΛ 2U T be the eigen decomposition

of covariance matrices def**in**ed **in** (6), (5), where Λ 1 and

Λ 2 are diagonal matrices **in**clud**in**g the eigen values of C vz vz

H and

C wz wz

T **in** non-**in**creas**in**g order, and Q and U are unitary matrices

**in**clud**in**g their correspond**in**g eigenvectors. Then

» –

1. Λ 1 = Λ 2 = Λ = σ2 z 1 + |αz| 0

2 0 1 − |α z|

» –

cos

ϕ

− s**in** ϕ

2. U = 2 2

s**in** ϕ cos ϕ = R ϕ

´

w`

2

2 2

"

#

3. Q = T H e j ϕ 2 0

U =

0 e −j ϕ T H = R ϕ

´

2

v`

TH

2

where ϕ denotes the phase of noncircularity coefficient α z, and R ϕ

w`

2

and R ϕ

´

v`

2 are unitary transformations on wz and v z , respectively,

both of which result **in** the rotation of z by phase ϕ . 2

C. REFERENCES

[1] F. D. Neeser and J. L. Massey, “Proper **complex** random processes

with applications to **in**formation theory,” IEEE Trans.

Inf. Theory, vol. 39, pp. 1293–1302, July 1993.

[2] B. Pic**in**bono, “On circularity,” IEEE Trans. Signal Process.,

vol. 42, pp. 3473–3482, Dec. 1994.

[3] Y. C. Yoon and H. Leib, “Maximiz**in**g SNR **in** **improper** **complex**

noise and applications to CDMA,” IEEE Commun. Lett.,

vol. 1, pp. 5–8, Jan. 1997.

[4] G. E. Bottomley, “CDMA downl**in**k **in**terference suppression

us**in**g I/Q projection,” IEEE Trans. Wireless Commun., vol. 2,

pp. 890–900, Sept. 2003.

[5] G. Taubock, “Noise analysis of DMT,” **in** Global Telecom.

Conf., 2003. IEEE, vol. 4, pp. 2136–2140, Dec. 2003.

[6] R. Nilsson, F. Sjoberg, and J. P. LeBlanc, “A rank-reduced

LMMSE canceller for narrowband **in**terference suppression

**in** OFDM-based systems,” IEEE Trans. Commun., vol. 51,

pp. 2126–2140, Dec. 2003.

[7] P. J. Schreier, L. L. Scharf, and C. T. Mullis, “Detection and

estimation of **improper** **complex** random signals,” IEEE Trans.

Inf. Theory, vol. 51, pp. 306–312, Jan. 2005.

[8] Y. Yoon and H.-M. Kim, “Maximum **likelihood** multiuser **detection**

of DS/CDMA signals **in** **improper** noise,” **in** 2005

Fifth Int. Conference on Inf., Commun. and Signal Process.,

pp. 224–228, Dec. 2005.

[9] B. Pic**in**bono, “Second-order **complex** random vectors and

normal distributions,” IEEE Trans. Signal Process., vol. 44,

pp. 2637–2640, Oct. 1996.

[10] H. L. Van Trees, Detection, Estimation, and Modulation Theory,

vol. I: Detection, Estimation, and L**in**ear Modulation Theory.

John Wiley and Sons, 1968.

´