General asymptotic analysis of the generalized ... - IEEE Xplore

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002

2617

General Asymptotic Analysis of the Generalized Likelihood Ratio Test for a Gaussian Point Source Under Statistical or Spatial Mismodeling Jonathan Friedmann, Eran Fishler, Member, IEEE, and Hagit Messer, Fellow, IEEE

Abstract—This paper investigates the robustness of the generalized likelihood ratio test (GLRT) for a far-field Gaussian point source. Given measurements from an array of sensors, the performance of the GLRT under two types of common modeling errors is investigated. The first type is spatial mismodeling, which relates to errors due to multipath effects or errors in the assumed number of sources, i.e, deviation from the single point source assumption. The second type is statistical mismodeling, which relates to errors due to non-Gaussianity in either the noise or the signal, i.e., deviation from the Gaussian assumption. It is shown that for some types of modeling errors, the detector’s performance improves, and general conditions for such an improvement are found. Moreover, for both types of errors, the change in performance is analyzed and quantified. This analysis shows that for a distributed source with small spatial spreading, the degradation in performance is significant, whereas for a constant modulus point source, the performance improves. Simulations of various cases are shown to verify the analytical results.

I. INTRODUCTION A. Motivation

P

ASSIVE array signal processing has gained considerable interest during the last three decades. The problems under study concern the extraction of information from measurements using an array of sensors. The measurements are assumed to be a superposition of plane waves corrupted by noise. Given the observations of the sensor outputs, the objective is to estimate the unknown parameters associated with the wavefronts. Traditionally, the unknown parameters are divided into three groups, and consequently, the literature is concerned with three types of problems: 1) Detection and determination of the number of sources: In this problem, we want to decide whether the measurements consist of noise only (the null hypothesis) or of a number of sources corrupted by noise (the alternate hypothesis). Once the null hypothesis is rejected, it is of interest to determine the exact number of sources impinging on the array (see, among many others, [1]–[3]). Manuscript received November 27, 2001; revised June 24, 2002. The associate editor coordinating the review of this paper and approving it for publication was Prof. S. M. Jesus. J. Friedmann was with the Department of Electrical Engineering—Systems, Tel Aviv University, Tel Aviv, Israel. He is now with Metalink, Yakum, Israel (e-mail: [email protected]). E. Fishler was with the Department of Electrical Engineering—Systems, Tel Aviv University, Tel Aviv, Israel. He is now with the Department of Electrical Engineering, Princeton University, Princeton NJ, 08544 USA (e-mail: [email protected]). H. Messer is with the Department of Electrical Engineering—Systems, Tel Aviv University, Tel Aviv, Israel (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2002.804098.

2) Estimation of the sources spectral parameters: For example, there are the sources powers [4]. This problem is usually of interest for the communication community. 3) Estimation of the sources spatial parameters: For example, we have the sources’ bearing or range (see, among many others, [4] and [5]). In general, problems 2) and 3) are approached by assuming that the solution of problem 1) is known, i.e., the number of sources is known. However, in some cases, all three problems are solved simultaneously, e.g., [6]. The most common approach for estimating the number of sources (problem 1) is to use an information theoretic criteria-based estimator like the minimum description length (MDL) or the Akaike information criterion (AIC). In [6], it is suggested that we use the MDL estimator and combine the three problems 1)–3) into a single estimation problem, resolving for all unknown parameters simultaneously. In spectrum monitoring systems, which employ broadband scanning schemes, a vast number of narrowband channels (up to a few million) are scanned every second. In these systems, a fast and simple detection procedure is employed. If the null hypothesis “noise only” is rejected, then a more powerful and complicated algorithm for estimating the number of sources and their bearings is employed on the relevant channel. Other systems that process huge amounts of data in a short time make use of a similar solution: First, a simple and computationally efficient detection algorithm is used. Then, once a signal is detected, computationally demanding algorithms are used to estimate the number of sources and their unknown parameters, like range, bearing, and so on. One of the most popular algorithms used for the detection at the first step is the generalized likelihood ratio test (GLRT). This commonly used detector assumes that both the source (if present) and the additive noise are Gaussian. Although planned for a single Gaussian point source embedded in additive white Gaussian noise, this detector usually operates in a different environment, i.e., when there exists more than one source or when the additive noise is non-Gaussian. Such a mismatch between the assumed model and the actual environment in which the detector operates may cause degradation in performance and gives rise to the question of the detector’s robustness. In this paper, we provide a systematic performance and robustness analysis of the GLRT. Specifically, we explore the performance of the GLRT under statistical and spatial modeling errors. This analysis is essential for understanding the advantages and disadvantages of the detector in scenarios where the assumed model is most likely to be incorrect.

1053-587X/02$17.00 © 2002 IEEE

2618

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002

B. Problem Formulation sensors, and denote by the reAssume an array of ceived -dimensional signal vector at instant . A common model for the received signal vector is (1) is an -dimensional vector, and lies on the where }. The vector is usually array manifold { called the array response vector or the steering vector. It represents the array complex response to a narrowband point source , and situated at an angle . The source signal is denoted by is an 1 additive noise vector. Finally, in addition, asand are ergodic, independent random vector sume that processes. Consider the following detection problem:

(2) , only noise exists, whereas under Under the null hypothesis , in addition to the noise, there exthe alternate hypothesis ists a point source at bearing . Based on independent array , we want to decide whether to acsnapshots or to accept the altercept the null hypothesis (noise only), . nate hypothesis (signal plus noise) C. GLRT The simplest variant of the detection problem in (2) is the case where under both hypotheses, the probability density function (pdf) of the measurements is known. In this case, the problem is a simple hypotheses testing problem, and the optimal (in the Neyman-Pearson sense) likelihood ratio test (LRT) can be implemented. This detector turns out to be the beamformer steered at direction , followed by a simple linear filter (Echardt) [7]. For cases in which some of the parameters are unknown, the detection problem in (2) becomes a composite hypotheses testing problem. Although not necessarily optimal, the GLRT is widely used in such problems. In many problems, including the problem at hand, the GLRT is a constant false alarm rate (CFAR) detector. This desirable property is one of the main reasons why this detector was investigated extensively over the last three decades. the pdf of the received vectors Denote by under hypothesis , where stands for denotes the unknown parameters under hypothesis , and the parameter space of . The GLRT for deciding whether to or to accept is given by accept

Assume the following. 1) The additive noise is a circular, zero mean, white (both temporally and spatially), complex Gaussian random vector process with unknown variance , i.e., , where 0 and are, respectively, a zero vector and the identity matrix of appropriate dimensions. is a circular, zero mean, temporally white, 2) The signal complex Gaussian random process with unknown vari. ance , i.e., 3) The signal and the noise are statistically independent, i.e., . 4) The source bearing (if the source exists) is unknown. Given assumptions 1)–4), each snapshot is a circular, zero mean, white, complex Gaussian random vector with counder the null hypothesis, and variance matrix under the alternate hypothesis. Thus, the detection problem (2) is now given by

(4) Note that under the null hypothesis, the unknown parameter contains only the unknown noise level, whereas under vector conthe alternate hypothesis, the unknown parameter vector tains the unknown source bearing, the unknown source power, and the unknown noise level, i.e.,

(5) Under the Gaussian assumptions, the GLRT of (3) takes the form of

Tr Tr Tr Tr

(6)

where is the empirical correlation matrix, i.e., , and is the correlation matrix of the received measurements, assuming that the un. Once again, known parameter vector is equal Tr is the ML estimate of the unknown parameter vector given the th hypothesis. Plugging the exact distributions in (4) into (6) results in

(3) Tr stands for the ML estimates of the unwhere known parameters under hypothesis , i.e., , and is the detector threshold.

Tr (7)

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST

where

Tr

, and

Tr . In the sequel, we refer to the detector given by (7) as the Gaussian generalized likelihood ratio test (GGLRT). Note that the GGLRT requires only a simple one-dimensional (1-D) search. In addition, note that for the problem of interest, is negative, or else, the false alarm the threshold probability is strictly one. This is since contains , which means that . D. Summary of Paper’s Main Results It is well known that in bearing estimation problems, the ML estimate of the bearing of a Gaussian source is robust. That is, under mild regularity conditions, the performance of the ML estimator for the bearing is insensitive to the source’s distribution [8]. This property is usually termed as “asymptotic robustness.” However, a similar study of the robustness of the GGLRT has yet to be conducted. This paper carries out such an analysis, investigating the robustness properties of the GGLRT (7). The first robustness study of the GLRT statistic of (3) appeared in [9]. In this study, the asymptotic distribution of the GLRT statistic, given the null hypothesis for nested hypotheses testing problem, was computed. The result was a generalization of Wilks theorem for the case of mismatch. In later studies (see, among many others, [10]–[12]), other theoretical aspects of the robustness of the GLRT, as well as other detectors, were studied. These studies are generally not concerned with some specific model, and thus, the results obtained do not provide insight into specific problems like the one at hand. In [1] and [13], the problem of estimating the number of sources is studied using statistical tools similar to the one used in this study. The study in [1] and [13] provides an insight into the performance of the MDL estimator when various sets of probabilistic assumptions are used. The improvement in performance due to the utilization of a priori information is studied as well. This paper investigates the behavior of the GGLRT under two types of modeling errors: statistical and spatial. Spatial modeling errors occur when the point source assumption is violated, i.e., when the frequency band contains several sources from different directions instead of a single one or when the received signal is a distributed source instead of a point source. Statistical modeling errors correspond to deviations from the Gaussian assumptions, i.e., when the additive noise and/or the signal are non-Gaussian. The paper provides a comprehensive analysis of the performance of the GGLRT in various scenarios. The results of this analysis are closed-form expressions for the asymptotic variation in the probability of detection for various types of modeling errors. Specifically, we start with analyzing the performance of the GGLRT with no modeling errors. The results obtained for this case are used as our baseline for comparison with the performance of the GGLRT under various modeling errors. Next, the performance of the GGLRT is examined when the source is a distributed source instead of the assumed point source, i.e., spatial mismatch. A closed-form expression for the degradation

2619

in performance as a function of the source SNR and the angular spreading is obtained. The cases of non-Gaussian noise or non-Gaussian source are also examined, and closed-form expressions for the performance change due to the various deviations from the Gaussian assumptions are provided as well. It is shown both theoretically and empirically that the existence of statistical mismatch might result in performance improvement compared with the case when no mismatch exists. This property is not only new but is rather surprising as well. Although, usually, mismodeling errors cause some degradation in performance, this is not the case in this detection problem. In fact, this paper gives exact conditions for cases in which the detection probability increases. The rest of the paper is organized as follows. Section II gives a general asymptotic analysis of the GGLRT. The case of a general type of model mismatch is examined in this section as well. Section III examines spatial mismodeling errors, whereas Section IV examines statistical mismodeling errors. To validate analytical results, simulations are presented at the end of each section. Finally, Section V summarizes the paper. II. GENERAL ANALYSIS OF THE GGLRT A. Analysis Based on the theory of misspecified models [14], this subsection presents an asymptotic analysis of the GGLRT evaluated under very general conditions. The result is then used to analyze the performance of the GGLRT, given normally distributed measurements with possible mismodeling errors. the distribution of a single snapshot . InDenote by enables one to consider cases where the actual troducing distribution of is different from its assumed distribution under or . In addition, assume that there exist two unique, either and (hereafter referred to as stationary interior points and , respectively, which minimize points) in Tr (8) is the true correlation matrix of the where , and measurements, . In addition, assume that for all matrices , the following two conditions are met: (9) (10) is the expectation of where with respect to the pdf . Theorem 1: Assume that the alternate hypothesis is true. Given the conditions stated above, the GGLRT statistic of (7) is asymptotically normally distributed, with mean and , where variance , i.e.,

(11)

2620

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002

Since is a consistent estimator of [14], is easily com, i.e., puted by replacing with its mean value (12) Tr Proof: See Appendix A. Note that Theorem 1 does not contradict the Wilks theorem [15]. The two parts of the Wilks theorem state that is true, under some regularity conditions, whenever is chi-squared distributed, whereas whenever is true, is noncentral chi-squared distributed. However, these reguor larity conditions require that either , which is not true under Theorem 1. In some sense, Theorem 1 extends the Wilks theorem. A detector’s performance is measured by the probability of detection given a predetermined false alarm probability. Theorem 1 provides a way to evaluate the asymptotic probability of detection of the GGLRT as a function of the threshold (and, therefore, as a function of the false alarm probability) set by the detector

(15)

Using similar arguments

(16) and , it is Note that in order to compute the ML estimates assumed that under both the null and the alternate hypothesis, the received vector is normally distributed. It is now straightforward to compute the mean and the variof the GGLRT statistic (full evaluation in Appendix B) ance

(13) complementary error function, , and , are given by (11) and (12), respectively. Thus, in order to quantify the performance of the GGLRT under various modeling errors for a given threshold, it suffices to examine , . In the following, we present an asymptotic analysis of the GGLRT (7) under some specific modeling errors. This analysis results in simple, closed-form expressions for the mean and the variance of the GGLRT statistic ( of (7)) given the alternate hypothesis. These simple expressions can be used with (13) to obtain approximations for the probability of detection in various cases.

where

is

the

B. Case of Normally Distributed Measurements Assume that the measurements are independent and identically distributed (iid), zero mean complex normal random vec, which can be written as tors with correlation matrix (14)

(17)

(18)

C. Case of No Modeling Errors The general results in (17) and (18) allow the evaluation of the performance of the GGLRT in various scenarios. The first in (14). is the case of no modeling errors, i.e., Assume that the measurements are an iid, zero mean, complex normal random vector with correlation matrix equal to . In this case, the eigenvalues of the correlation matrix are given by (19)

and { } are the eigenvalues and where eigenvectors of the received vector correlation matrix, respecis the normalized steering tively, and vector. Note that under the alternate hypothesis (with no modeling errors), the GGLRT assumes that the correlation matrix of is given by (14) with . the received vector for some , there exists a Whenever modeling error. Analyzing the performance of the GGLRT requires the evaland . The ML estimate of uation of the stationary points is [16] Tr

It is now straightforward to show that the asymptotic distribution of the GGLRT statistic of (7) is given by (20) is the per-element SNR. Clearly, this means where that when there are no modeling errors, the asymptotic performance of the GGLRT is strictly a function of the SNR and the number of sensors. It is well known that in the corresponding bearing estimation problem, the mean square error (MSE) of the ML estimate for the bearing decreases as the SNR increases [5]. It seems natural

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST

2621

Most of the existing literature model the statistics of the observed data in the case of a distributed source by a zero mean, white, complex Gaussian random vector (for full derivation, see, e.g., [23]) (21) A number of parametric models have been used to describe the . Most involve the use of some channel covariance matrix parameterized form of the angular power density function, e.g., [17], [23] (22) is the average power intensity of the received signal where include from direction . Common models of Fig. 1. Performance of the GGLRT for a point source as a function of the SNR for various probabilities of false alarm.

that in the described detection problem, for any given false alarm probability, an increase in the SNR will imply an increase in the detection probability. The following proposition proves that this is indeed the case when using the GGLRT. Proposition 1: For any given false alarm probability, the probability of detection is an asymptotically monotonic increasing function of the SNR. Proof: See Appendix C. D. Simulations To validate proposition 1, simulations of a uniform linear array (ULA) with six sensors at half wavelength spacing were carried out. One hundred independent snapshots of a white, were taken. Fig. 1 Gaussian point source at angle depicts the probability of detection as a function of the SNR for false alarm probabilities equal to 10 and 10 . The SNR . The theowas defined at the element level, i.e., SNR retical curves were evaluated using (13) and (20), whereas the empirical curves were calculated based on 1000 Monte Carlo runs. The empirical probability of detection was calculated as the number of runs in which the source was detected divided by 1000. The figure shows a very good fit between the empirical and the theoretical results.

Gaussian otherwise

Uniform (23)

is assumed to be the source The mean angle parameter bearing, and the spatial spreading parameter is denoted by . Since the point source assumption is sometimes incorrect, it is of interest to investigate the performance of the GGLRT under such a modeling error. In what follows, we investigate the performance of the GGLRT when the source is a scattered source with small spreading. It is proved that the performance of the GGLRT deteriorates as the spatial spreading of the source increases. In [24], it is shown that small spreading implies a shift of energy from the first eigenvalue of the channel covariance matrix to the second. As the spreading increases, more energy is shifted with some being drifted to the next eigenvalue. In the sequel, we assume that energy is shifted from the first eigenvalue to the are given second only, i.e., the eigenvalues of by

(24) III. ANALYSIS OF THE GLRT WITH SPATIAL MISMODELING A. Analysis Perhaps the most restrictive assumption of the model in (2) is the point source assumption. Frequently, there exist local scatters at the vicinity of the emitter that create multipath effects. Such may be the case in radar, sonar, astronomy, or wireless communications applications (e.g., [17]). As a result, bearing estimation for a distributed source has begun to attract interest in the literature (see, among others, [18]–[21]). The spatial extent of a distributed source is typically characterized by some type of a parametric model. The models in the litrature have formed the basis of a variety of recently reported bearing estimation techniques and performance studies, e.g., [22].

, and is the decrease (increase) in the first where (second) eigenvalue with respect to a point source with the same SNR. Note that this assumption corresponds to small spreading. Analyzing the probability of detection in such a scenario can be done by plugging (24) into (17) and (18). , the mean and standard deviaDenote by tion of the GGLRT statistic as a function of the SNR, , and the scaled energy shift from the first eigenvalue to the second . Once again, the probability of detection is equal , where is defined by to (25)

2622

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002

Proposition 2: For a fixed threshold level , the probability of detection of the GGLRT decreases as the spreading increases. corresponds to Proof: First, note that the case of about a point source with SNR equal to . Expanding using a Taylor’s expansion results in

(26) The first term on the right-hand side of (26) is the expression for for the point source case when there is no modeling error. Therefore, in order to show the degradation in performance of the GGLRT, it suffices to investigate the sign of the second term. In Appendix D, it is shown that the second term in (26) deis strictly negative. Therefore, for small spreading, creases linearly with the energy shift from the first eigenvalue to the second. This decrease implies a decrease in the probability is a monotonic decreasing function. of detection since B. Simulations To validate theoretical results, simulations of a ULA with six elements at half wavelength spacing were carried out. The source in the simulations was assumed to be at an angle with a Gaussian scattering function (23). One hundred snapshots were taken with SNR per element equal to 8 dB. Fig. 2 depicts the probability of detection as a function of the source spreading for various levels of false alarm probability. The source spreading was varied between 0 (point source) and 16 . The theoretical curves were evaluated using (13), (17), and (18), whereas the empirical curves were calculated based on 1000 Monte Carlo runs. The empirical probability of detection was calculated as the number of runs in which the source was detected divided by 1000. The figure shows good fit between the empirical and theoretical curves. It is seen that the prediction of the empirical results by theoretical curves degrades as the spreading increases. This is since the theoretical analysis assumes that the energy shifts from the first eigenvalue to the second. This assumption becomes invalid as the spreading increases. As mentioned earlier, bearing estimation of scattered sources is an extensively investigated subject. It is known that as long as the source spreading is less than the array beamwidth [5], the performance of the beamformer, which is the ML estimator for the bearing of a point source, is almost invariant to the source spreading [25]. In our example, the array beamwidth is equal to 20 , and as such, the performance of the bearing estimator is hardly affected by the existence of the scattered source. On the other hand, the GGLRT suffers from a significant performance degradation, even in the presence of small spreading. IV. ANALYSIS OF THE GLRT IN STATISTICAL MISMODELING A. Analysis It is well known that in bearing estimation problems, the asymptotic performance of the stochastic Gaussian ML estimator is invariant to the source distribution [8] and to the noise

Fig. 2. Performance of the GGLRT as a function of the source spreading for various levels of probability of false alarms.

distribution [26], i.e., even when the signal (or the noise) distribution is non-Gaussian, the asymptotic covariance of the ML estimate of the bearing remains the same. This section examines the performance of the GGLRT under similar mismodeling errors. The analysis carried out in this section studies how a nonGaussian environment affects the performance of the GGLRT, which assumes that both the source and the noise are Gaussian. Note that it is assumed that while the Gaussian assumption is violated, the point source assumption is not. In the context of non-Gaussian distributions, the focus is on the mixed Gaussian distribution. The model of a finite mixed Gaussian distribution is of special interest since it provides a general framework for modeling many non-Gaussian distributions. For example, Middleton’s class A model [27] can be approximated by a finite Gaussian mixture. In addition, it is known that any monotonic spherically symmetric pdf can be approximated by a continuous Gaussian mixture distribution (see [28] and references therein). The pdf of a finite Gaussian mixture is given by (27) where complex Gaussian pdf; number of elements in the mixture; mean of the th element in the mixture; covariance matrix of the th element in the mixture. The probability of choosing the th element in the mixture is denoted by . In what follows, we examine two types of mixed Gaussian models. The first model is associated with a finite alphabet source (digital signal) in Gaussian additive noise. In this is a zero mean, discrete model, the signal at each instant complex random variable taking values from some discrete of size signal constellation denoted by . The discrete time baseband equivalent model, after down conversion, matched filtering, and sampling at the symbol rate

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST

is . The pdf of is a finite, mixed, complex Gaussian distribution. This mixture has components, each corresponding to a different value of the [29], i.e., signal. The covariance matrix is

(28) where is the probability of the source transmitting (usually ). Note that the received vector correlation matrix is equal to , where is the average equal to . signal power The second model is associated with a Gaussian point source in additive mixed Gaussian noise. In this model, the noise pdf is for every . As a result, the measuregiven by (27) with ments are distributed according to

(29) Note that the received vector correlation matrix is equal to , where is the average noise power . As proven in Theorem 1, the GGLRT statistic is asymptotically normally distributed. In Appendix B, it is shown that the mean of the GGLRT statistic is independent of the underlying and depends only on the correlation matrix of distribution . Thus, the relation between the perforthe measurements mance of the GGLRT when there are no modeling errors and the performance of the GGLRT when there exist statistical modeling errors depends only on the relation between the variance of the GGLRT statistic in the two cases. the variance of the GGLRT statistic when Denote by the measurements are distributed according to (28), i.e., digital sources. Lemma 1: For fixed SNR, the variance of the GGLRT statistic when no modeling errors exist is smaller than its variance when the measurements are distributed according to (28) . That is, if and only if (iff) iff , where denotes the variance of the GGLRT statistic when no modeling errors exist. Proof: See Appendix E. Note that Appendix E contains explicit expressions for . Lemma 1 provides a characterization of the relation between the performance of the GGLRT with and without statistical modeling errors. In order to examine the performance of the and . GGLRT, it suffices to compare , the performance Whenever of the GGLRT improves. On the other hand, whenever , the performance of the GGLRT , then the performance deteriorates. If of the GGLRT is not affected by the existence of modeling errors. Since the mean of the GGLRT statistic is independent of the exact source distribution, it suffices to examine the standard deviation of the GGLRT statistic.

2623

Conditions for an Increase in the Detection Probability due to Statistical Mismodeling: 1)

(which implies that the probability of detection is larger than 50%), and the source is radiating a digital , then signal. By using Lemma 1, if , which implies that the probability of detection of the GGLRT increases. (which implies that the probability of detection 2) is smaller than 50%), and the source is radiating a digital , then signal. By using Lemma 1, if , which implies that the probability of detection of the GGLRT increases. Conditions for a Decrease in the Detection Probability due to Statistical Mismodeling : , AND THE SOURCE IS RADIATING A DIGITAL , By using Lemma 1, if , which implies that the probability of then detection of the GGLRT decreases. , AND THE SOURCE IS RADIATING A DIGITAL 2) , SIGNAL: By using Lemma 1, if , which implies that the probability of then detection of the GGLRT decreases. 1)

SIGNAL:

Conditions for the Detection Probability not to Be Effected by Statistical Mismodeling: 1)

, I.E., THE MEAN OF THE GGLRT STATISTIC IS EQUAL TO THE THRESHOLD: In this case, the probability of detec-

tion is equal 50% and is independent of modeling errors. Due to their frequent use, special attention is given to the case of digital signals. It is easily shown that for constant modulus sources, such as binary phase shifting keys (BPSKs) or . M-phase shifting key (MPSK) signals, Thus, for constant modulus signals, whenever the probability of detection is larger than 50%, the probability of detection of the GGLRT increases. It is also easily seen that the condition is not met by sources that are impulsive in nature, e.g., the source signals emitted by an impulse radio transmissions [30]. Fig. 3 depicts a qualitive graph of the detection probability of the GGLRT as a function of the threshold for three scenarios: 1) There are no modeling errors. 2) The source radiates a digital constant modulus signal. 3) The source radiates an impulsive signal. As shown by the figure, all the curves intersect when the probability of detection is equal 50%, i.e., when the probability of detection is equal to 50%, the modeling errors have no effect on the detector’s performance. When the probability of detection is less than 50%, the case of a source emitting an impulsive signal exhibits the best performance, whereas the constant modulus digital source case exhibits the worst performance. On the other hand, when the probability of detection is larger than 50%, it is the other way around. We now turn our attention to the case of non-Gaussian noise. the variance of the GGLRT statistic when Denote by the measurements are distributed according to (29), i.e., nonGaussian noise. The following lemma proves that the variance of the GLRT statistic is increased due to the existence of the non-Gaussian noise.

2624

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002

Fig. 3. Performance of the GGLRT as a function of the threshold for various scenarios.

Fig. 4. Performance of the GGLRT as a function of the threshold for BPSK signal.

Lemma 2: For fixed SNR, is minimized when , i.e., when the measurements are complex Gaussian random vectors (no mismatch). Proof: See Appendix F. Lemma 2 characterizes the behavior of the GGLRT in the non-Gaussian noise as a function of the threshold used by the detector. Similarly to the case of digital sources, we note that for a fixed threshold , if is smaller (greater) than , the probability of detection will decrease (increase) due to the existence of the non-Gaussian noise. It should be noted that the conclusion drawn from Lemma 2 is under the assumption of a fixed threshold. However, for a fixed threshold, both the false alarm probability and the probability of detection are a function of the exact noise distribution (in contrast with the previous cases of a distributed source or a digital source). B. Simulations Once again, to validate the analytical results, simulations of a ULA with six elements at half a wavelength spacing were carried out. Two hundred snapshots of a source at bearing with SNR equal 0 dB were taken. Fig. 4 depicts the probability of detection of the GGLRT as a function of the threshold. Since both Lemma 1 and 2 analyze the performance as a function of the threshold, depicting the probability of detection as a function of the threshold is a natural choice for this figure. We tested three sources: The first was and Gaussian adtaken to be a BPSK signal, i.e., ditive noise; the second was taken to be an ”impulsive” signal, , and , ; the i.e., third was taken to be a Gaussian source. The empirical curves were calculated based on 1000 Monte Carlo runs. As expected, when the probability of detection is larger than 50%, the performance of GGLRT with BPSK source is the best, whereas when the probability of detection is smaller than 50%, the impulsive source performs the best. In addition, the accuracy of the analysis can be seen by the exact fit between the analytical and empirical curves. Note that both curves match the qualitative Fig. 3.

Fig. 5. Performance of the GGLRT as a function of the threshold for a mixed Gaussian noise.

Fig. 5 depicts the results of a similar scenario to the previous one. In this case, the source was taken to be a white Gaussian signal, whereas the additive noise was mixed-Gaussian with , , , and . Once again, the figure shows good fit between the theoretical and the empirical results. V. SUMMARY This paper provides an analysis of the performance of the GLRT in various scenarios. The analysis contains closed-form expressions for the probability of detection under various modeling errors. Focus is set on the GLRT for the problem of detecting a far-field, Gaussian point source (GGLRT) using an array of sensors. Using the general analysis, the performance of the detector is evaluated for a wide variety of scenarios. Specifically, spatial and statistical modeling errors are examined for the problem of interest, and the variation in performance is quantified. It is shown that the performance of the detector increases

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST

as the SNR increases and degrades as the source becomes more scattered. It is also shown that the performance of the GGLRT may improve in cases of statistical errors (when the source is digital or when the noise is non-Gaussian).

2625

where

, 1, and it is assumed that both and exist at least in the local vicinity of . Plugging (31) back into (30) results in the following approximation for the GLRT:

APPENDIX A PROOF OF THEOREM 1 This Appendix proves a slightly more general theorem than the one presented in Theorem 1. It is proved that under general conditions, the GLRT statistic of (3) is asymptotically normally distributed. The asymptotic distribution of the GGLRT statistic, and hence Theorem 1, can be viewed as a special case of this more general proof. The following proof is based on the derivations done in [9, proof of Th. 1], [14], and [31]. the Kullback–Leibler distance (diDenote by and , i.e., vergence) between two pdfs . Assume that and exist for every and , and are the parameter spaces of respectively, where , , respectively. In addition, assume that there exist two and (hereafter, referred to as unique, interior points and , respectively, that minimize stationary points) in and , respectively. We note that for Theorem 1, the existence of these two points is ensured by the conditions given in (8). The GLRT statistic denoted by in (3) is given by

(30)

and are consistent estimators for The ML estimators and , respectively [14]. The Taylor expansion for the funcand , around tions and , respectively, is [9]

(35) The exact asymptotic distribution of the GLRT statistic is very complicated. Both and are Gaussian quadratic forms and, as such, can be shown to be distributed as a weighted sum of chi-squared random variables. Noting that the mean and variance of these quadratic forms are bounded approaches infinity the mean and variance of and that as are unbounded, the approximation of the GLRT statistic can be further simplified by neglecting the quadratic forms. As a result, asymptotically, is given by (36) Invoking the central limit theorem, the asymptotic distribution of is given by (37)

(31)

(38) (39) (32) where

is

asymptotic distribution of is given by [14]

equal ;

We note that Theorem 1 is a special case of (37)–(39), where and are the normal distributions detailed in (4).

to , 1. The ,

(33) (34)

APPENDIX B MOMENTS OF THE GLRT STATISTIC CASE OF MISMODELING

IN THE

Denote by , 1 the correlation matrix under and by the underlying hypothesis at the stationary point correlation matrix of the data. In order to compute the mean

2626

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002

and the variance of the GLRT statistic in (11) and (12), we will make use of the following identities: Tr

is the complex Gaussian distribution, and

(40)

(41) (47)

In addition, since Tr

(42)

Straightforward calculations result in

Tr

Tr

Tr

Tr

(43)

then

(48)

Tr Tr

(44)

Similarly Tr

Therefore, since under both hypotheses vector

is a Gaussian random

(45) Note that the asymptotic mean is independent of the underlying . Similarly distribution

Tr

Tr

(46)

where the last equality holds since the underlying distribution

(49)

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST

2627

APPENDIX C PROOF OF PROPOSITION 1 Define

(50) where is the threshold of the GGLRT. The probability of detec. Since is a monotonic decreasing tion is simply function of , in order to show that the detection probability of the GGLRT increases with the SNR, it suffices to show that is a monotonic increasing function of . Differentiating with respect to and neglecting positive constants yields

(53) Next, evaluate

(54) The first inequality holds since for any . To . establish the second inequality, set , It can now be easily verified that for any (note that the threshold is always negative, and thus, is positive). Therefore, is a monotonic increasing function of , which implies that the detection probability of the GGLRT increases with . APPENDIX D SPATIAL MODELING ERROR

Therefore (55) which means that (56) In addition, we have (57), shown at the bottom of the next page, which means (58)

It is of interest to investigate the sign of the second term in (26). Note that the second term’s sign is determined completely by its nominator multiplied by

Hence

(59) (51) Clearly, the sign of is invariant to multiplication with positive constants. Hence, the focus is on

First, evaluate

(52)

(60) is negative for (recall that is negative). In since is negative, it suffices to show that order to show that

2628

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002

is a monotonic decreasing function of . Differentiating with respect to yields (61), shown at the bottom of the page, which is strictly negative, which is clearly negative, implying that concludes the proof. APPENDIX E DIGITAL SIGNAL IN GAUSSIAN NOISE In this Appendix, we evaluate the asymptotic variance of the GGLRT statistic for the case of a digital signal and Gaussian noise (28) and find the condition in which it is greater than the variance of the GGLRT statistic when there are no modeling . The distribution of the measurements errors, i.e., is given by

where is defined in (47), is the second mo, and is a complex Gaussian ment of the quadratic form and covariance matrix random vector with mean . Therefore, we have (64), shown at the bottom of the next page, where we used the fact that for a complex Gaussian random vector with mean and covariance matrix Tr

Tr

Tr Recall that

(62) . Once again, the correlation matrix of where , where the measurements is . Using Theorem 1 and following (75), is given by (63)

Thus Tr

(65) (66)

(57)

(61)

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST

Averaging (66) with respect to

2629

and thus

yields

Tr

Tr

(70)

(67)

Finally, combining (68) and (70), results in (71), shown at the from yields bottom of the page. Subtracting

Therefore Tr Tr (68) In addition, since , then

(72) iff

Thus

.

APPENDIX F GAUSSIAN SIGNAL IN MIXED-GAUSSIAN NOISE In this Appendix, we evaluate the asymptotic variance of the GGLRT statistic for the case where the signal is Gaussian and

(69)

Tr

Tr

Tr

Tr

Tr

Tr

Tr

Tr

Tr

(64)

(71)

2630

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002

the noise is mixed-Gaussian (29) and show that it is greater than the variance of the GGLRT statistic when there are no modeling . The distribution of the measurements errors, i.e., is given by (73) The correlation matrix of the measurements is given by , where . Using Theorem is given by 1, (74)

(75) is the second where is defined in (47), and , where is a complex moment of the quadratic form Gaussian random vector with zero mean and covariance matrix . Therefore Tr

Tr

Tr

(76)

It is easy to verify that Tr

Thus

(77) where the second inequality holds since for any random variable , . REFERENCES [1] E. Fishler, M. Grosmann, and H. Messer, “Detection of signals by information theoretic criteria: General asymptotic performance analysis,” IEEE Trans. Signal Processing, vol. 50, pp. 1027–1036, May 2002.

[2] M. Wax and T. Kailath, “Detection of signals by information theoretic criteria,” IEEE Trans. Acoust., Speech Signal Processing, vol. ASSP-33, pp. 387–392, Apr. 1985. [3] H. T. Wu, J. F. Yang, and F. K. Chen, “Source number estimators using transformed Gerschgorin radii,” IEEE Trans. Signal Processing, vol. 43, pp. 1325–1333, June 1995. [4] P. Stoica and A. Nehorai, “MUSIC, maximum likelihood, and Cramér-Rao bound,” IEEE Trans. Signal Processing, vol. 37, pp. 720–741, May 1989. [5] P. M. Schultheiss and H. Messer, “Optimal and suboptimal broad-band source location estimation,” IEEE Trans. Signal Processing, vol. 41, pp. 2752–2763, Sept. 1993. [6] M. Wax, “Detection and localization of multiple sources via the stochastic signals model,” IEEE Trans. Signal Processing, vol. 39, pp. 2450–2456, Nov. 1991. [7] J. B. Lewis and P. M. Schultheiss, “Optimum and conventional detection using a linear array,” J. Acoust. Soc. Amer., vol. 49, pp. 1083–1091, Apr. 1971. [8] B. Ottersten, M. Viberg, and T. Kailath, “Analysis of subspace fitting and ML techniques for parameter estimation from sensor array data,” IEEE Trans. Signal Processing, vol. 40, pp. 590–599, Mar. 1992. [9] J. T. Kent, “Robust properties of likelihood ratio test,” Biometrika, vol. 69, pp. 19–27, 82. , “The underlying structure of nonnested hypothesis tests,” [10] Biometrika, vol. 73, no. 2, pp. 333–343, August 1986. [11] A. W. van der Vaart, “Asymptotic statistics,” in Cambridge Series in Statistical and Probabilistic Mathematics, 1st ed. Cambridge, U.K.: Cambridge Univ. Press, 1998. [12] K. Viraswami and N. Reid, “Higher-order asymptotic under model misspecification,” Can. J. Statist., vol. 24, no. 2, pp. 263–278, June 1996. [13] E. Fishler and H. Messer, “On the effect of a-priori information on performance of the MDL estimator,” Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2002. [14] H. White, “Maximum likelihood estimation of misspecified models,” Econometrica, vol. 50, no. 1, pp. 1–25, Jan. 1982. [15] S. M. Kay, Fundamentals of Statistical Signal Processing – Detection Theory. Englewood Cliffs, NJ: Prentice-Hall, 1998. [16] M. A. Doron, A. J. Weiss, and H. Messer, “Maximum-likelihood direction finding of wide-band sources,” IEEE Trans. Signal Processing, vol. 41, pp. 411–414, Jan. 1993. [17] Y. Meng, P. Stoica, and K. M. Wong, “Estimation of the directions of arrival of spatially dispersed signals in array processing,” Proc. Inst. Elect. Eng., Radar, Sonar, Navigat., vol. 143, pp. 1–9, Feb. 1996. [18] Y. U. Lee, J. Choi, I. song, and S. R. Lee, “Distributed source modeling and direction-of-arrival estimation techniques,” IEEE Trans. Signal Processing, vol. 45, pp. 960–969, Apr. 1997. [19] S. Valaee, B. Champagne, and P. Kabal, “Parametric localization of distributed sources,” IEEE Trans. Signal Processing, vol. 43, pp. 2144–2153, Aug. 1995. [20] J. Friedmann, R. Raich, J. Goldberg, and H. Messer, “Bearing estimation for a distributed source of non constant modulus,” IEEE Trans. Signal Processing, Apr. 2001, submitted for publication. [21] O. Besson, F. Vincent, P. Stoica, and A. B. Gershman, “Approximate maximum likelihood estimators for array processing in multiplicative noise environments,” IEEE Trans. Signal Processing, vol. 48, pp. 2506–2518, Sept. 2000. [22] O. Besson and P. Stoica, “Decoupled estimation of DOA and angular spread for a spatially distributed source,” IEEE Trans. Signal Processing, vol. 48, pp. 1872–1882, July 2000. [23] T. Trump and B. Ottersten, “Estimation of nominal direction of arrival and angular spread using an array of sensors,” Signal Process., vol. 50, pp. 57–69, Apr. 1996. [24] H. B. Lee, “Eigenvalues and eigenvectors of covariance matrices for signals closely spaced in frequency,” IEEE Trans. Signal Processing, vol. 40, pp. 2518–2535, Oct. 1992. [25] R. Raich, J. Goldberg, and H. Messer, “Bearing estimation for a distributed source via the conventional beamformer,” in Proc. Ninth IEEE SP Workshop Statist. Array Process., Portland, OR, 1998. [26] A. Satora and P. Bentler, “Model conditions for asymptotic robustness in the analysis of linear relations,” Comput. Statist. Data Anal., vol. 10, pp. 235–249, 1990. [27] A. Spaulding and D. Middleton, “Optimum reception in an impulsive interference environment-Part I: Coherent detection,” IEEE Trans. Commun., vol. 25, pp. 910–923, Sept. 1977. [28] R. J. Kozick and B. Sadler, “Maximum-likelihood array processing in non-Gaussian noise with Gaussian mixtures,” IEEE Trans. Signal Processing, vol. 48, pp. 3520–3536, Dec. 2000.

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST

[29] E. Moulines, J.-F. Cardoso, and E. Gassiat, “Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models,” Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., vol. 5, pp. 3617–3620, 1997. [30] M. Z. Win and R. A. Scholtz, “Impulse radio: How it works,” IEEE Commun. Lett., vol. 2, no. 2, pp. 36–38, Feb. 1998. [31] H. White, “Correction to: Maximum likelihood estimation of misspecified models,” Econometrica, vol. 51, p. 513, 1983. [32] S. Bose and A. O. Steinhardt, “A maximal invariance framework for adaptive detection with structures and unstructured covariance matrices,” IEEE Trans. Signal Processing, vol. 43, pp. 2164–2175, Sept. 1995. [33] S. A. Kassam and H. V. Poor, “Robust techniques for signal processing: A survey,” Proc. IEEE, vol. 73, pp. 433–481, Mar. 1985. [34] H. Messer, “The use of spectral information in optimal detection of a source in the present of a directional interference,” IEEE J. Ocean. Eng., vol. 19, pp. 422–430, July 1994. [35] D. Parsons, The Mobile Radio Propagation Channel. New York: Pentech, 1992. [36] S. Pasupathy and P. M. Schultheiss, “Passive detection of Gaussian signals with narrow-band and broad band components,” J. Acoust. Soc. Amer., vol. 56, no. 3, pp. 917–921, Sept. 1974. [37] K. I. Pedersen, P. E. Mogensen, B. H. Fleury, F. Frederikse, K. Olesen, and S. L. Larsen, “Analysis for time, Azimuth and Doppler dispersion in outdoor radio channels,” in Proc. ACTS Summit, Aalborg, Denmark, 1997. [38] A. M. Richardson and A. H. Welsh, “Covriate screening in mixed linear models,” J. Multivariate Anal., vol. 58, no. 1, pp. 27–54, 1996. [39] P. Stoica and A. Nehorai, “MUSIC, maximum likelihood, and Cramér Rao bound – Further results and comparisons,” IEEE Trans. Signal Processing, vol. 38, pp. 2140–2150, Dec. 1990.

Jonathan Friedmann was born in Israel in 1971. He received the B.Sc. degree in industrial engineering in 1996, the M.Sc. degree in electrical engineering in 1998, and the Ph.D. degree in electrical engineering in 2002 from Tel Aviv University, Tel Aviv, Israel. His research interests include statistical signal processing and digital communications.

2631

Eran Fishler (M’01) was born in Tel Aviv, Israel, in 1972. He received the B.Sc. degree in mathematics and the M.Sc. and Ph.D. degrees in electrical engineering in 1995, 1997, and 2001, respectively, from Tel Aviv University, Tel Aviv, Israel. From 1993 to 1995, he was with the Israeli Navy as a research engineer. From 1996 to 2000, he was with Tadiran Electronic Systems, Holon, Israel. He is currently a post-doctoral fellow at Princeton University, Princeton, NJ. His research interests include statistical signal processing and communication theory.

Hagit Messer (F’01) received the B.Sc, M.Sc., and Ph.D. degrees in electrical engineering from Tel Aviv University, Tel Aviv, Israel, in 1977, 1979, and 1984, respectively. Since 1977, she has been with the Department of Electrical Engineering—Systems at the Faculty of Engineering, Tel Aviv University, first as a Research and Teaching Assistant and then, after a one-year post-doctoral appointment at Yale University (from 1985 to 1986), as a Faculty member. Her research interests are in the field of signal processing and statistical signal analysis. Her main research projects are related to tempo-spatial and non-Gaussian signal detection and parameter estimation. Dr. Messer was an associate editor of the IEEE TRANSACTIONS ON SIGNAL PROCESSING from 1994 to 1996, and she is now an associate editor for the IEEE SIGNAL PROCESSING LETTERS. She is a member of the Technical Committee of the IEEE Signal Processing Society for Signal Processing, Theory, and Methods and was a co-organizer of the IEEE Workshop on Higher Oreder Statistics in 1999. Since 2000, she has been the chief scientist of the Israeli Ministry of Science, Culture, and Sport.