separating two binary sources from a single nonlinear ... - IEEE Xplore
SEPARATING TWO BINARY SOURCES FROM A SINGLE NONLINEAR MIXTURE Konstantinos I. Diamantaras
Theophilos Papadimitriou
TEI of Thessaloniki Department of Informatics Sindos 57400, Greece [email protected]
Democritus University of Thrace Dept. of Int. Econ. Relat. & Devel. Komotini 69100, Greece [email protected]
ABSTRACT In this paper we present a novel blind method for separating two binary sources from a single, arbitrary nonlinear mixture. The method is analytical and does not involve nonlinear optimization. Our approach proceeds by linearizing the problem and extending known, clustering-based results from the linear binary BSS case to the nonlinear case. The proposed algorithm is computationally efficient. Due to the structure of the problem, the true sources are extracted together with a source product adding one more indeterminacy to the usual sign and order indeterminacy of the sources. In some applications (eg. imaging) this indeterminacy can be resolved by visual inspection. Index Terms— Signal processing, signal reconstruction 1. INTRODUCTION Blind Source Separation (BSS) methods attempt to the recovery of n unknown signals, using m recorded mixtures. The term “blind” is justified by the fact that both the sources and the underlying mixing operator are unknown. BSS problems have attracted a lot of attention from the scientific community in the past two decades. In general, BSS methods can be divided according to the mixing process into linear and nonlinear ones. The former case can be subdivided into memoryless linear mixture BSS (also known as instantaneous BSS) and convolutive mixture BSS (the problem is also referred to as multichannel blind deconvolution/equalization). These problems have been extensively studied in the past and they resulted some well known and established methods such as AMUSE [1], SOBI [2], JADE [3], ICA [10]. In the nonlinear case the research is still under development although quite a few results have emerged under certain assumptions. The most important difference between the two classes of problems comes from their indeterminacies. The linear BSS suffers from its inability to recover the sources scale and order. The sources order is often not very important; in real world applications the sources scale can be re-introduced in the estimated sources. In the nonlinear BSS problem the methods are unable to find the sources order, but the bigger problem arises from the lack of solution uniqueness. In fact, any nonlinear transformation of the true sources forms a potential solution to the nonlinear BSS problem [11]. The recovery inconsistency has been fronted by adding further a priori information directly in the model or as a regularization term in the optimization processing. Since the first method addressing the nonlinear BSS [12] in 1992 various approaches have been investigated. Almeida in [13] proposes a generalization of the INFOMAX method, which is able to
978-1-4244-4296-6/10/$25.00 ©2010 IEEE
1946
deal with nonlinear mixtures. The method yields a single multilayer network performing ICA and recovering the sources simultaneously from a single maximization procedure. Jutten and colleagues investigated extensively a particular class of nonlinear systems, called postnonlinear systems, composed by a linear subsystem followed by a memoryless nonlinear distortion [6, 7]. Lappalainen and Honkela in [14] perform nonlinear ICA using a multi-layer perceptron network for the separation, while the sources are modelled using the mixtureof-Gaussians density. Valpola e.a. in [15] attack the BSS problem of a nonlinear static or dynamic mixing model using unsupervised Bayesian modeling, where the necessary posterior pdf’s of the unknown variables for the bayesian estimation is approximated using the Variational Bayesian Learning. A kernel-based approach of the nonlinear BSS problem can be found in [16]. The authors transform the nonlinear problem in input space into a linear one in the the parameter space, where there are well known methods for the sources recovery. In this paper we will address the BSS problem of nonlinear systems driven by two binary sources. To our knowledge none of the existing methods can treat this problem. We will show that the Taylor expansion of the nonlinear model can be seen as an instantaneous linear system and we can extend known results for the linear case to attack the problem. The paper is structured as follows: Section 2 describes the problem formulation and basic assumptions; Section 3 extends previous results on linear binary BSS to solve the problem; Section 4 proposes an extension to treat the noisy case; Section 5 presents simulations and finally we conclude in Section 6. 2. SYSTEM MODEL AND PROBLEM DESCRIPTION Consider two binary signals s1 , s2 ∈ {−1, +1}, combined into a single mixture, x(k) = f (s1 (k), s2 (k)) (1) via some unknown analytical nonlinear function f . Our goal is to extract the sources given only the output x and tolerating ambiguities such as order or sign of the sources. To that end we shall try to linearize the problem and use previous results on linear MISO (Multi-Input Single-Output) BSS problems with binary inputs. Let us take the Taylor expansion of f around the point [0, 0] to obtain f (s1 , s2 )
=
f (0, 0) + +
∞
∂f ∂f s1 + s2 ∂s1 ∂s2 sp1 sp2 ∂r f
r=2 p1 +p2 =r
1
2
∂sp11 ∂sp22 p1 !p2 !
(2)
ICASSP 2010
where all derivatives are evaluated at the point [0, 0]. The Taylor expansion involves all powers of the sources si , i = 1, 2. However, 2p+1 = si , for any positive since si = ±1 we have s2p i = 1 and si integer p. Therefore, we can rewrite (2) as f (s1 , s2 ) = a0 + a1 s1 + a2 s2 + a3 s1 s2
(3)
for some unknown coefficients aj , j = 0, · · · , 3. The new expression (3) is quite simplified, being a linear combination of only the terms s1 , s2 , and s1 s2 , plus an additive constant, a0 . Defining the augmented source vector q = [s1 , s2 , s1 s2 ]T and the mixing vector a = [a1 , a2 , a3 ]T we can write x(k) = aT q(k) + a0
(4)
We make the following assumptions about the sources:
sources is unobservable. By clustering the output values we obtain 2n cluster centers ci , i = 1, · · · 2n (in the noise-free case, cluster i is just a single point ci ). The mixing coefficients and the sources can be found from the values ci , in general, using a recursive algorithm. However, especially for n = 3, the mixing coefficients can be found by an analytical expression. Arranging the centers in ascending order, c1 < c2 < · · · < c8 , we have a ˆ3 a ˆ2 a ˆ1
aT t} q ˆ(k) = arg min{y(k) − ˆ t∈B
A.2 The samples s1 (k), s2 (k), are i.i.d. and zero mean: E{s1 (k)} = E{s2 (k)} = 0, ∀k Under assumption A.1, and since f (s1 , s2 ) can only take 4 different values, it is straightforward to see that it admits a linear expansion as in (3), even if the function is not infinitely derivable. Also, under assumption A.2 it is straightforward to get rid of the additive constant a0 by subtracting the average signal x ¯ = E{x} from x (5)
The problem has been now transformed into a linear blind source separation problem with three sources, q1 , q2 , q3 , and one output signal, y, under the constraint that q3 = q1 q2 . A similar problem has been treated in [4] except that, in this previous work, all sources were assumed independent. Next we describe this method and show how to extend these results in order to solve the problem at hand.
(7) (8) (9)
(10)
where B is the set of the 8 binary triplets [±1, ±1, ±1]T . 3.1. Extension to the nonlinear case The problem with the nonlinear case, described in section 2, is that not all 8 binary triplets are readily available in the input. This is because the 3rd source is dependent on the other two. In particular, only 4 triplets will appear in the input, i.e. q ∈ BN L , where BN L
⎧⎡ ⎤⎫ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ 1 ⎬ −1 1 ⎨ −1 = ⎣ −1 ⎦ , ⎣ −1 ⎦ , ⎣ 1 ⎦ , ⎣ 1 ⎦ . ⎩ 1 ⎭ −1 −1 1
Luckily, the remaining four triplets are simply the negatives of the above four, ie. ¯N L B = BN L ∪ B
3. SEPARATION OF N BINARY SOURCES FROM A SINGLE OBSERVED SIGNAL The method introduced in [4] treats the generic problem of blindly separating n ≥ 2 binary sources qi , i = 1, · · · , n, from a single linear mixture n y(k) = ai qi (k). (6) i=1
The method is based on the assumption that all possible 2n input vectors q(k) appear at least once in the data sequence. We also assume that the linear mixing coefficients are binary independent (or bi-independent [5]): A.3 Binary independence: The coefficients ai are such that n
(c2 − c1 )/2, (c3 − c1 )/2, −(c2 + c3 )/2.
Once we have obtained the mixing coefficients, the source values can be found by exhaustive search:
A.1 The signals s1 , s2 , are statistically independent
y(k) = x(k) − x ¯ = aT q(k).
= = =
(λi − λi )ai = 0, λi , λi ∈ {−1, +1} → λi = λi
i=1
Therefore each unique input vector corresponds to a unique output value or, equivalently, the function is invertible in the binary domain. Assumption A.3 excludes the class of functions for which ai are bi-dependent. The set of even functions is an important example of such a class. If the Taylor expansion of f involves only even powers of si it is straightforward to see that a1 = a2 = a3 = 0 and assumption A.3 fails. Moreover, we assume that the mixing coefficients are positive and arranged in decreasing order a1 > a2 > a3 > 0. This assumption does not hurt generality because the sign and order of the
1947
where
¯N L = {−bi | all bi ∈ B} B Moreover, the cluster centers d¯i corresponding to inputs from the set ¯N L are the negatives of the centers di corresponding to inputs from B the set BN L . Namely, ¯N L ¯i , b ¯i ∈ B di = aT bi , bi ∈ BN L → − di = aT b Therefore, the complete set of centers, C, corresponding to inputs from the set B are simply the union of the sets {di , i = 1, · · · , 4} and {−di , i = 1, · · · , 4}. With the complete set C, we can proceed to the estimation of the mixing parameters and the inputs using formulas (7) - (10). Note that due to the well known sign and order ambiguity we have qˆi = ±qπ(i) , i = 1, 2, 3, for some unknown permutation π(·). Since q1 = q2 q3 and q2 = q1 q3 and q3 = q1 q2 , it is impossible to separate the true sources s1 , s2 , from their product s1 s2 . This is an additional ambiguity appearing in the nonlinear case. Without additional information about the sources, it is not possible to resolve this ambiguity. Summarizing the above discussion, we obtain the following algorithm for the nonlinear case:
Algorithm 1 Algorithm: Nonlinear binary source separation • Collect the output data x(k) and compute the 4 clusters with centers di , i = 1, · · · , 4 • Form the set C = {di , i = 1, · · · , 4}∪{−di , i = 1, · · · , 4} • Sort the members of C in ascending order, c1 < c2 < · · · < c8 • Estimate the mixing coefficients using (7) - (9) • Estimate the sources using (10)
(a) Source 1
(b) Source 2
4. NOISY CASE With the addition of noise e(k), the model becomes y(k) = aT q(k) + e(k)
(11)
A straightforward application of the Bayes rule yields p(y)
=
p(y | q)p(q) (c) Nonlinear mixture
q
=
pe (y − aT q)p(q)
(12)
q
Therefore, the distribution of y is a mixture of the distributions pe centered at the points aT q and scaled by the factors p(q). For example, if the noise is zero-mean and Gaussian, then p(y) is a Gaussian mixture. Our algorithm applies directly using the centers of the clusters in the mixture. The centers can be estimated using either a parametric method, such as the EM algorithm [8], or a non-parametric method such as the K-means algorithm [9]. Our method of choice is the EM algorithm since it typically yields better results. 5. SIMULATIONS One possible application of our method is thin paper scanning on top of marked surface (could be another paper), where both the thin paper and the marked surface contains black and white markings (e.g. text). We ran experiments using images produced by scanning text and binarizing the images by thresholding. The two source images are shown in Figures 1a, b. Black pixels correspond to the value −1 while white pixels correspond to the value +1. Then a mixture was produced using the following nonlinear function: x(k)
=
exp{α1 s1 (k) + α2 s2 (k)} +β1 s1 (k) + β2 s2 (k) +γ11 s1 (k)2 + γ12 s1 (k)s2 (k) + γ22 s2 (k)2
with parameters: α1 = 0.3273, α1 = 1.1746, β1 = 1.1867, β2 = 0.7258, γ11 = −0.5883, γ12 = 2.0468, γ22 = 0.1139. The values of x were shifted and scaled to fit into the range 0255 and small quantization noise was added by rounding up to the closest integer. The resulting image mixture is shown in Fig. 1c. The results of the source separation are shown in Fig. 2a-c. Although the source images are not zero-mean the algorithm is quite robust resulting in perfect reconstruction of the sources (Fig. 2b-c). Additionally we obtain a third image which is the product of the other two. In this case the third image can be separated by plain visual inspection. Mathematically however, it is not possible to separate it unless more information about the sources is given.
1948
Fig. 1. (a),(b) Two binary source images and (c) their nonlinear mixture. Next we tried the performance of the method with the addition of noise. We generated random binary sources of length N = 5000. The nonlinear mixture was generated by the function x(k) = log{|α1 s1 (k) + α2 s2 (k)|} + β1 s1 (k) + β2 s2 (k) with parameters α1 = −0.6856, α2 = −0.4094, β1 = 1.1690, β2 = 1.5769. Then additive white gaussian noise was injected with SNR levels varying from 15 to 40 dB. Our performance index is the bit-error-rate (BER) of the resulting reconstruction taking into account the sign and order indeterminacy: we consider qˆi = −qj as perfect recontruction of qj (BER=0). Fig. 3 shows how the performance increases with SNR and BER goes down to zero for SNR> 25. 6. CONCLUSIONS We presented a new blind method for the separation of two binary sources from a single nonlinear mixture. The method takes advantage of the fact that only a small number of powers of the binary sources appear in the Taylor expansion and so the nonlinear model can be successfully transformed into a finite linear model. Then we can extend previous results on linear binary BSS to solve the problem. One side effect of the problem formulation is the fact that, in addition to the true sources, we extract an additional pseudo-source signal which is the product of the true sources. This is an extra indeterminacy added to the usual sign and order indeterminacies of BSS. In some applications, however, (eg. binary image separation) this extra signal can be simply separated by visual inspection. The proposed method is computationally efficient as it is based on 1-d clustering and does not involve nonlinear optimization. However, it does not easily extend to multilevel sources or more than two sources. This extension will be the focus of future work.
0.14
0.12
0.1
BER
0.08
0.06
0.04
(a) Result 1
0.02
0 15
20
25
SNR(dB)
30
35
40
Fig. 3. The average Bit Error Rate (BER) for the three estimated sources qˆi after 100 Monte Carlo simulations.
(b) Result 2
(c) Result 3 Fig. 2. The three signals extracted by the algorithm. Two of them are the actual sources and the third is the product of the other two (-1:black pixel, +1=white pixel). 7. REFERENCES [1] L. Tong, R. Liu, V.C. Soon, and Y.F. Huang, “Indeterminacy and identifiability of blind identification,” IEEE Trans. Circuits Syst. I, vol. 38, no. 5, pp. 499–509, 1991. [2] A. Belouchrani, K. Abed-Meraim, J.-F. Cardoso, and E. Moulines, “A Blind Source Separation Technique Using Second-Order Statistics,” IEEE Trans. Signal Processing, vol. 45, no. 2, pp. 434–444, Feb. 1997. [3] J.-F. Cardoso and A. Souloumiac, “Blind Beamforming for non Gaussian Signals,” IEE Proceesings-F, vol. 140, no. 6, pp. 362–370, Dec 1993. [4] K. Diamantaras, “Blind Separation of Multiple Binary Sources using a Single Linear Mixture,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP-2000), vol. V, pp. 2889–2892, Istanbul, Turkey, June 2000.
1949
[5] Y. Li, A. Cichocki and L. Zhang, “Blind Separation and Extraction of Binary Sources,” IEICE Trans. Fundamentals, vol. E86-A, no. 3, pp. 580–589, March 2003. [6] J. Sole i Casals, A. Taleb, and C. Jutten, “Parametric Apporach to Blind Deconvolution of Nonlinear Channels,” in Proc. European Symposium on Artificial Neural Networks (ESANN2000), pp. 21–26, Bruges, Belgium, April 2000. [7] S. Achard and C. Jutten, “Identifiability of Post-Nonlinear Mixtures,” IEEE Signal Processing Letters, vol. 12, no. 5, pp. 423–426, May 2005. [8] C. M. Bishop, Neural Netowrks for Pattern Recognition, Oxford University Press, 1995. [9] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, John Wiley, NY, 2001. [10] A. Hyv¨arinen, “Fast and Robust Fixed-Point Algorithms for Independent Component Analysis,” IEEE Trans. on Neural Networks, vol. 10, no. 3, pp. 626–634, 1999. [11] A. Hyv¨arinen and P. Pajunen, “Nonlinear independent component analysis: Existence and uniqueness results,” Neural Networks, vol. 12, no. 3, pp. 429–439, 1999. [12] G. Burel, “Blind Separation of Sources: A nonlinear neural algorithm,” Neural Networks, vol. 5, no. 6, pp. 937–947, 1992. [13] L.B. Almeida, “MISEP - Linear and Nonlinear ICA Based on Mutual Information,” The journal of Machine Learning Research, vol. 4, pp. 1297–1318, 2003. [14] H. Lappalainen and A. Honkela, “Bayesian Nonlinear Independent Component Analysis by Multi-Layer Perceptrons,” in Adv. In Ind. Comp. Anal., ed. By Mark Girolami, pp. 93– 121,Springer, 2000. [15] H. Valpola, E. Oja, A. Ilin, A. Honkela, and J. Karhunen, “Nonlinear blind source separation by variational Bayesian learning,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Vol. 86, pp. 532–541, 2003. [16] S. Harmeling, A. Ziehe, and M. Kawanabe, “Kernel-Based Nonlinear Blind Source Separation,” Neurocomputing, Vol. 15, pp. 1089–1124, 2003.
Theophilos Papadimitriou
TEI of Thessaloniki Department of Informatics Sindos 57400, Greece [email protected]
Democritus University of Thrace Dept. of Int. Econ. Relat. & Devel. Komotini 69100, Greece [email protected]
ABSTRACT In this paper we present a novel blind method for separating two binary sources from a single, arbitrary nonlinear mixture. The method is analytical and does not involve nonlinear optimization. Our approach proceeds by linearizing the problem and extending known, clustering-based results from the linear binary BSS case to the nonlinear case. The proposed algorithm is computationally efficient. Due to the structure of the problem, the true sources are extracted together with a source product adding one more indeterminacy to the usual sign and order indeterminacy of the sources. In some applications (eg. imaging) this indeterminacy can be resolved by visual inspection. Index Terms— Signal processing, signal reconstruction 1. INTRODUCTION Blind Source Separation (BSS) methods attempt to the recovery of n unknown signals, using m recorded mixtures. The term “blind” is justified by the fact that both the sources and the underlying mixing operator are unknown. BSS problems have attracted a lot of attention from the scientific community in the past two decades. In general, BSS methods can be divided according to the mixing process into linear and nonlinear ones. The former case can be subdivided into memoryless linear mixture BSS (also known as instantaneous BSS) and convolutive mixture BSS (the problem is also referred to as multichannel blind deconvolution/equalization). These problems have been extensively studied in the past and they resulted some well known and established methods such as AMUSE [1], SOBI [2], JADE [3], ICA [10]. In the nonlinear case the research is still under development although quite a few results have emerged under certain assumptions. The most important difference between the two classes of problems comes from their indeterminacies. The linear BSS suffers from its inability to recover the sources scale and order. The sources order is often not very important; in real world applications the sources scale can be re-introduced in the estimated sources. In the nonlinear BSS problem the methods are unable to find the sources order, but the bigger problem arises from the lack of solution uniqueness. In fact, any nonlinear transformation of the true sources forms a potential solution to the nonlinear BSS problem [11]. The recovery inconsistency has been fronted by adding further a priori information directly in the model or as a regularization term in the optimization processing. Since the first method addressing the nonlinear BSS [12] in 1992 various approaches have been investigated. Almeida in [13] proposes a generalization of the INFOMAX method, which is able to
978-1-4244-4296-6/10/$25.00 ©2010 IEEE
1946
deal with nonlinear mixtures. The method yields a single multilayer network performing ICA and recovering the sources simultaneously from a single maximization procedure. Jutten and colleagues investigated extensively a particular class of nonlinear systems, called postnonlinear systems, composed by a linear subsystem followed by a memoryless nonlinear distortion [6, 7]. Lappalainen and Honkela in [14] perform nonlinear ICA using a multi-layer perceptron network for the separation, while the sources are modelled using the mixtureof-Gaussians density. Valpola e.a. in [15] attack the BSS problem of a nonlinear static or dynamic mixing model using unsupervised Bayesian modeling, where the necessary posterior pdf’s of the unknown variables for the bayesian estimation is approximated using the Variational Bayesian Learning. A kernel-based approach of the nonlinear BSS problem can be found in [16]. The authors transform the nonlinear problem in input space into a linear one in the the parameter space, where there are well known methods for the sources recovery. In this paper we will address the BSS problem of nonlinear systems driven by two binary sources. To our knowledge none of the existing methods can treat this problem. We will show that the Taylor expansion of the nonlinear model can be seen as an instantaneous linear system and we can extend known results for the linear case to attack the problem. The paper is structured as follows: Section 2 describes the problem formulation and basic assumptions; Section 3 extends previous results on linear binary BSS to solve the problem; Section 4 proposes an extension to treat the noisy case; Section 5 presents simulations and finally we conclude in Section 6. 2. SYSTEM MODEL AND PROBLEM DESCRIPTION Consider two binary signals s1 , s2 ∈ {−1, +1}, combined into a single mixture, x(k) = f (s1 (k), s2 (k)) (1) via some unknown analytical nonlinear function f . Our goal is to extract the sources given only the output x and tolerating ambiguities such as order or sign of the sources. To that end we shall try to linearize the problem and use previous results on linear MISO (Multi-Input Single-Output) BSS problems with binary inputs. Let us take the Taylor expansion of f around the point [0, 0] to obtain f (s1 , s2 )
=
f (0, 0) + +
∞
∂f ∂f s1 + s2 ∂s1 ∂s2 sp1 sp2 ∂r f
r=2 p1 +p2 =r
1
2
∂sp11 ∂sp22 p1 !p2 !
(2)
ICASSP 2010
where all derivatives are evaluated at the point [0, 0]. The Taylor expansion involves all powers of the sources si , i = 1, 2. However, 2p+1 = si , for any positive since si = ±1 we have s2p i = 1 and si integer p. Therefore, we can rewrite (2) as f (s1 , s2 ) = a0 + a1 s1 + a2 s2 + a3 s1 s2
(3)
for some unknown coefficients aj , j = 0, · · · , 3. The new expression (3) is quite simplified, being a linear combination of only the terms s1 , s2 , and s1 s2 , plus an additive constant, a0 . Defining the augmented source vector q = [s1 , s2 , s1 s2 ]T and the mixing vector a = [a1 , a2 , a3 ]T we can write x(k) = aT q(k) + a0
(4)
We make the following assumptions about the sources:
sources is unobservable. By clustering the output values we obtain 2n cluster centers ci , i = 1, · · · 2n (in the noise-free case, cluster i is just a single point ci ). The mixing coefficients and the sources can be found from the values ci , in general, using a recursive algorithm. However, especially for n = 3, the mixing coefficients can be found by an analytical expression. Arranging the centers in ascending order, c1 < c2 < · · · < c8 , we have a ˆ3 a ˆ2 a ˆ1
aT t} q ˆ(k) = arg min{y(k) − ˆ t∈B
A.2 The samples s1 (k), s2 (k), are i.i.d. and zero mean: E{s1 (k)} = E{s2 (k)} = 0, ∀k Under assumption A.1, and since f (s1 , s2 ) can only take 4 different values, it is straightforward to see that it admits a linear expansion as in (3), even if the function is not infinitely derivable. Also, under assumption A.2 it is straightforward to get rid of the additive constant a0 by subtracting the average signal x ¯ = E{x} from x (5)
The problem has been now transformed into a linear blind source separation problem with three sources, q1 , q2 , q3 , and one output signal, y, under the constraint that q3 = q1 q2 . A similar problem has been treated in [4] except that, in this previous work, all sources were assumed independent. Next we describe this method and show how to extend these results in order to solve the problem at hand.
(7) (8) (9)
(10)
where B is the set of the 8 binary triplets [±1, ±1, ±1]T . 3.1. Extension to the nonlinear case The problem with the nonlinear case, described in section 2, is that not all 8 binary triplets are readily available in the input. This is because the 3rd source is dependent on the other two. In particular, only 4 triplets will appear in the input, i.e. q ∈ BN L , where BN L
⎧⎡ ⎤⎫ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ 1 ⎬ −1 1 ⎨ −1 = ⎣ −1 ⎦ , ⎣ −1 ⎦ , ⎣ 1 ⎦ , ⎣ 1 ⎦ . ⎩ 1 ⎭ −1 −1 1
Luckily, the remaining four triplets are simply the negatives of the above four, ie. ¯N L B = BN L ∪ B
3. SEPARATION OF N BINARY SOURCES FROM A SINGLE OBSERVED SIGNAL The method introduced in [4] treats the generic problem of blindly separating n ≥ 2 binary sources qi , i = 1, · · · , n, from a single linear mixture n y(k) = ai qi (k). (6) i=1
The method is based on the assumption that all possible 2n input vectors q(k) appear at least once in the data sequence. We also assume that the linear mixing coefficients are binary independent (or bi-independent [5]): A.3 Binary independence: The coefficients ai are such that n
(c2 − c1 )/2, (c3 − c1 )/2, −(c2 + c3 )/2.
Once we have obtained the mixing coefficients, the source values can be found by exhaustive search:
A.1 The signals s1 , s2 , are statistically independent
y(k) = x(k) − x ¯ = aT q(k).
= = =
(λi − λi )ai = 0, λi , λi ∈ {−1, +1} → λi = λi
i=1
Therefore each unique input vector corresponds to a unique output value or, equivalently, the function is invertible in the binary domain. Assumption A.3 excludes the class of functions for which ai are bi-dependent. The set of even functions is an important example of such a class. If the Taylor expansion of f involves only even powers of si it is straightforward to see that a1 = a2 = a3 = 0 and assumption A.3 fails. Moreover, we assume that the mixing coefficients are positive and arranged in decreasing order a1 > a2 > a3 > 0. This assumption does not hurt generality because the sign and order of the
1947
where
¯N L = {−bi | all bi ∈ B} B Moreover, the cluster centers d¯i corresponding to inputs from the set ¯N L are the negatives of the centers di corresponding to inputs from B the set BN L . Namely, ¯N L ¯i , b ¯i ∈ B di = aT bi , bi ∈ BN L → − di = aT b Therefore, the complete set of centers, C, corresponding to inputs from the set B are simply the union of the sets {di , i = 1, · · · , 4} and {−di , i = 1, · · · , 4}. With the complete set C, we can proceed to the estimation of the mixing parameters and the inputs using formulas (7) - (10). Note that due to the well known sign and order ambiguity we have qˆi = ±qπ(i) , i = 1, 2, 3, for some unknown permutation π(·). Since q1 = q2 q3 and q2 = q1 q3 and q3 = q1 q2 , it is impossible to separate the true sources s1 , s2 , from their product s1 s2 . This is an additional ambiguity appearing in the nonlinear case. Without additional information about the sources, it is not possible to resolve this ambiguity. Summarizing the above discussion, we obtain the following algorithm for the nonlinear case:
Algorithm 1 Algorithm: Nonlinear binary source separation • Collect the output data x(k) and compute the 4 clusters with centers di , i = 1, · · · , 4 • Form the set C = {di , i = 1, · · · , 4}∪{−di , i = 1, · · · , 4} • Sort the members of C in ascending order, c1 < c2 < · · · < c8 • Estimate the mixing coefficients using (7) - (9) • Estimate the sources using (10)
(a) Source 1
(b) Source 2
4. NOISY CASE With the addition of noise e(k), the model becomes y(k) = aT q(k) + e(k)
(11)
A straightforward application of the Bayes rule yields p(y)
=
p(y | q)p(q) (c) Nonlinear mixture
q
=
pe (y − aT q)p(q)
(12)
q
Therefore, the distribution of y is a mixture of the distributions pe centered at the points aT q and scaled by the factors p(q). For example, if the noise is zero-mean and Gaussian, then p(y) is a Gaussian mixture. Our algorithm applies directly using the centers of the clusters in the mixture. The centers can be estimated using either a parametric method, such as the EM algorithm [8], or a non-parametric method such as the K-means algorithm [9]. Our method of choice is the EM algorithm since it typically yields better results. 5. SIMULATIONS One possible application of our method is thin paper scanning on top of marked surface (could be another paper), where both the thin paper and the marked surface contains black and white markings (e.g. text). We ran experiments using images produced by scanning text and binarizing the images by thresholding. The two source images are shown in Figures 1a, b. Black pixels correspond to the value −1 while white pixels correspond to the value +1. Then a mixture was produced using the following nonlinear function: x(k)
=
exp{α1 s1 (k) + α2 s2 (k)} +β1 s1 (k) + β2 s2 (k) +γ11 s1 (k)2 + γ12 s1 (k)s2 (k) + γ22 s2 (k)2
with parameters: α1 = 0.3273, α1 = 1.1746, β1 = 1.1867, β2 = 0.7258, γ11 = −0.5883, γ12 = 2.0468, γ22 = 0.1139. The values of x were shifted and scaled to fit into the range 0255 and small quantization noise was added by rounding up to the closest integer. The resulting image mixture is shown in Fig. 1c. The results of the source separation are shown in Fig. 2a-c. Although the source images are not zero-mean the algorithm is quite robust resulting in perfect reconstruction of the sources (Fig. 2b-c). Additionally we obtain a third image which is the product of the other two. In this case the third image can be separated by plain visual inspection. Mathematically however, it is not possible to separate it unless more information about the sources is given.
1948
Fig. 1. (a),(b) Two binary source images and (c) their nonlinear mixture. Next we tried the performance of the method with the addition of noise. We generated random binary sources of length N = 5000. The nonlinear mixture was generated by the function x(k) = log{|α1 s1 (k) + α2 s2 (k)|} + β1 s1 (k) + β2 s2 (k) with parameters α1 = −0.6856, α2 = −0.4094, β1 = 1.1690, β2 = 1.5769. Then additive white gaussian noise was injected with SNR levels varying from 15 to 40 dB. Our performance index is the bit-error-rate (BER) of the resulting reconstruction taking into account the sign and order indeterminacy: we consider qˆi = −qj as perfect recontruction of qj (BER=0). Fig. 3 shows how the performance increases with SNR and BER goes down to zero for SNR> 25. 6. CONCLUSIONS We presented a new blind method for the separation of two binary sources from a single nonlinear mixture. The method takes advantage of the fact that only a small number of powers of the binary sources appear in the Taylor expansion and so the nonlinear model can be successfully transformed into a finite linear model. Then we can extend previous results on linear binary BSS to solve the problem. One side effect of the problem formulation is the fact that, in addition to the true sources, we extract an additional pseudo-source signal which is the product of the true sources. This is an extra indeterminacy added to the usual sign and order indeterminacies of BSS. In some applications, however, (eg. binary image separation) this extra signal can be simply separated by visual inspection. The proposed method is computationally efficient as it is based on 1-d clustering and does not involve nonlinear optimization. However, it does not easily extend to multilevel sources or more than two sources. This extension will be the focus of future work.
0.14
0.12
0.1
BER
0.08
0.06
0.04
(a) Result 1
0.02
0 15
20
25
SNR(dB)
30
35
40
Fig. 3. The average Bit Error Rate (BER) for the three estimated sources qˆi after 100 Monte Carlo simulations.
(b) Result 2
(c) Result 3 Fig. 2. The three signals extracted by the algorithm. Two of them are the actual sources and the third is the product of the other two (-1:black pixel, +1=white pixel). 7. REFERENCES [1] L. Tong, R. Liu, V.C. Soon, and Y.F. Huang, “Indeterminacy and identifiability of blind identification,” IEEE Trans. Circuits Syst. I, vol. 38, no. 5, pp. 499–509, 1991. [2] A. Belouchrani, K. Abed-Meraim, J.-F. Cardoso, and E. Moulines, “A Blind Source Separation Technique Using Second-Order Statistics,” IEEE Trans. Signal Processing, vol. 45, no. 2, pp. 434–444, Feb. 1997. [3] J.-F. Cardoso and A. Souloumiac, “Blind Beamforming for non Gaussian Signals,” IEE Proceesings-F, vol. 140, no. 6, pp. 362–370, Dec 1993. [4] K. Diamantaras, “Blind Separation of Multiple Binary Sources using a Single Linear Mixture,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP-2000), vol. V, pp. 2889–2892, Istanbul, Turkey, June 2000.
1949
[5] Y. Li, A. Cichocki and L. Zhang, “Blind Separation and Extraction of Binary Sources,” IEICE Trans. Fundamentals, vol. E86-A, no. 3, pp. 580–589, March 2003. [6] J. Sole i Casals, A. Taleb, and C. Jutten, “Parametric Apporach to Blind Deconvolution of Nonlinear Channels,” in Proc. European Symposium on Artificial Neural Networks (ESANN2000), pp. 21–26, Bruges, Belgium, April 2000. [7] S. Achard and C. Jutten, “Identifiability of Post-Nonlinear Mixtures,” IEEE Signal Processing Letters, vol. 12, no. 5, pp. 423–426, May 2005. [8] C. M. Bishop, Neural Netowrks for Pattern Recognition, Oxford University Press, 1995. [9] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, John Wiley, NY, 2001. [10] A. Hyv¨arinen, “Fast and Robust Fixed-Point Algorithms for Independent Component Analysis,” IEEE Trans. on Neural Networks, vol. 10, no. 3, pp. 626–634, 1999. [11] A. Hyv¨arinen and P. Pajunen, “Nonlinear independent component analysis: Existence and uniqueness results,” Neural Networks, vol. 12, no. 3, pp. 429–439, 1999. [12] G. Burel, “Blind Separation of Sources: A nonlinear neural algorithm,” Neural Networks, vol. 5, no. 6, pp. 937–947, 1992. [13] L.B. Almeida, “MISEP - Linear and Nonlinear ICA Based on Mutual Information,” The journal of Machine Learning Research, vol. 4, pp. 1297–1318, 2003. [14] H. Lappalainen and A. Honkela, “Bayesian Nonlinear Independent Component Analysis by Multi-Layer Perceptrons,” in Adv. In Ind. Comp. Anal., ed. By Mark Girolami, pp. 93– 121,Springer, 2000. [15] H. Valpola, E. Oja, A. Ilin, A. Honkela, and J. Karhunen, “Nonlinear blind source separation by variational Bayesian learning,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Vol. 86, pp. 532–541, 2003. [16] S. Harmeling, A. Ziehe, and M. Kawanabe, “Kernel-Based Nonlinear Blind Source Separation,” Neurocomputing, Vol. 15, pp. 1089–1124, 2003.
