Separating a Real-Life Nonlinear Mixture of Images
Separating a Real-Life Nonlinear Mixture of Images Lu´ıs B. Almeida and Miguel Faria INESC ID and IST, R. Alves Redol, 9, 1000-029 Lisboa, Portugal [email protected] http://neural.inesc-id.pt/~lba
Abstract. This manuscript presents results obtained using an ICA technique in a real-life nonlinear image separation problem: the separation of the images of the two pages of a paper document when the image from the back page shows through, superimposed on the image of the front page. For this manuscript, two images were printed on opposite sides of a sheet of onion skin paper, and then both sides of the sheet were scanned. The scanned images contained a markedly nonlinear mixture of the original images. Nonlinear ICA, using the MISEP technique, was used to recover the original images. It showed to be able to achieve a reasonable, but not perfect separation. The best results were obtained with a separating system which was somewhat customized, based on prior knowledge about the mixture process, and which used explicit regularization.
1
Introduction
When scanning or photographing a paper document, the image of the back page sometimes shows through. This is normally due to partial transparency of the paper or to bleeding of the ink through the paper. In either case, the image that is acquired consists of a mixture of the original images contained in each of the pages, and it would be of interest to be able to eliminate the superposition and recover the original images. Since it is possible to acquire both sides of the document, two different mixtures of the original images can be obtained, and therefore ICA is a natural candidate for source separation. Often, however, the mixture is substantially nonlinear, and linear ICA techniques are not adequate. This constitutes, therefore, an interesting test case for nonlinear ICA methods. We decided to implement a difficult version of this problem, using a relatively transparent paper (“onion skin”), resulting in a mixture that is both strong and significantly nonlinear. We show separation results obtained with (1) linear ICA, (2) nonlinear ICA (MISEP method) with the basic separating structure, and (3) nonlinear ICA (MISEP) using a customized separating structure that incorporates some knowledge about the mixing process. The latter results correspond to the best separation, which is still not perfect, leaving room for improvement. There are still very few published results of source separation on nonlinear mixtures of real-life data. An example is [1]. Other applications to real-life data, e.g. [2, 3], do not provide any means to assess whether the extracted components correspond to actual sources.
2
Lu´ıs B. Almeida et al.
Fig. 1. Photographs used for the second test problem. The right hand photograph has been horizontally flipped, to correspond with the position in which it appears in the mixture and separation images.
2
Source Images, Printing, Acquisition and Preprocessing
We present separation results on two test problems. For each problem we printed two gray scale images on opposite pages of onion skin paper. The images of the first problem were artificially generated, each of them consisting of parallel bars with randomly chosen gray levels. In one of the images the bars were oriented horizontally and in the other they were oriented vertically. We don’t present those images here to save space, but the results shown ahead clarify the images’ contents. The second pair of images consisted of the photos shown in Fig. 1. We used a monochrome laser printer at a resolution of 1200 dpi, with the printer’s default halftoning system. Both sides of the onion skin paper were scanned in monochrome mode with a resolution of 100 dpi. A low resolution was purposely selected for scanning, so that the printer’s halftoning grid would not be strongly noticeable in the scanned images. The inner face of the scanner’s cover (facing the back page of the onion skin paper) was white, originating a strong mixture of the contents of both pages in the acquired images. After acquisition one of the images was horizontally flipped, to make the orientations of both images match. The two images were then coarsely aligned by hand, using alignment marks printed together with the images. It was found that even a careful alignment based just on those marks could not properly align all parts of the images, probably due to some slight geometrical distortions introduced by the scanner. Therefore, after the coarse manual alignment an automatic alignment procedure was run. For this purpose the images were first increased in resolution by a factor of four in each direction (using bicubic interpolation) so that the alignment could be made with a precision of 1/4 pixel. The alignment procedure operated on 100 × 100 pixel squares (corresponding to 25 × 25 pixel squares in the scanned images), and was based on finding the maximum of the local correlation between both images. After the automatic alignment the images were brought back to their original resolution. The preprocessing was completed
Lecture Notes in Computer Science
3
Fig. 2. Mixture components (preprocessed acquired images) in the ‘bars’ and ‘photos’ problems.
by scaling the intensity range of each image to the interval [0,1]. Figure 2 shows the mixture components after preprocessing.
3
Outline of the Separation Method
All separation tests, both linear and nonlinear, were based on the MISEP method. A detailed description of the method is given in [4]. A brief outline is given here, to clarify the main concepts and the nomenclature. The method is an extension of the well known INFOMAX method [5], and is based on the minimization of the mutual information of the extracted components. The structure of the system that is used for separation is shown in Fig. 3. The observations oi enter the F block, which performs the separation, yielding the separated components yi . The ψi blocks are used only during the training phase, and yield the auxiliary outputs zi . In INFOMAX the ψi blocks implement nonlinearities which are fixed a priori. In MISEP these blocks are adaptive, learning nonlinearities suited to the components’ statistical distributions. In the linear mode, MISEP uses a linear F block, and corresponds to INFOMAX with adaptive output nonlinearities. In the nonlinear mode, the F block is a nonlinear parameterized system (a multilayer perceptron – MLP – in our case). The whole system is trained by maximizing the joint entropy of the auxiliary outputs zi . This results both in
4
Lu´ıs B. Almeida et al. o1 F o2
y1
ψ1
z1
y2
ψ2
z2
Fig. 3. Structure of the system used for separation.
Fig. 4. Scatter plots (left to right): original photos, before printing; mixture; linear separation; nonlinear separation through the improved method.
the adaptation of the ψi blocks to the distributions of the extracted components and in the minimization of the mutual information of those components.
4
Results
All tests used a training set of 5000 points randomly selected from the preprocessed images. The ψi blocks were implemented with MLPs with one hidden layer of 10 sigmoidal units each. Figure 4 shows several scatter plots corresponding to the photos problem. The leftmost plot corresponds to the two original images, before printing, and shows that the images were not completely independent. The next plot corresponds to the preprocessed acquired images, and shows that the mixture was nonlinear: a linear mixture would correspond to a ‘parallelogram’ distortion of the original distribution. This scatter plot also shows that the mixture was almost singular in the lighter parts of the images. The remaining plots correspond to separation results, and are discussed ahead. 4.1
Linear Separation
Linear separation was used as a standard against which to compare the results of nonlinear separation. The linear separation system used the MISEP method as described in [4]. The F block was linear, performing just a product by the separation matrix. The separation results are shown in Fig. 51 . We can see that 1
All separated images were subject to a normalization of the intensity histogram before printing, to compensate for the nonlinearities that are sometimes introduced by nonlinear ICA [4]. This facilitates the comparison of separation results.
Lecture Notes in Computer Science
5
Fig. 5. Linear separation results.
only a partial separation was achieved, as expected. The third plot from the left, in Fig. 4, corresponds to the linearly separated components. It shows, again, that the mixture was nonlinear. A linear method can’t separate it completely. 4.2
Basic Nonlinear Separation
The first set of nonlinear separation tests used a separating system similar to those used in [4]. Block F consisted of an MLP with one hidden layer of sigmoidal units, with linear output units and with direct “shortcut” connections between inputs and outputs. The hidden layer had 20 units, 10 of which were connected to each of the output units. The separations shown both in this and in the next section were obtained with 1000 training epochs, corresponding to about 15 minutes in a 1.6 GHz Centrino processor programmed in Matlab. The results that were obtained had a relatively large variability, sometimes being better than those of linear separation, and sometimes worse. Figure 6 shows two “extremes” of the range os results that were obtained, for the component that had the largest variability in the photos problem. Somewhat infrequently (in about 10% of the tests) the system yielded results much outside this range. Normally the source images were then strongly mixed, in the extracted components. This variability is probably related to the ill-posedness of nonlinear ICA [6]. Attempts to make the outputs more stable by means of explicit regular-
6
Lu´ıs B. Almeida et al.
Fig. 6. Two “extreme” results of nonlinear separation with the basic method. The same extracted source is shown in both images, the difference being only in the random initialization of network weights. ×
MLP
×
Fig. 7. Structure of the F block in the improved nonlinear method.
ization through weight decay didn’t show much success: a weak regularization would not constrain the system enough, and a stronger regularization would make it essentially equivalent to a linear separator. 4.3
Improved Nonlinear Separation
The results of the basic nonlinear separation tests led us to try to incorporate more prior information in the separation system. The shape of the mixture distribution, together with a qualitative knowledge of the physical mixing process, led us to hypothesize that an F block with the structure shown in Fig. 7 would yield a more stable separation. In this structure each output is obtained by multiplying the corresponding input by a variable gain. The gains are computed by the MLP. These gains should be rather smooth functions, which we expected to be able to adequately constrain through regularization. The MLP that we used had a hidden layer of 20 sigmoidal units, linear output units and no direct connections from inputs to outputs. Ten of the hidden layer’s units were connected to each of the output units. The separation results produced by this network, with adequate regularization through weight decay, were significantly more stable than those form the basic nonlinear method (although, as with the basic method, the system pro-
Lecture Notes in Computer Science
7
Fig. 8. Nonlinear separation of the bars images with the improved nonlinear method.
duced rather wrong results in about 10% of the tests). Figures 8 and 9 show results for the two problems, obtained by training the system with the same set of parameters. Figure 9 also gives an idea of the degree of variability of the results on the photos problem. These results were somewhat better than those obtained with linear separation. The rightmost plot of Fig. 4 confirms that a better degree of independence was achieved, than with linear ICA. This plot also suggests, however, that the non-independence of the original images had somewhat a negative impact on the separation: in an attempt to “fill” the upper left corner of the distribution, the system tilted the upper part of the distribution somewhat, resulting in some amount of mixing in the extracted components. Both in the linear and nonlinear methods, the separation results show an amount of noise that is significantly higher than that of the mixture components. This may not be easily noticeable in the printed version of this paper, but should be visible to readers of the electronic version, by zooming in on the pictures. This noise probably comes mostly from a residual effect of the printer’s halftoning process and from slight inhomogeneities of the onion skin paper. The noise was then amplified by the inversion of the quasi-singular mixture.
5
Conclusion
We have shown results of separation of a real-life nonlinear mixture of images. The results show that nonlinear ICA can outperform linear ICA in this problem, but they also shows that the ill-posedness of nonlinear ICA makes stabilization of the separation somewhat difficult. Two main directions for improvement are envisaged, based on these results. On the one hand, it is desirable to develop better methods to stabilize the ICA results, possibly by making use of more prior information. On the other hand, it is of interest to develop separation criteria that are better suited to this specific problem than independence. These may yield better results, both in terms of quality and of stability, and may make the separation less affected by the possible statistical dependence of the original images.
8
Lu´ıs B. Almeida et al.
Fig. 9. Nonlinear separation of the photographs with the improved nonlinear method. The upper images correspond to “best” results, and the lower ones to “worst” ones.
References 1. Haritopoulos, M., Yin, H., Allinson, N.: Image denoising using SOM-based nonlinear independent component analysis. Neural Networks 15 (2002) 1085–1098 2. Lappalainen, H., Honkela, A.: Bayesian nonlinear independent component analysis by multi-layer perceptrons. In Girolami, M., ed.: Advances in Independent Component Analysis, Springer-Verlag (2000) 93–121 3. Lee, S.I., Batzoglou, S.: Application of independent component analysis to microarrays. Genome Biology 4 (2003) R76 http://genomebiology.com/2003/4/11/R76. 4. Almeida, L.B.: MISEP – Linear and nonlinear ICA based on mutual information. Journal of Machine Learning Research 4 (2003) 1297–1318 http://www.jmlr.org/papers/volume4/almeida03a/almeida03a.pdf. 5. Bell, A., Sejnowski, T.: An information-maximization approach to blind separation and blind deconvolution. Neural Computation 7 (1995) 1129–1159 6. Hyvarinen, A., Pajunen, P.: Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks 12 (1999) 429–439
Abstract. This manuscript presents results obtained using an ICA technique in a real-life nonlinear image separation problem: the separation of the images of the two pages of a paper document when the image from the back page shows through, superimposed on the image of the front page. For this manuscript, two images were printed on opposite sides of a sheet of onion skin paper, and then both sides of the sheet were scanned. The scanned images contained a markedly nonlinear mixture of the original images. Nonlinear ICA, using the MISEP technique, was used to recover the original images. It showed to be able to achieve a reasonable, but not perfect separation. The best results were obtained with a separating system which was somewhat customized, based on prior knowledge about the mixture process, and which used explicit regularization.
1
Introduction
When scanning or photographing a paper document, the image of the back page sometimes shows through. This is normally due to partial transparency of the paper or to bleeding of the ink through the paper. In either case, the image that is acquired consists of a mixture of the original images contained in each of the pages, and it would be of interest to be able to eliminate the superposition and recover the original images. Since it is possible to acquire both sides of the document, two different mixtures of the original images can be obtained, and therefore ICA is a natural candidate for source separation. Often, however, the mixture is substantially nonlinear, and linear ICA techniques are not adequate. This constitutes, therefore, an interesting test case for nonlinear ICA methods. We decided to implement a difficult version of this problem, using a relatively transparent paper (“onion skin”), resulting in a mixture that is both strong and significantly nonlinear. We show separation results obtained with (1) linear ICA, (2) nonlinear ICA (MISEP method) with the basic separating structure, and (3) nonlinear ICA (MISEP) using a customized separating structure that incorporates some knowledge about the mixing process. The latter results correspond to the best separation, which is still not perfect, leaving room for improvement. There are still very few published results of source separation on nonlinear mixtures of real-life data. An example is [1]. Other applications to real-life data, e.g. [2, 3], do not provide any means to assess whether the extracted components correspond to actual sources.
2
Lu´ıs B. Almeida et al.
Fig. 1. Photographs used for the second test problem. The right hand photograph has been horizontally flipped, to correspond with the position in which it appears in the mixture and separation images.
2
Source Images, Printing, Acquisition and Preprocessing
We present separation results on two test problems. For each problem we printed two gray scale images on opposite pages of onion skin paper. The images of the first problem were artificially generated, each of them consisting of parallel bars with randomly chosen gray levels. In one of the images the bars were oriented horizontally and in the other they were oriented vertically. We don’t present those images here to save space, but the results shown ahead clarify the images’ contents. The second pair of images consisted of the photos shown in Fig. 1. We used a monochrome laser printer at a resolution of 1200 dpi, with the printer’s default halftoning system. Both sides of the onion skin paper were scanned in monochrome mode with a resolution of 100 dpi. A low resolution was purposely selected for scanning, so that the printer’s halftoning grid would not be strongly noticeable in the scanned images. The inner face of the scanner’s cover (facing the back page of the onion skin paper) was white, originating a strong mixture of the contents of both pages in the acquired images. After acquisition one of the images was horizontally flipped, to make the orientations of both images match. The two images were then coarsely aligned by hand, using alignment marks printed together with the images. It was found that even a careful alignment based just on those marks could not properly align all parts of the images, probably due to some slight geometrical distortions introduced by the scanner. Therefore, after the coarse manual alignment an automatic alignment procedure was run. For this purpose the images were first increased in resolution by a factor of four in each direction (using bicubic interpolation) so that the alignment could be made with a precision of 1/4 pixel. The alignment procedure operated on 100 × 100 pixel squares (corresponding to 25 × 25 pixel squares in the scanned images), and was based on finding the maximum of the local correlation between both images. After the automatic alignment the images were brought back to their original resolution. The preprocessing was completed
Lecture Notes in Computer Science
3
Fig. 2. Mixture components (preprocessed acquired images) in the ‘bars’ and ‘photos’ problems.
by scaling the intensity range of each image to the interval [0,1]. Figure 2 shows the mixture components after preprocessing.
3
Outline of the Separation Method
All separation tests, both linear and nonlinear, were based on the MISEP method. A detailed description of the method is given in [4]. A brief outline is given here, to clarify the main concepts and the nomenclature. The method is an extension of the well known INFOMAX method [5], and is based on the minimization of the mutual information of the extracted components. The structure of the system that is used for separation is shown in Fig. 3. The observations oi enter the F block, which performs the separation, yielding the separated components yi . The ψi blocks are used only during the training phase, and yield the auxiliary outputs zi . In INFOMAX the ψi blocks implement nonlinearities which are fixed a priori. In MISEP these blocks are adaptive, learning nonlinearities suited to the components’ statistical distributions. In the linear mode, MISEP uses a linear F block, and corresponds to INFOMAX with adaptive output nonlinearities. In the nonlinear mode, the F block is a nonlinear parameterized system (a multilayer perceptron – MLP – in our case). The whole system is trained by maximizing the joint entropy of the auxiliary outputs zi . This results both in
4
Lu´ıs B. Almeida et al. o1 F o2
y1
ψ1
z1
y2
ψ2
z2
Fig. 3. Structure of the system used for separation.
Fig. 4. Scatter plots (left to right): original photos, before printing; mixture; linear separation; nonlinear separation through the improved method.
the adaptation of the ψi blocks to the distributions of the extracted components and in the minimization of the mutual information of those components.
4
Results
All tests used a training set of 5000 points randomly selected from the preprocessed images. The ψi blocks were implemented with MLPs with one hidden layer of 10 sigmoidal units each. Figure 4 shows several scatter plots corresponding to the photos problem. The leftmost plot corresponds to the two original images, before printing, and shows that the images were not completely independent. The next plot corresponds to the preprocessed acquired images, and shows that the mixture was nonlinear: a linear mixture would correspond to a ‘parallelogram’ distortion of the original distribution. This scatter plot also shows that the mixture was almost singular in the lighter parts of the images. The remaining plots correspond to separation results, and are discussed ahead. 4.1
Linear Separation
Linear separation was used as a standard against which to compare the results of nonlinear separation. The linear separation system used the MISEP method as described in [4]. The F block was linear, performing just a product by the separation matrix. The separation results are shown in Fig. 51 . We can see that 1
All separated images were subject to a normalization of the intensity histogram before printing, to compensate for the nonlinearities that are sometimes introduced by nonlinear ICA [4]. This facilitates the comparison of separation results.
Lecture Notes in Computer Science
5
Fig. 5. Linear separation results.
only a partial separation was achieved, as expected. The third plot from the left, in Fig. 4, corresponds to the linearly separated components. It shows, again, that the mixture was nonlinear. A linear method can’t separate it completely. 4.2
Basic Nonlinear Separation
The first set of nonlinear separation tests used a separating system similar to those used in [4]. Block F consisted of an MLP with one hidden layer of sigmoidal units, with linear output units and with direct “shortcut” connections between inputs and outputs. The hidden layer had 20 units, 10 of which were connected to each of the output units. The separations shown both in this and in the next section were obtained with 1000 training epochs, corresponding to about 15 minutes in a 1.6 GHz Centrino processor programmed in Matlab. The results that were obtained had a relatively large variability, sometimes being better than those of linear separation, and sometimes worse. Figure 6 shows two “extremes” of the range os results that were obtained, for the component that had the largest variability in the photos problem. Somewhat infrequently (in about 10% of the tests) the system yielded results much outside this range. Normally the source images were then strongly mixed, in the extracted components. This variability is probably related to the ill-posedness of nonlinear ICA [6]. Attempts to make the outputs more stable by means of explicit regular-
6
Lu´ıs B. Almeida et al.
Fig. 6. Two “extreme” results of nonlinear separation with the basic method. The same extracted source is shown in both images, the difference being only in the random initialization of network weights. ×
MLP
×
Fig. 7. Structure of the F block in the improved nonlinear method.
ization through weight decay didn’t show much success: a weak regularization would not constrain the system enough, and a stronger regularization would make it essentially equivalent to a linear separator. 4.3
Improved Nonlinear Separation
The results of the basic nonlinear separation tests led us to try to incorporate more prior information in the separation system. The shape of the mixture distribution, together with a qualitative knowledge of the physical mixing process, led us to hypothesize that an F block with the structure shown in Fig. 7 would yield a more stable separation. In this structure each output is obtained by multiplying the corresponding input by a variable gain. The gains are computed by the MLP. These gains should be rather smooth functions, which we expected to be able to adequately constrain through regularization. The MLP that we used had a hidden layer of 20 sigmoidal units, linear output units and no direct connections from inputs to outputs. Ten of the hidden layer’s units were connected to each of the output units. The separation results produced by this network, with adequate regularization through weight decay, were significantly more stable than those form the basic nonlinear method (although, as with the basic method, the system pro-
Lecture Notes in Computer Science
7
Fig. 8. Nonlinear separation of the bars images with the improved nonlinear method.
duced rather wrong results in about 10% of the tests). Figures 8 and 9 show results for the two problems, obtained by training the system with the same set of parameters. Figure 9 also gives an idea of the degree of variability of the results on the photos problem. These results were somewhat better than those obtained with linear separation. The rightmost plot of Fig. 4 confirms that a better degree of independence was achieved, than with linear ICA. This plot also suggests, however, that the non-independence of the original images had somewhat a negative impact on the separation: in an attempt to “fill” the upper left corner of the distribution, the system tilted the upper part of the distribution somewhat, resulting in some amount of mixing in the extracted components. Both in the linear and nonlinear methods, the separation results show an amount of noise that is significantly higher than that of the mixture components. This may not be easily noticeable in the printed version of this paper, but should be visible to readers of the electronic version, by zooming in on the pictures. This noise probably comes mostly from a residual effect of the printer’s halftoning process and from slight inhomogeneities of the onion skin paper. The noise was then amplified by the inversion of the quasi-singular mixture.
5
Conclusion
We have shown results of separation of a real-life nonlinear mixture of images. The results show that nonlinear ICA can outperform linear ICA in this problem, but they also shows that the ill-posedness of nonlinear ICA makes stabilization of the separation somewhat difficult. Two main directions for improvement are envisaged, based on these results. On the one hand, it is desirable to develop better methods to stabilize the ICA results, possibly by making use of more prior information. On the other hand, it is of interest to develop separation criteria that are better suited to this specific problem than independence. These may yield better results, both in terms of quality and of stability, and may make the separation less affected by the possible statistical dependence of the original images.
8
Lu´ıs B. Almeida et al.
Fig. 9. Nonlinear separation of the photographs with the improved nonlinear method. The upper images correspond to “best” results, and the lower ones to “worst” ones.
References 1. Haritopoulos, M., Yin, H., Allinson, N.: Image denoising using SOM-based nonlinear independent component analysis. Neural Networks 15 (2002) 1085–1098 2. Lappalainen, H., Honkela, A.: Bayesian nonlinear independent component analysis by multi-layer perceptrons. In Girolami, M., ed.: Advances in Independent Component Analysis, Springer-Verlag (2000) 93–121 3. Lee, S.I., Batzoglou, S.: Application of independent component analysis to microarrays. Genome Biology 4 (2003) R76 http://genomebiology.com/2003/4/11/R76. 4. Almeida, L.B.: MISEP – Linear and nonlinear ICA based on mutual information. Journal of Machine Learning Research 4 (2003) 1297–1318 http://www.jmlr.org/papers/volume4/almeida03a/almeida03a.pdf. 5. Bell, A., Sejnowski, T.: An information-maximization approach to blind separation and blind deconvolution. Neural Computation 7 (1995) 1129–1159 6. Hyvarinen, A., Pajunen, P.: Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks 12 (1999) 429–439
