IMPROVED COLOR FILTER ARRAY DEMOSAICKING BY ... - CiteSeerX
IMPROVED COLOR FILTER ARRAY DEMOSAICKING BY ACCURATE LUMINANCE ESTIMATION Naixiang Lian Lanlan Chang
and Yap-Peng Tan
School of Electrical and Electronic Engineering Nanyang Technological University, Singapore ABSTRACT Luminance information plays an important role in dictating the quality of a color image, and some existing color filter array (CFA) demosaicking methods achieve superiority by first reconstructing a satisfactory luminance plane. It is however difficult to accurately estimate the luminance from CFA samples since the color spectra are generally aliased. Extending a state-of-the-art luminance-based demosaicking scheme, in this paper we propose an improved demosaicking method using an efficient filter to estimate the luminance at green pixels and employing an effective edge-adaptive interpolation scheme to obtain the luminance at red and blue pixels. Experimental results demonstrate that not only is the proposed method less complex, it also performs noticeably better, both visually and in terms of peak signal-to-noise ratio, comparing with several recent methods. 1. INTRODUCTION Most digital still cameras capture imagery using a single electronic sensor overlaid with a color filter array (CFA). Today the most commonly used CFA pattern today is “Bayer” pattern [1], a schematic of which is shown in Fig. 1 (a). Thus in the captured CFA image, each pixel contains only one of the three primary colors. To restore a full color image from the CFA samples, the two missing color values at each pixel need to be estimated. This process is referred to as CFA demosaicking. G
R
G
R
G
R
G
R
G
R
G
R
G
R
G
R
B
G
B
G
B
G
B
G
B
G
B
G
B
G
B
G
G
R
G
R
G
R
G
R
G
R
G
R
G
R
G
R
B
G
B
G
B
G
B
G
B
G
B
G
B
G
B
G
(a)
(b)
(c)
(d)
Fig. 1. (a) A Bayer color filter array pattern, and its (b) red, (c) green, and (d) blue samples. A Bayer CFA image consists of 50% green, 25% red, and 25% blue samples. This particular arrangement stems from the fact that the sensitivity of the human visual system (HVS) to the luminance is dominated by the green spectrum. The importance of an image’s luminance is also articulated well by Hunt: “An accurate and sharp luminance signal is just as important for color photography as it is for black and white photography” [2]. However, since the CFA samples do not possess the complete color information at each pixel, reconstructing the full-resolution luminance from a CFA image has been known to be difficult. Recently, based on the observation that the luminance and chrominance signals of a CFA image are well separated in the frequency domain, Alleysson et al. have made a novel attempt to estimate the luminance of an image by low-pass filtering its CFA samples and obtained good demosaicked results [3].
In this paper, we show that luminance estimated by Alleysson’s method is not optimal and could lead to visible demosaicking artifacts, as illustrated in Fig. 4. Specifically, by gaining a new insight into the CFA spectrum, we obtain a more efficient filter to estimate the luminance at green pixels. Furthermore, an effective edgeadaptive interpolation scheme is also devised to obtain more accurate luminance at red and blue pixels. Experimental results demonstrate that not only is the proposed method less complex, it also performs better, both visually and in terms of peak signal-to-noise ratio, compared with the existing state-of-the-art methods. The remainder of this paper is organized as follows. Section 2 discusses the properties of the CFA spectrum and the present filtering scheme for obtaining the luminance from a CFA image. Section 3 presents our proposed demosaicking method. Section 4 reports the experimental results of our proposed method in comparison with those obtained by four existing state-of-the-art methods. In Section 5, we conclude the paper by summarizing our contributions. 2. CFA SPECTRUM AND LUMINANCE ESTIMATION A CFA image can be decomposed into three sub-sampled color planes as shown Fig. 1(b)-(d). Let RO , GO , BO be the red, green and blue planes of the original full-resolution color image, and RC , GC , BC the corresponding sub-sampled color planes in the CFA image. Their relationship can be formulated as follows [3]: RC (x, y)
=
GC (x, y)
=
BC (x, y)
=
1 RO (x, y)(1 − cos πx)(1 + cos πy) 4 1 GO (x, y)(1 + cos πx cos πy) 2 1 BO (x, y)(1 + cos πx)(1 − cos πy) 4
(1)
Combining the terms in (1), a CFA image can be represented as: ICF A (x, y) = RC (x, y) + GC (x, y) + BC (x, y) 1 = [RO (x, y) + 2GO (x, y) + BO (x, y)] 4 1 + [BO (x, y) − RO (x, y)](cos πx − cos πy) (2) 4 1 + [−RO (x, y) + 2GO (x, y) − BO (x, y)] cos πx cos πy 4 where (RO + 2GO + BO ) can be regarded as the luminance of the CFA image, while (RO − BO )) and (−RO + 2GO − BO ) the color differences encoding the chrominance of the CFA image. In the Fourier transform domain, the modulation functions (cos πx − cos πy) and cos πx cos πy essentially shift the energy spectra of the color differences (RO −BO )) and (−RO +2GO −BO ) to the sides (Fx = ±0.5 and Fy = 0 or Fx = 0 and Fy = ±0.5) and the corners (Fx = ±0.5 and Fy = ±0.5) of the frequency plane, respectively. One example of this frequency modulation is shown in
Fig. 2, where the luminance spectrum of a color image is shown in Fig. 2(a), and the Fourier transform of the corresponding CFA image is shown in Fig. 2(b). We can see from Fig. 2(b) that, besides the central luminance spectrum, the color difference spectra are shifted to the four sides and corners of the frequency plane. Since the luminance and color differences of a color image are not necessary band-limited, they are likely to be aliased due to CFA sampling and could lead to various artifacts after demosaicking.
4
(c) demosaicked image
(d) estimated luminance
x 10
6
6
5
5
4
4 Magnitude
Magnitude
(b) luminance
4
x 10
3 2
Fig. 4. The (a) original image, (b) original luminance, (c) demosaicked image, and (d) estimated luminance by method in [3].
3 2
1
1
0 0.5
0 0.5 0.5 0
0.5
Table 1. The mean-squared errors (MSE) of the luminance at the green pixels obtained by different low-pass filters.
0 0 −0.5
F
−0.5
0
F
Fx
y
−0.5
−0.5
y
(a)
Fx
(b)
Fig. 2. (a) Luminance spectrum and (b) CFA spectrum of an image. 2.1. Estimating Luminance by Low-pass Filtering Assuming that the spectra of the luminance and color-difference signals are Gaussian-like distributed, Alleysson et al. propose to use a low-pass filter with frequency response as Fig. 3(a) for extracting the luminance from the CFA spectrum [3]. The filter is optimized by adjusting two parameters r1 and r2 , which denote the diameters of the Gaussian functions representing the color-difference signals centered at the corners and sides of the frequency plane, respectively. After the luminance is obtained, the color-difference signals are estimated by subtracting the luminance from the CFA image, i.e., essentially a high-pass filtering process. Promising results have been obtained by this method as reported in [3]. However, since the bandwidths of the luminance and color-difference signals of different color images or image regions are not likely the same, using a filter with preoptimized parameters r1 and r2 may not work well for all images. Estimating the luminance inaccurately from a CFA image will lead to such demosaicking artifacts as zipper effects, which are abrupt intensity changes in regions around edges and borders of image regions and objects. Fig. 4(c) shows an example of the zipper effects presenting in a demosaicked image obtained by Alleysson’s method.
00001010000 00010201000
1
00112121100 0.8
01153935110 Magnitude
(a) original image
0.6
10231713201
0.4
0219710479120
0.2
10231713201
/128
01153935110
0 1 1
0.5 0.5
0 0 −0.5 Fy
−0.5 −1
−1
(a)
Fx
00112121100 00010201000 00001010000
(b)
Fig. 3. (a) Frequency response and (b) impulse response of the filter proposed in [3] for estimating the luminance from a CFA image. 3. PROPOSED DEMOSAICKING METHOD 3.1. Low-pass filtering for Luminance at Green Pixels One key insight of our proposed method is that the coordinates of the quincuncial sampled green (G) pixels in a CFA image satisfy particular property, such as x + y = even in Fig. 1 (c). It follows that
Image 1 2 3 4 5 6 7 8 9 10 Average
11 × 11 filter r2 = 0 9.04 2.62 1.64 2.37 4.85 7.14 1.99 21.73 2.26 1.88 5.55
11 × 11 filter r2 = 0 1.16 1.09 0.85 1.05 2.08 0.84 0.73 1.87 0.69 0.70 1.11
5 × 5 filter r2 = 0 1.86 0.85 0.88 1.09 2.54 1.25 0.77 2.59 0.69 0.73 1.33
cos π(x + y) = 1, sin π(x + y) = 0, and cos πx
= = =
cos π(x + y − y) cos π(x + y) cos πy + sin π(x + y) sin πy cos πy (3)
for G pixels in a CFA image. Substituting (3) into (2), the CFA image at the G pixels can be expressed as: ICF A (x, y) =
1 [RO (x, y) + 2GO (x, y) + BO (x, y)] 4
(4)
1 + [−RO (x, y) + 2GO (x, y) − BO (x, y)] cos πx cos πy, 4 which indicates that at the G pixels the spectra of the color-difference signals at sides of horizontal and vertical axes are cancelled. Same conclusion can be obtained for the other Bayer CFA patterns, for example the first pixel is red or blue sample. Consequently, as color-difference signals are cancelled at the sides of the frequency plane, filtering the CFA image for luminance at the green pixels with a low-pass filter like that in Fig. 3(a) will incorrectly discard part of the luminance information. To preserve more accurately the luminance at the green pixels, the diameter parameter r2 should be set to zero to retain the luminance spectra at the sides of the frequency plane. We verify this point by experiments. The first two columns of Table 1 show the mean-squared errors (MSE) in estimating the luminance of 10 test images by using the 11 × 11 filter without or with setting r2 = 0. We can see that the MSE values obtained by the filter with r2 = 0 are generally much smaller. Since the spectra of the color-difference signals at the four corners have less aliasing with the luminance spectra, a filter with a larger support (say, 11 × 11) is not necessary. We propose to use a 5 × 5 low-pass filter as shown in Fig.(5) for estimating the luminance at the green pixels. The last column of Table 1 shows that the MSE values obtained by the proposed filter is very close to those obtained by a
1.4 1.2
Magnitude
1 0.8 0.6 0.4 0.2 0 1 1
0.5 0.5
0 0 −0.5
−0.5 −1
Fy
−1
F
x
(a)
(b)
Fig. 5. (a) Frequency response and (b) impulse response of our proposed low-pass filter for estimating luminance at the green pixels. 11 × 11 filter as shown in the second column. An obvious advantage of the proposed filter is its low computational cost. Compared with the original 11 × 11 filter (Fig. 3(a)), our proposed filter reduces the multiplications from 41 to 13 and additions from 69 to 21 counts, respectively. 3.2. Adaptive Scheme for Luminance at Red and Blue Pixels From the discussion in Section 2.1, a low-pass filter like Fig.3(a) is required to obtain the luminance at the red and blue pixels, since the property shown in (3) does not hold at these pixels. However, as it is also not optimal to use a filter with fixed r1 and r2 parameters for all images, we propose to reconstruct the luminance values at the red and blue pixels by using an edge-adaptive scheme to exploit both the image spatial and spectral correlation. R
L
R
L
R
L
B
L
B
L
R
L
Rm,n
L
R
L
B
L
B
L
R
L
R
L
R
(a)
B
Lr
B
Lr
R m,n
Lr
B
Lr
B
(b)
1 (R(x, y) + 2G(x, y) + B(x, y)) 4
(5)
We can define three ‘chrominance’ (luminance-color difference) planes as = = =
− 1, y) = 1 (R(x − 2, y) + R(x, y)) R(x 2 − 1, y) R (x − 1, y) = L(x − 1, y) − R(x L w(x − 1, y) = 1/(1 + |R(x, y) − R(x − 2, y)| +|L(x + 1, y) − L(x − 1, y)|)
y) = R(x, y) + L(x,
Fig. 6(a) shows the CFA image, in which the luminance at all the green pixels have been estimated. As illustrated in Section 2, the luminance at pixel (x, y) can be expressed as:
LR (x, y) LG (x, y) LB (x, y)
estimation processes at the red and blue pixels are the same, we shall describe in the following only the estimation at the red (R) pixels. Referring to Fig. 6(b), to obtain the luminance L(x, y) value at R(x, y) pixel, the chrominance LR (x, y) values at the four surrounding pixels (x − 1, y), (x + 1, y), (x, y − 1) and (x, y + 1) are first computed. Since only the luminance and green values are available at these neighboring pixels, the LR values are obtained by estimating the R values from the two nearest red samples by bilinear interpolation. Specifically, the R, LR , and w values at pixel (x − 1, y) are obtained respectively as:
L(x, y) − R(x, y) L(x, y) − G(x, y) L(x, y) − B(x, y),
(7) (8) (9)
The luminance L(x, y) value is then obtained as
Fig. 6. (a) CFA pattern with the estimated luminance at green pixels. (b) Reference neighboring samples.
L(x, y) =
Fig. 7. (a) Luminance L and chrominance components (b) LR , (c) LG , and (d) LB of a test image.
(6)
These three chrominance planes are generally smooth because the luminance plane and each color planes are highly correlated. To see this, Fig. 7 shows the luminance and the three chrominance planes of a test image. Exploiting this strong spatial correlation in the chrominance planes, we estimate the luminance at the red and blue pixels by adaptively combining the neighboring chrominance values. Since the
R (p)) (w(p) × L , p∈Ω w(p)
p∈Ω
(10)
where Ω = {(x − 1, y), (x + 1, y), (x, y − 1), (x, y + 1)}. The luminance at the other red pixels can be similarly obtained. At the end of this step, a full-resolution luminance plane can be obtained. With the full-resolution luminance available, we can update the red and blue estimates to gain higher accuracy. Specifically, we calculate the LR (x, y) and LR (x − 2, y) by using the current luminance estimates, and then update the red value at pixel (x − 1, y) as R (x − 2, y)). (11) R (x, y) + L − 1, y) = L(x − 1, y) + 1 (L R(x 2 Using the more accurate red and blue estimates obtained, we can update the luminance estimation using (10). Our experiments show that the proposed method can obtain more accurate luminance at red and blues pixels as compared to Alleysson’s method [3]. 3.3. Chrominance Estimation Finally, we obtain the sub-sampled chrominance planes LR , LG , and LB by subtracting the CFA image from the full-resolution, estimated luminance plane. Because the chrominance planes contain little spatial details, bilinear interpolation is used to populate the chrominance planes. We found that using a more sophisticated interpolation scheme in this step could only attain very marginal performance gain. It can be shown that the overall computational complexity of our proposed method is close to that of the Alleysson’s method [3].
4. EXPERIMENTAL RESULTS To evaluate the performance of our proposed method, we compare its demosaicking results with those obtained by four recent state-of-theart methods which can obtain superior results, including Gunturk’s alternating projections method (AP) [4], Wu’s primary-consistent soft-decision (PCSD) method [5], Li’s successive approximation (SA) demosaicking method [6], and Alleysson’s frequency selection (FS) demosaicking method [3]. Fig. 8 shows our test image set; the images in the test set have been widely used for CFA demosaicking researches and experiments in the literature [3–6].
Table 2. Performance Comparison—PSNR results (in dB) of the red, green and blue color planes are listed in the 1st, 2nd and 3rd rows of each image. The best PSNR result of each row is underlined. Image 1 2 3 4 5 6 7 8 9 10 11 12
Fig. 8. Test images (referred as Image 1 to Image 24, enumerated from left-to-right and top-to-bottom.) In what follows, we evaluate the performance of the proposed method from two aspects: subjective visual evaluation and objective peak signal-to-noise ratio (PSNR) comparison.
(a) Original
(b) AP
(c) PCSD
(d) SA
AP
PCSD
SA
FS
Proposed
36.59 40.49 36.86 35.87 40.81 38.60 40.43 43.48 40.39 36.68 43.33 41.42 36.82 39.81 36.55 37.72 41.52 37.27 40.84 43.62 39.91 34.06 38.59 34.29 40.54 43.26 41.43 40.55 43.89 40.68 37.86 41.64 39.06 41.12 44.90 41.40
32.89 38.74 33.04 36.70 42.48 38.82 40.75 44.46 40.42 36.49 42.89 39.68 34.69 39.69 34.35 36.11 41.36 35.54 39.96 43.63 39.47 31.25 37.38 31.25 39.43 43.73 40.14 39.01 43.97 39.26 35.61 40.84 36.05 40.13 45.18 40.54
36.66 40.88 37.64 35.54 40.06 39.07 39.08 42.21 39.36 36.17 42.64 40.88 35.59 38.27 35.34 37.22 41.40 36.67 40.12 42.67 39.35 34.44 38.74 35.00 40.60 43.33 41.36 40.12 43.86 40.56 37.60 41.70 38.69 41.07 44.71 41.39
35.06 39.64 35.45 34.04 39.70 36.57 38.68 42.20 38.49 35.21 42.99 40.76 34.97 38.47 34.80 36.25 40.81 35.67 38.89 42.44 38.08 31.47 36.63 31.64 38.68 42.34 39.74 39.28 43.55 40.14 36.45 41.05 37.51 39.05 43.62 39.51
36.68 39.92 36.90 37.71 43.71 40.10 42.19 45.63 41.41 37.16 44.67 42.63 37.51 40.84 36.86 37.70 40.74 36.85 42.27 45.61 40.93 34.54 38.21 34.53 41.37 45.17 43.07 41.17 45.54 42.06 38.28 41.81 38.98 41.75 45.77 41.86
Image 13 14 15 16 17 18 19 20 21 22 23 24 Average
AP
PCSD
SA
FS
Proposed
34.05 36.91 33.16 34.79 38.37 35.62 36.37 41.49 39.31 40.80 44.88 40.62 40.78 42.92 40.22 35.52 38.64 36.36 37.90 41.85 39.23 39.25 42.72 38.01 38.26 41.63 37.55 37.09 39.58 36.81 37.61 41.64 40.42 34.87 37.33 32.70 39.41 39.09 39.43
29.96 34.83 29.54 34.06 38.21 34.56 36.04 41.71 38.46 40.23 44.99 39.87 38.39 42.17 38.05 33.46 37.76 33.46 36.39 41.26 36.80 38.04 42.43 36.95 34.85 39.93 35.12 35.26 39.46 35.12 40.74 44.64 41.33 33.12 37.14 31.55 38.25 38.13 38.37
34.70 37.90 33.35 33.33 36.74 34.39 35.34 40.58 38.83 38.66 43.40 39.40 40.71 43.13 39.93 35.51 38.61 36.43 38.55 42.33 39.63 38.75 41.74 37.22 37.83 41.61 36.98 36.93 39.63 36.87 38.06 41.83 40.54 34.79 37.59 32.80 39.07 38.77 39.09
34.57 37.72 33.30 32.44 36.88 33.17 34.85 40.55 37.45 39.16 43.99 38.85 40.06 43.06 40.03 34.90 38.37 35.55 35.75 40.46 36.73 38.59 41.97 36.60 36.93 40.98 36.22 36.40 39.28 35.51 36.97 41.27 37.65 35.29 37.83 32.74 38.15 37.89 38.22
33.68 35.98 32.74 36.10 40.52 36.22 37.19 43.13 40.24 40.72 43.83 40.22 40.96 43.45 40.69 35.57 38.79 36.35 38.68 42.30 39.99 40.14 43.59 38.39 38.34 41.18 37.38 37.81 41.20 37.42 38.94 44.93 40.75 35.04 37.51 32.83 40.07 39.77 40.10
ing demosaicked results obtained by our proposed method and other methods under comparison. Observably, our proposed method produces more visually pleasing results with fewer zipper effect artifacts. The PSNR results of the methods under comparison are listed in Table 2. The table shows that our proposed method outperforms others in terms of PSNR for most test images. 5. CONCLUSION We have proposed a new demosaicking method based on estimating more accurate luminance from CFA samples. By analyzing the spectrum of a CFA image, we identified that the luminance and chrominance components at the green pixels are better separated in the frequency domain than those at red and blue pixels. Driven by this property, our algorithm employs an efficient filter to estimate more accurately the luminance at green pixels and then uses an edge-adaptive scheme to reconstruct more effectively the luminance at the red/blue samples by exploiting both the image spectral and spatial correlations. Compared with the four recent state-of-the-art methods, not only is our proposed method less computational complex, it also obtains more visually pleasing results with higher PSNR for most of the test images. 6. REFERENCES
(e) FS
(f) Proposed
Fig. 9. The original and demosaicked results of a cropped region from Image 7. Fig. 9 shows a cropped region of a test image and its correspond-
[1] B. Bayer, “Color imaging array,” United States Patent, no. 3,971,065, 1976. [2] R.Hunt, “Why is black-and-white so important in color?,” 4th IS & T/SID Color Image Conference Scottsdale, Arizona, pp. 54–57, 1995. [3] D. Alleysson, S. Susstrunk, and J. Herault, “Linear demosaicking inspired by human visual system,” IEEE Transactions on Image Processing, vol. 14, no. 4, pp. 439–449, 2005. [4] B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, “Color plane interpolation using alternating projections,” IEEE Transcations on Image Processing, vol. 11, no. 9, 2002. [5] X. Wu and N. Zhang, “Primary-consistent soft-decision color demosaicking for digital cameras,” IEEE Trans. on Image Processing, vol. 13, pp. 1263–1274, 2004. [6] X. Li, “Demosaicing by successive approximation,” IEEE Transcations on Image Processing, vol. 14, no. 3, pp. 370–379, 2005.
and Yap-Peng Tan
School of Electrical and Electronic Engineering Nanyang Technological University, Singapore ABSTRACT Luminance information plays an important role in dictating the quality of a color image, and some existing color filter array (CFA) demosaicking methods achieve superiority by first reconstructing a satisfactory luminance plane. It is however difficult to accurately estimate the luminance from CFA samples since the color spectra are generally aliased. Extending a state-of-the-art luminance-based demosaicking scheme, in this paper we propose an improved demosaicking method using an efficient filter to estimate the luminance at green pixels and employing an effective edge-adaptive interpolation scheme to obtain the luminance at red and blue pixels. Experimental results demonstrate that not only is the proposed method less complex, it also performs noticeably better, both visually and in terms of peak signal-to-noise ratio, comparing with several recent methods. 1. INTRODUCTION Most digital still cameras capture imagery using a single electronic sensor overlaid with a color filter array (CFA). Today the most commonly used CFA pattern today is “Bayer” pattern [1], a schematic of which is shown in Fig. 1 (a). Thus in the captured CFA image, each pixel contains only one of the three primary colors. To restore a full color image from the CFA samples, the two missing color values at each pixel need to be estimated. This process is referred to as CFA demosaicking. G
R
G
R
G
R
G
R
G
R
G
R
G
R
G
R
B
G
B
G
B
G
B
G
B
G
B
G
B
G
B
G
G
R
G
R
G
R
G
R
G
R
G
R
G
R
G
R
B
G
B
G
B
G
B
G
B
G
B
G
B
G
B
G
(a)
(b)
(c)
(d)
Fig. 1. (a) A Bayer color filter array pattern, and its (b) red, (c) green, and (d) blue samples. A Bayer CFA image consists of 50% green, 25% red, and 25% blue samples. This particular arrangement stems from the fact that the sensitivity of the human visual system (HVS) to the luminance is dominated by the green spectrum. The importance of an image’s luminance is also articulated well by Hunt: “An accurate and sharp luminance signal is just as important for color photography as it is for black and white photography” [2]. However, since the CFA samples do not possess the complete color information at each pixel, reconstructing the full-resolution luminance from a CFA image has been known to be difficult. Recently, based on the observation that the luminance and chrominance signals of a CFA image are well separated in the frequency domain, Alleysson et al. have made a novel attempt to estimate the luminance of an image by low-pass filtering its CFA samples and obtained good demosaicked results [3].
In this paper, we show that luminance estimated by Alleysson’s method is not optimal and could lead to visible demosaicking artifacts, as illustrated in Fig. 4. Specifically, by gaining a new insight into the CFA spectrum, we obtain a more efficient filter to estimate the luminance at green pixels. Furthermore, an effective edgeadaptive interpolation scheme is also devised to obtain more accurate luminance at red and blue pixels. Experimental results demonstrate that not only is the proposed method less complex, it also performs better, both visually and in terms of peak signal-to-noise ratio, compared with the existing state-of-the-art methods. The remainder of this paper is organized as follows. Section 2 discusses the properties of the CFA spectrum and the present filtering scheme for obtaining the luminance from a CFA image. Section 3 presents our proposed demosaicking method. Section 4 reports the experimental results of our proposed method in comparison with those obtained by four existing state-of-the-art methods. In Section 5, we conclude the paper by summarizing our contributions. 2. CFA SPECTRUM AND LUMINANCE ESTIMATION A CFA image can be decomposed into three sub-sampled color planes as shown Fig. 1(b)-(d). Let RO , GO , BO be the red, green and blue planes of the original full-resolution color image, and RC , GC , BC the corresponding sub-sampled color planes in the CFA image. Their relationship can be formulated as follows [3]: RC (x, y)
=
GC (x, y)
=
BC (x, y)
=
1 RO (x, y)(1 − cos πx)(1 + cos πy) 4 1 GO (x, y)(1 + cos πx cos πy) 2 1 BO (x, y)(1 + cos πx)(1 − cos πy) 4
(1)
Combining the terms in (1), a CFA image can be represented as: ICF A (x, y) = RC (x, y) + GC (x, y) + BC (x, y) 1 = [RO (x, y) + 2GO (x, y) + BO (x, y)] 4 1 + [BO (x, y) − RO (x, y)](cos πx − cos πy) (2) 4 1 + [−RO (x, y) + 2GO (x, y) − BO (x, y)] cos πx cos πy 4 where (RO + 2GO + BO ) can be regarded as the luminance of the CFA image, while (RO − BO )) and (−RO + 2GO − BO ) the color differences encoding the chrominance of the CFA image. In the Fourier transform domain, the modulation functions (cos πx − cos πy) and cos πx cos πy essentially shift the energy spectra of the color differences (RO −BO )) and (−RO +2GO −BO ) to the sides (Fx = ±0.5 and Fy = 0 or Fx = 0 and Fy = ±0.5) and the corners (Fx = ±0.5 and Fy = ±0.5) of the frequency plane, respectively. One example of this frequency modulation is shown in
Fig. 2, where the luminance spectrum of a color image is shown in Fig. 2(a), and the Fourier transform of the corresponding CFA image is shown in Fig. 2(b). We can see from Fig. 2(b) that, besides the central luminance spectrum, the color difference spectra are shifted to the four sides and corners of the frequency plane. Since the luminance and color differences of a color image are not necessary band-limited, they are likely to be aliased due to CFA sampling and could lead to various artifacts after demosaicking.
4
(c) demosaicked image
(d) estimated luminance
x 10
6
6
5
5
4
4 Magnitude
Magnitude
(b) luminance
4
x 10
3 2
Fig. 4. The (a) original image, (b) original luminance, (c) demosaicked image, and (d) estimated luminance by method in [3].
3 2
1
1
0 0.5
0 0.5 0.5 0
0.5
Table 1. The mean-squared errors (MSE) of the luminance at the green pixels obtained by different low-pass filters.
0 0 −0.5
F
−0.5
0
F
Fx
y
−0.5
−0.5
y
(a)
Fx
(b)
Fig. 2. (a) Luminance spectrum and (b) CFA spectrum of an image. 2.1. Estimating Luminance by Low-pass Filtering Assuming that the spectra of the luminance and color-difference signals are Gaussian-like distributed, Alleysson et al. propose to use a low-pass filter with frequency response as Fig. 3(a) for extracting the luminance from the CFA spectrum [3]. The filter is optimized by adjusting two parameters r1 and r2 , which denote the diameters of the Gaussian functions representing the color-difference signals centered at the corners and sides of the frequency plane, respectively. After the luminance is obtained, the color-difference signals are estimated by subtracting the luminance from the CFA image, i.e., essentially a high-pass filtering process. Promising results have been obtained by this method as reported in [3]. However, since the bandwidths of the luminance and color-difference signals of different color images or image regions are not likely the same, using a filter with preoptimized parameters r1 and r2 may not work well for all images. Estimating the luminance inaccurately from a CFA image will lead to such demosaicking artifacts as zipper effects, which are abrupt intensity changes in regions around edges and borders of image regions and objects. Fig. 4(c) shows an example of the zipper effects presenting in a demosaicked image obtained by Alleysson’s method.
00001010000 00010201000
1
00112121100 0.8
01153935110 Magnitude
(a) original image
0.6
10231713201
0.4
0219710479120
0.2
10231713201
/128
01153935110
0 1 1
0.5 0.5
0 0 −0.5 Fy
−0.5 −1
−1
(a)
Fx
00112121100 00010201000 00001010000
(b)
Fig. 3. (a) Frequency response and (b) impulse response of the filter proposed in [3] for estimating the luminance from a CFA image. 3. PROPOSED DEMOSAICKING METHOD 3.1. Low-pass filtering for Luminance at Green Pixels One key insight of our proposed method is that the coordinates of the quincuncial sampled green (G) pixels in a CFA image satisfy particular property, such as x + y = even in Fig. 1 (c). It follows that
Image 1 2 3 4 5 6 7 8 9 10 Average
11 × 11 filter r2 = 0 9.04 2.62 1.64 2.37 4.85 7.14 1.99 21.73 2.26 1.88 5.55
11 × 11 filter r2 = 0 1.16 1.09 0.85 1.05 2.08 0.84 0.73 1.87 0.69 0.70 1.11
5 × 5 filter r2 = 0 1.86 0.85 0.88 1.09 2.54 1.25 0.77 2.59 0.69 0.73 1.33
cos π(x + y) = 1, sin π(x + y) = 0, and cos πx
= = =
cos π(x + y − y) cos π(x + y) cos πy + sin π(x + y) sin πy cos πy (3)
for G pixels in a CFA image. Substituting (3) into (2), the CFA image at the G pixels can be expressed as: ICF A (x, y) =
1 [RO (x, y) + 2GO (x, y) + BO (x, y)] 4
(4)
1 + [−RO (x, y) + 2GO (x, y) − BO (x, y)] cos πx cos πy, 4 which indicates that at the G pixels the spectra of the color-difference signals at sides of horizontal and vertical axes are cancelled. Same conclusion can be obtained for the other Bayer CFA patterns, for example the first pixel is red or blue sample. Consequently, as color-difference signals are cancelled at the sides of the frequency plane, filtering the CFA image for luminance at the green pixels with a low-pass filter like that in Fig. 3(a) will incorrectly discard part of the luminance information. To preserve more accurately the luminance at the green pixels, the diameter parameter r2 should be set to zero to retain the luminance spectra at the sides of the frequency plane. We verify this point by experiments. The first two columns of Table 1 show the mean-squared errors (MSE) in estimating the luminance of 10 test images by using the 11 × 11 filter without or with setting r2 = 0. We can see that the MSE values obtained by the filter with r2 = 0 are generally much smaller. Since the spectra of the color-difference signals at the four corners have less aliasing with the luminance spectra, a filter with a larger support (say, 11 × 11) is not necessary. We propose to use a 5 × 5 low-pass filter as shown in Fig.(5) for estimating the luminance at the green pixels. The last column of Table 1 shows that the MSE values obtained by the proposed filter is very close to those obtained by a
1.4 1.2
Magnitude
1 0.8 0.6 0.4 0.2 0 1 1
0.5 0.5
0 0 −0.5
−0.5 −1
Fy
−1
F
x
(a)
(b)
Fig. 5. (a) Frequency response and (b) impulse response of our proposed low-pass filter for estimating luminance at the green pixels. 11 × 11 filter as shown in the second column. An obvious advantage of the proposed filter is its low computational cost. Compared with the original 11 × 11 filter (Fig. 3(a)), our proposed filter reduces the multiplications from 41 to 13 and additions from 69 to 21 counts, respectively. 3.2. Adaptive Scheme for Luminance at Red and Blue Pixels From the discussion in Section 2.1, a low-pass filter like Fig.3(a) is required to obtain the luminance at the red and blue pixels, since the property shown in (3) does not hold at these pixels. However, as it is also not optimal to use a filter with fixed r1 and r2 parameters for all images, we propose to reconstruct the luminance values at the red and blue pixels by using an edge-adaptive scheme to exploit both the image spatial and spectral correlation. R
L
R
L
R
L
B
L
B
L
R
L
Rm,n
L
R
L
B
L
B
L
R
L
R
L
R
(a)
B
Lr
B
Lr
R m,n
Lr
B
Lr
B
(b)
1 (R(x, y) + 2G(x, y) + B(x, y)) 4
(5)
We can define three ‘chrominance’ (luminance-color difference) planes as = = =
− 1, y) = 1 (R(x − 2, y) + R(x, y)) R(x 2 − 1, y) R (x − 1, y) = L(x − 1, y) − R(x L w(x − 1, y) = 1/(1 + |R(x, y) − R(x − 2, y)| +|L(x + 1, y) − L(x − 1, y)|)
y) = R(x, y) + L(x,
Fig. 6(a) shows the CFA image, in which the luminance at all the green pixels have been estimated. As illustrated in Section 2, the luminance at pixel (x, y) can be expressed as:
LR (x, y) LG (x, y) LB (x, y)
estimation processes at the red and blue pixels are the same, we shall describe in the following only the estimation at the red (R) pixels. Referring to Fig. 6(b), to obtain the luminance L(x, y) value at R(x, y) pixel, the chrominance LR (x, y) values at the four surrounding pixels (x − 1, y), (x + 1, y), (x, y − 1) and (x, y + 1) are first computed. Since only the luminance and green values are available at these neighboring pixels, the LR values are obtained by estimating the R values from the two nearest red samples by bilinear interpolation. Specifically, the R, LR , and w values at pixel (x − 1, y) are obtained respectively as:
L(x, y) − R(x, y) L(x, y) − G(x, y) L(x, y) − B(x, y),
(7) (8) (9)
The luminance L(x, y) value is then obtained as
Fig. 6. (a) CFA pattern with the estimated luminance at green pixels. (b) Reference neighboring samples.
L(x, y) =
Fig. 7. (a) Luminance L and chrominance components (b) LR , (c) LG , and (d) LB of a test image.
(6)
These three chrominance planes are generally smooth because the luminance plane and each color planes are highly correlated. To see this, Fig. 7 shows the luminance and the three chrominance planes of a test image. Exploiting this strong spatial correlation in the chrominance planes, we estimate the luminance at the red and blue pixels by adaptively combining the neighboring chrominance values. Since the
R (p)) (w(p) × L , p∈Ω w(p)
p∈Ω
(10)
where Ω = {(x − 1, y), (x + 1, y), (x, y − 1), (x, y + 1)}. The luminance at the other red pixels can be similarly obtained. At the end of this step, a full-resolution luminance plane can be obtained. With the full-resolution luminance available, we can update the red and blue estimates to gain higher accuracy. Specifically, we calculate the LR (x, y) and LR (x − 2, y) by using the current luminance estimates, and then update the red value at pixel (x − 1, y) as R (x − 2, y)). (11) R (x, y) + L − 1, y) = L(x − 1, y) + 1 (L R(x 2 Using the more accurate red and blue estimates obtained, we can update the luminance estimation using (10). Our experiments show that the proposed method can obtain more accurate luminance at red and blues pixels as compared to Alleysson’s method [3]. 3.3. Chrominance Estimation Finally, we obtain the sub-sampled chrominance planes LR , LG , and LB by subtracting the CFA image from the full-resolution, estimated luminance plane. Because the chrominance planes contain little spatial details, bilinear interpolation is used to populate the chrominance planes. We found that using a more sophisticated interpolation scheme in this step could only attain very marginal performance gain. It can be shown that the overall computational complexity of our proposed method is close to that of the Alleysson’s method [3].
4. EXPERIMENTAL RESULTS To evaluate the performance of our proposed method, we compare its demosaicking results with those obtained by four recent state-of-theart methods which can obtain superior results, including Gunturk’s alternating projections method (AP) [4], Wu’s primary-consistent soft-decision (PCSD) method [5], Li’s successive approximation (SA) demosaicking method [6], and Alleysson’s frequency selection (FS) demosaicking method [3]. Fig. 8 shows our test image set; the images in the test set have been widely used for CFA demosaicking researches and experiments in the literature [3–6].
Table 2. Performance Comparison—PSNR results (in dB) of the red, green and blue color planes are listed in the 1st, 2nd and 3rd rows of each image. The best PSNR result of each row is underlined. Image 1 2 3 4 5 6 7 8 9 10 11 12
Fig. 8. Test images (referred as Image 1 to Image 24, enumerated from left-to-right and top-to-bottom.) In what follows, we evaluate the performance of the proposed method from two aspects: subjective visual evaluation and objective peak signal-to-noise ratio (PSNR) comparison.
(a) Original
(b) AP
(c) PCSD
(d) SA
AP
PCSD
SA
FS
Proposed
36.59 40.49 36.86 35.87 40.81 38.60 40.43 43.48 40.39 36.68 43.33 41.42 36.82 39.81 36.55 37.72 41.52 37.27 40.84 43.62 39.91 34.06 38.59 34.29 40.54 43.26 41.43 40.55 43.89 40.68 37.86 41.64 39.06 41.12 44.90 41.40
32.89 38.74 33.04 36.70 42.48 38.82 40.75 44.46 40.42 36.49 42.89 39.68 34.69 39.69 34.35 36.11 41.36 35.54 39.96 43.63 39.47 31.25 37.38 31.25 39.43 43.73 40.14 39.01 43.97 39.26 35.61 40.84 36.05 40.13 45.18 40.54
36.66 40.88 37.64 35.54 40.06 39.07 39.08 42.21 39.36 36.17 42.64 40.88 35.59 38.27 35.34 37.22 41.40 36.67 40.12 42.67 39.35 34.44 38.74 35.00 40.60 43.33 41.36 40.12 43.86 40.56 37.60 41.70 38.69 41.07 44.71 41.39
35.06 39.64 35.45 34.04 39.70 36.57 38.68 42.20 38.49 35.21 42.99 40.76 34.97 38.47 34.80 36.25 40.81 35.67 38.89 42.44 38.08 31.47 36.63 31.64 38.68 42.34 39.74 39.28 43.55 40.14 36.45 41.05 37.51 39.05 43.62 39.51
36.68 39.92 36.90 37.71 43.71 40.10 42.19 45.63 41.41 37.16 44.67 42.63 37.51 40.84 36.86 37.70 40.74 36.85 42.27 45.61 40.93 34.54 38.21 34.53 41.37 45.17 43.07 41.17 45.54 42.06 38.28 41.81 38.98 41.75 45.77 41.86
Image 13 14 15 16 17 18 19 20 21 22 23 24 Average
AP
PCSD
SA
FS
Proposed
34.05 36.91 33.16 34.79 38.37 35.62 36.37 41.49 39.31 40.80 44.88 40.62 40.78 42.92 40.22 35.52 38.64 36.36 37.90 41.85 39.23 39.25 42.72 38.01 38.26 41.63 37.55 37.09 39.58 36.81 37.61 41.64 40.42 34.87 37.33 32.70 39.41 39.09 39.43
29.96 34.83 29.54 34.06 38.21 34.56 36.04 41.71 38.46 40.23 44.99 39.87 38.39 42.17 38.05 33.46 37.76 33.46 36.39 41.26 36.80 38.04 42.43 36.95 34.85 39.93 35.12 35.26 39.46 35.12 40.74 44.64 41.33 33.12 37.14 31.55 38.25 38.13 38.37
34.70 37.90 33.35 33.33 36.74 34.39 35.34 40.58 38.83 38.66 43.40 39.40 40.71 43.13 39.93 35.51 38.61 36.43 38.55 42.33 39.63 38.75 41.74 37.22 37.83 41.61 36.98 36.93 39.63 36.87 38.06 41.83 40.54 34.79 37.59 32.80 39.07 38.77 39.09
34.57 37.72 33.30 32.44 36.88 33.17 34.85 40.55 37.45 39.16 43.99 38.85 40.06 43.06 40.03 34.90 38.37 35.55 35.75 40.46 36.73 38.59 41.97 36.60 36.93 40.98 36.22 36.40 39.28 35.51 36.97 41.27 37.65 35.29 37.83 32.74 38.15 37.89 38.22
33.68 35.98 32.74 36.10 40.52 36.22 37.19 43.13 40.24 40.72 43.83 40.22 40.96 43.45 40.69 35.57 38.79 36.35 38.68 42.30 39.99 40.14 43.59 38.39 38.34 41.18 37.38 37.81 41.20 37.42 38.94 44.93 40.75 35.04 37.51 32.83 40.07 39.77 40.10
ing demosaicked results obtained by our proposed method and other methods under comparison. Observably, our proposed method produces more visually pleasing results with fewer zipper effect artifacts. The PSNR results of the methods under comparison are listed in Table 2. The table shows that our proposed method outperforms others in terms of PSNR for most test images. 5. CONCLUSION We have proposed a new demosaicking method based on estimating more accurate luminance from CFA samples. By analyzing the spectrum of a CFA image, we identified that the luminance and chrominance components at the green pixels are better separated in the frequency domain than those at red and blue pixels. Driven by this property, our algorithm employs an efficient filter to estimate more accurately the luminance at green pixels and then uses an edge-adaptive scheme to reconstruct more effectively the luminance at the red/blue samples by exploiting both the image spectral and spatial correlations. Compared with the four recent state-of-the-art methods, not only is our proposed method less computational complex, it also obtains more visually pleasing results with higher PSNR for most of the test images. 6. REFERENCES
(e) FS
(f) Proposed
Fig. 9. The original and demosaicked results of a cropped region from Image 7. Fig. 9 shows a cropped region of a test image and its correspond-
[1] B. Bayer, “Color imaging array,” United States Patent, no. 3,971,065, 1976. [2] R.Hunt, “Why is black-and-white so important in color?,” 4th IS & T/SID Color Image Conference Scottsdale, Arizona, pp. 54–57, 1995. [3] D. Alleysson, S. Susstrunk, and J. Herault, “Linear demosaicking inspired by human visual system,” IEEE Transactions on Image Processing, vol. 14, no. 4, pp. 439–449, 2005. [4] B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, “Color plane interpolation using alternating projections,” IEEE Transcations on Image Processing, vol. 11, no. 9, 2002. [5] X. Wu and N. Zhang, “Primary-consistent soft-decision color demosaicking for digital cameras,” IEEE Trans. on Image Processing, vol. 13, pp. 1263–1274, 2004. [6] X. Li, “Demosaicing by successive approximation,” IEEE Transcations on Image Processing, vol. 14, no. 3, pp. 370–379, 2005.
