Correction of Clipped Pixels in Color Images
Correction of Clipped Pixels in Color Images IEEE Transaction on Visualization and Computer Graphics, Vol. 17, No. 3, 2011 Di Xu, Colin Doutre, and Panos Nasiopoulos Presented by In-Yong Song
School of Electrical Engineering and Computer Science Kyungpook National Univ.
Abstract Proposed
method
– Enhancement of luma and chroma of clipped pixels – Based on the strong chroma spatial correlation • Between clipped pixels and surrounding unclipped area
– Procedure • Identifying the clipped areas • Partition of the clipped areas • Estimating chroma • Correcting Clipped R, G, or B color channels • Smoothing of the boundaries
2 / 27
Introduction Representation
of high dynamic range (HDR)
– Limited dynamic range of conventional display • How to properly display HDR content on a conventional low dynamic range (LDR) media
– Using tone mapping • Reducing strong contrast from the HDR scene • Preserving the image details and color apperance – Appreciation of the original scene content
– Lost of Information • Very bright or dark parts of a scene – Clipping upper or lower displayable limits
3 / 27
Proposed
method
– Color distortion due to the clipping • For a clipped pixel, often not clipping all three red, green, and blue channels, nor occurring the same amount of clipping in each channel
Fig. 1. Example of color distortion due to the clipping. (a) A correct exposure image, (c) the corresponding overexposed image, and (e) the enhanced image with corrected luma for clipped pixels. (b), (d), and (f) The plots of the RGB intensities versus the pixel index along the highlighted horizontal lines of images (a), (c), and (e), respectively.
4 / 27
– Proposing effective clipped-pixel-enhancing algorithm • Automatically restoring both the luma and chroma of the clipped pixels – Exploiting strong correlation in chroma » Between saturated pixels and surrounding unsaturated pixels
• Experiment of proposed method – Objective and subjective quality evaluations
5 / 27
Our Proposed Method Aim
of proposed method
– Restoring lost information in overexposed color images • Based on the strong spatial correlation in the chroma channels – Using YCbCr color space
Fig. 2. Normalized autocorrelation of R, G, B, Y, Cb, and Cr signals (average over 24 true-color Kodak images). 6 / 27
– Flowchart
Fig. 3. Flowchart of our proposed method.
7 / 27
Identify
clipped areas
– Simply selecting pixels from all three color channels • Often generating very small isolated clipped areas and large clipped areas with small holes – Due to image noise light from the scene, camera response, sensor noise, and color filter array interpolation – Slightly lower than the maximum value
– Proposed method • Apply of bilateral filter – Removing noise
• Applying threshold τ to each color channel of pixel – Experimentally selecting τ to 252.5 for 8-bit per channel images 8 / 27
Fig. 4. Example of clipped areas. (a) Clipped image with a rectangular region of interest,(b) clipped areas in the region of interest identified with a simple threshold, and (c) clipped areas in the region of interest identified with our proposed method.
9 / 27
Partition
clipped areas
– Partitioning group the clipped pixels into regions with similar chroma • Partitioning clipped areas into spatially disconnected regions – Belonging different surfaces and different chroma – Containing multiple clipped objects (different colors)
• Segmenting each region according to chroma – Eliminating illumination difference on the same color surface – Considering only chroma quantities (Cb and Cr)
10 / 27
• 2 or 3-channel saturated pixels (pixels with heavier saturation) – Considered as one subregion
• 1-channel saturated pixels – Further segmented by using histogram based multithreshold algorithm
Fig. 5. Example of clipped area partition. (a) The clipped areas before partition (white pixels) and (b) the clipped areas partitioned into regions with similar chroma. 11 / 27
Correct
chroma
– Finding an unclipped region with similar chroma • Selecting neighboring pixels with similar color to the clipped region as seed points
– Apply of region growing algorithm from each seed point • Union of all surrounding areas obtained from different seed points as the surrounding region for a clipped region
Fig. 6. Example of a surrounding region. (a) Clipped image, (b) clipped areas (in color) superimposed on the image luma, and (c) the surrounding region (white pixels) for the clipped ara on the girl’s arm.
12 / 27
– Problem of missing pixels • Not used in the interpolation – Clipped pixels or nonclipped pixels with different chroma to the current clipped region
• Not applying traditional convolution
– Solve of problem • Use of normalized convolution – Adjusting the filter weights – Certainty component m
[ c ( x, y ) ⋅ m ( x, y ) ] ∗ h ( x, y ) c ( x, y ) = m ( x, y ) ∗ h ( x, y ) 1, for unsaturated pixels in the surrouding region m ( x, y ) = and saturated pixels that have been corrected 0, otherwise 13 / 27
• Ω : the saturated pixel set = Ω
{( x, y ) : R( x, y) ≥ τ , or G( x, y) ≥ τ , or B( x, y) ≥ τ }
• Ω1 , Ω 2 , and Ω3 : set of clipped pixels with 1, 2, and 3 saturated channel
– Closing clipped pixel to the surrounding region • Strong correlation with the surrounding unclipped pixels
Fig. 7. Example of chroma correction. The Cr channel of (a) unclipped image, (b) clipped image, and (c) corrected image using our proposed algorithm. The difference between (a) and (b) is shown in (d), and the difference between (a) and (c) is shown in (e).
14 / 27
Correct
RGB values
– Correct 2-channel saturated pixels • Use of conversion from RGB to YCbCr – RGB and YCbCr values » Range of 0.0 ~ 1.0 0.5041 0.0979 R 0.0627 Y 0.2568 Cb = −0.1482 −0.2910 0.4392 × G + 0.5020 Cr 0.4392 −0.3678 −0.0714 B 0.5020 Cb = [ −0.1482 −0.2910 0.4392] × [ R G B ] + 0.5020 T
Cr=
[0.4392
−0.3678 −0.0714] × [ R G B ] + 0.5020 T
(2)
(3) (4)
– U : known (unsaturated channel) – S1 and S 2 : two clipped channel ( saturated channel)
15 / 27
• Solving for using the two equation (3) and (4) S1 = f1 (U , Cb, Cr ) S2 = f 2 (U , Cb, Cr )
where f1 and f 2 are functions of U , Cb , and Cr , and S denotes the corrected value of color channel S
• Example of correct 2-channel saturated pixels – R and G : two clipped unknown channels – B : unclipped channel ( Cb − 0.5020 − 0.4392 B ) × ( −0.3678 ) = R − ( Cr − 0.5020 + 0.0714 B ) × ( −0.2910 ) / ( −0.1482 ) × ( −0.3678 ) − 0.4392 × ( −0.2910 ) =( Cb − 0.5020 − 0.4392 B ) × 0.4392 G − ( Cr − 0.5020 + 0.0714 B ) × ( −0.1482 ) / ( −0.2910 ) × 0.4392 − ( −0.3678 ) × ( −0.1482 ) 16 / 27
– Correct 1-channel saturated pixels • Similar to correcting 2-channel saturated pixels – S : clipped channel (saturated channel) – U1 and U 2 : two known channel ( unsaturated channel) S + S2 S = 1 , where 2 S1 = f3 (U1 ,U 2 , Cb) S2 = f 4 (U1 ,U 2 , Cr )
• Example of correct 2-channel saturated pixels – R : clipped unknown channels – G and B : two unclipped channel = R1 + R2 , where R 2 Cb − (−0.2910) × G − 0.4392 × B − 0.5020 R1 = −0.1482 Cr − (−0.3678) × G − (−0.0714) × B − 0.5020 S2 = 0.4392 17 / 27
– Correct 3-channel saturated pixels • Needed to solve three equations • First, estimating luma Y value of the 3-channel saturated pixels based on the surrounding region – 2D Gaussian function – Not assuming center of the Gaussian function as the centroid of the clipped region = g ( x, y ) Ae
− a ( x − x0 )2 + 2 b ( x − x0 )( y − y0 )+c ( y − y0 )2
+B
a b where A, B, a, b, c, x0 ,and y0 are the parameters, and is b c positive definite
– Solving least-squares surface-fitting problem » Use of optimization form n
arg min
2
∑ [Yi − g ( xi , yi )] ,
A, B ,a ,b ,c , x0 , y0 i =1
a b Subject to : 0 b c
where ( xi , yi , Yi ) is the i th the surrounding area, xi and yi represent the pixel location, Yi is the luma at pixel ( xi , yi ) , and the symbol " " stands for positive definite. 18 / 27
– Decomposing symmetric and positive definite matrix M M = LLT where L is a lower triangular matrix
» For the matrix a b b c
» In the constraint, we have a b l11 0 l11 l21 = × 0 l b c l l 21 22 22
– Substituting a, b, and c using the new variables l11 , l21 , and l22 2
n
arg min
∑ [Y − g ( x , y )]
A, B ,l11 ,l21 ,l22 , x0 , y0 i =1
= where g ( x, y ) Ae
i
i
i
l2 l 2 +l 2 − 11 ( x − x0 )2 +l11l21 ( x − x0 )( y − y0 )+ 21 22 ( y − y0 )2 2 2
+B
19 / 27
• Solving optimization – Standard least-squares fitting algorithm
• In the end, Solving corrected RGB values by using (2) R = G B
1.16438356 0.00000030 1.59602688 Y 0.0627 1.16438356 −0.39176253 −0.81296829 × Cb − 0.5020 1.16438356 2.01723263 0.00000305 Cr 0.5020
• Lower bound α d (α 0 − 1) × + 1, when 0 < d < d 0 d0 α = α , when d ≥ d 0 0 where the enhancement factor α 0 is a constant and α 0 > 1, the constant d 0 is the transition width, and d denotes the distance between the current saturated pixel and the closest unsaturated pixel in the surrounding region. 20 / 27
Smooth
enhancement
– Small jumps of enhance values between adjacent 1, 2, and 3-channel saturated regions • Use of different strategies – Calculating saturated RGB channel
– Reduction of disturbing contours – Obtainment of natural looking enhanced images
Fig. 8. Illustration of the smoothing process between 1, 2, and 3channel saturated regions
21 / 27
– Adjusted saturated channel value ω ω 1 (x= + 1 − × × ∑ P ( x, y ) P P( x0 , y0 ) × 0 , y0 ) ω0 ω0 N ( x , y )∈Ax0 , y0
(5)
where ω0 is the width of the transition band, ω is the distance between ( x0 , y0 ) and ( x1 , y1 ) , and N is the number of pixels in the area Ax0 , y0
• Choosing ω0 to adjust the amount of the smoothing – Reasonable range of ω0 » 3 ~ 10 pixels
Fig. 9. Example of the smoothing effect. (a) The saturation category map, (b) enhanced image before smoothing, and (c) enhanced image after smoothing.
22 / 27
Experimental Results Experiment
of proposed method
– Objective and subjective evaluation
Fig. 10. Thumbnails of our test images. In reading order: girl, landscape, baby_girl, mountain, shoes, sunset, kodim03(caps), kodim05(motor-cycles), kodim06(boat), kodim12(beach), kodim16(lake), kodim21(light-house), and kodim23(parrots). 23 / 27
– Objective evaluation Table 1. Quality comparison between different algorithms
24 / 27
– Result image • (1)
Fig. 11. Results of clipped pixel enhancement. For each row, we show (from left to right) the original image, clipped image, clipped areas superimposed on the image luma, enhanced image using Bayesian algorithm, and enhanced image using our proposed algorithm.
25 / 27
• (2)
Fig. 12. Two virtual exposures of HDR images, The four column of images correspond to (from left to right) the original unclipped LDR image, the clipped LDR image, the enhanced image obtained by the Bayesian algorithm, and the enhanced image produced by our 26 / 27 proposed algorithm, respectively.
Conclusion Proposed
method
– Effective method for enhancing clipped pixels in color image • Use of the strong correlation between the chroma of the clipped pixels and surrounding unclipped pixels
– Greatly reducing color distortion caused by clipping – Objective and subjective evaluation • Visually more plausible than clipped images • More enhanced images produced by previous algorithm
27 / 27
School of Electrical Engineering and Computer Science Kyungpook National Univ.
Abstract Proposed
method
– Enhancement of luma and chroma of clipped pixels – Based on the strong chroma spatial correlation • Between clipped pixels and surrounding unclipped area
– Procedure • Identifying the clipped areas • Partition of the clipped areas • Estimating chroma • Correcting Clipped R, G, or B color channels • Smoothing of the boundaries
2 / 27
Introduction Representation
of high dynamic range (HDR)
– Limited dynamic range of conventional display • How to properly display HDR content on a conventional low dynamic range (LDR) media
– Using tone mapping • Reducing strong contrast from the HDR scene • Preserving the image details and color apperance – Appreciation of the original scene content
– Lost of Information • Very bright or dark parts of a scene – Clipping upper or lower displayable limits
3 / 27
Proposed
method
– Color distortion due to the clipping • For a clipped pixel, often not clipping all three red, green, and blue channels, nor occurring the same amount of clipping in each channel
Fig. 1. Example of color distortion due to the clipping. (a) A correct exposure image, (c) the corresponding overexposed image, and (e) the enhanced image with corrected luma for clipped pixels. (b), (d), and (f) The plots of the RGB intensities versus the pixel index along the highlighted horizontal lines of images (a), (c), and (e), respectively.
4 / 27
– Proposing effective clipped-pixel-enhancing algorithm • Automatically restoring both the luma and chroma of the clipped pixels – Exploiting strong correlation in chroma » Between saturated pixels and surrounding unsaturated pixels
• Experiment of proposed method – Objective and subjective quality evaluations
5 / 27
Our Proposed Method Aim
of proposed method
– Restoring lost information in overexposed color images • Based on the strong spatial correlation in the chroma channels – Using YCbCr color space
Fig. 2. Normalized autocorrelation of R, G, B, Y, Cb, and Cr signals (average over 24 true-color Kodak images). 6 / 27
– Flowchart
Fig. 3. Flowchart of our proposed method.
7 / 27
Identify
clipped areas
– Simply selecting pixels from all three color channels • Often generating very small isolated clipped areas and large clipped areas with small holes – Due to image noise light from the scene, camera response, sensor noise, and color filter array interpolation – Slightly lower than the maximum value
– Proposed method • Apply of bilateral filter – Removing noise
• Applying threshold τ to each color channel of pixel – Experimentally selecting τ to 252.5 for 8-bit per channel images 8 / 27
Fig. 4. Example of clipped areas. (a) Clipped image with a rectangular region of interest,(b) clipped areas in the region of interest identified with a simple threshold, and (c) clipped areas in the region of interest identified with our proposed method.
9 / 27
Partition
clipped areas
– Partitioning group the clipped pixels into regions with similar chroma • Partitioning clipped areas into spatially disconnected regions – Belonging different surfaces and different chroma – Containing multiple clipped objects (different colors)
• Segmenting each region according to chroma – Eliminating illumination difference on the same color surface – Considering only chroma quantities (Cb and Cr)
10 / 27
• 2 or 3-channel saturated pixels (pixels with heavier saturation) – Considered as one subregion
• 1-channel saturated pixels – Further segmented by using histogram based multithreshold algorithm
Fig. 5. Example of clipped area partition. (a) The clipped areas before partition (white pixels) and (b) the clipped areas partitioned into regions with similar chroma. 11 / 27
Correct
chroma
– Finding an unclipped region with similar chroma • Selecting neighboring pixels with similar color to the clipped region as seed points
– Apply of region growing algorithm from each seed point • Union of all surrounding areas obtained from different seed points as the surrounding region for a clipped region
Fig. 6. Example of a surrounding region. (a) Clipped image, (b) clipped areas (in color) superimposed on the image luma, and (c) the surrounding region (white pixels) for the clipped ara on the girl’s arm.
12 / 27
– Problem of missing pixels • Not used in the interpolation – Clipped pixels or nonclipped pixels with different chroma to the current clipped region
• Not applying traditional convolution
– Solve of problem • Use of normalized convolution – Adjusting the filter weights – Certainty component m
[ c ( x, y ) ⋅ m ( x, y ) ] ∗ h ( x, y ) c ( x, y ) = m ( x, y ) ∗ h ( x, y ) 1, for unsaturated pixels in the surrouding region m ( x, y ) = and saturated pixels that have been corrected 0, otherwise 13 / 27
• Ω : the saturated pixel set = Ω
{( x, y ) : R( x, y) ≥ τ , or G( x, y) ≥ τ , or B( x, y) ≥ τ }
• Ω1 , Ω 2 , and Ω3 : set of clipped pixels with 1, 2, and 3 saturated channel
– Closing clipped pixel to the surrounding region • Strong correlation with the surrounding unclipped pixels
Fig. 7. Example of chroma correction. The Cr channel of (a) unclipped image, (b) clipped image, and (c) corrected image using our proposed algorithm. The difference between (a) and (b) is shown in (d), and the difference between (a) and (c) is shown in (e).
14 / 27
Correct
RGB values
– Correct 2-channel saturated pixels • Use of conversion from RGB to YCbCr – RGB and YCbCr values » Range of 0.0 ~ 1.0 0.5041 0.0979 R 0.0627 Y 0.2568 Cb = −0.1482 −0.2910 0.4392 × G + 0.5020 Cr 0.4392 −0.3678 −0.0714 B 0.5020 Cb = [ −0.1482 −0.2910 0.4392] × [ R G B ] + 0.5020 T
Cr=
[0.4392
−0.3678 −0.0714] × [ R G B ] + 0.5020 T
(2)
(3) (4)
– U : known (unsaturated channel) – S1 and S 2 : two clipped channel ( saturated channel)
15 / 27
• Solving for using the two equation (3) and (4) S1 = f1 (U , Cb, Cr ) S2 = f 2 (U , Cb, Cr )
where f1 and f 2 are functions of U , Cb , and Cr , and S denotes the corrected value of color channel S
• Example of correct 2-channel saturated pixels – R and G : two clipped unknown channels – B : unclipped channel ( Cb − 0.5020 − 0.4392 B ) × ( −0.3678 ) = R − ( Cr − 0.5020 + 0.0714 B ) × ( −0.2910 ) / ( −0.1482 ) × ( −0.3678 ) − 0.4392 × ( −0.2910 ) =( Cb − 0.5020 − 0.4392 B ) × 0.4392 G − ( Cr − 0.5020 + 0.0714 B ) × ( −0.1482 ) / ( −0.2910 ) × 0.4392 − ( −0.3678 ) × ( −0.1482 ) 16 / 27
– Correct 1-channel saturated pixels • Similar to correcting 2-channel saturated pixels – S : clipped channel (saturated channel) – U1 and U 2 : two known channel ( unsaturated channel) S + S2 S = 1 , where 2 S1 = f3 (U1 ,U 2 , Cb) S2 = f 4 (U1 ,U 2 , Cr )
• Example of correct 2-channel saturated pixels – R : clipped unknown channels – G and B : two unclipped channel = R1 + R2 , where R 2 Cb − (−0.2910) × G − 0.4392 × B − 0.5020 R1 = −0.1482 Cr − (−0.3678) × G − (−0.0714) × B − 0.5020 S2 = 0.4392 17 / 27
– Correct 3-channel saturated pixels • Needed to solve three equations • First, estimating luma Y value of the 3-channel saturated pixels based on the surrounding region – 2D Gaussian function – Not assuming center of the Gaussian function as the centroid of the clipped region = g ( x, y ) Ae
− a ( x − x0 )2 + 2 b ( x − x0 )( y − y0 )+c ( y − y0 )2
+B
a b where A, B, a, b, c, x0 ,and y0 are the parameters, and is b c positive definite
– Solving least-squares surface-fitting problem » Use of optimization form n
arg min
2
∑ [Yi − g ( xi , yi )] ,
A, B ,a ,b ,c , x0 , y0 i =1
a b Subject to : 0 b c
where ( xi , yi , Yi ) is the i th the surrounding area, xi and yi represent the pixel location, Yi is the luma at pixel ( xi , yi ) , and the symbol " " stands for positive definite. 18 / 27
– Decomposing symmetric and positive definite matrix M M = LLT where L is a lower triangular matrix
» For the matrix a b b c
» In the constraint, we have a b l11 0 l11 l21 = × 0 l b c l l 21 22 22
– Substituting a, b, and c using the new variables l11 , l21 , and l22 2
n
arg min
∑ [Y − g ( x , y )]
A, B ,l11 ,l21 ,l22 , x0 , y0 i =1
= where g ( x, y ) Ae
i
i
i
l2 l 2 +l 2 − 11 ( x − x0 )2 +l11l21 ( x − x0 )( y − y0 )+ 21 22 ( y − y0 )2 2 2
+B
19 / 27
• Solving optimization – Standard least-squares fitting algorithm
• In the end, Solving corrected RGB values by using (2) R = G B
1.16438356 0.00000030 1.59602688 Y 0.0627 1.16438356 −0.39176253 −0.81296829 × Cb − 0.5020 1.16438356 2.01723263 0.00000305 Cr 0.5020
• Lower bound α d (α 0 − 1) × + 1, when 0 < d < d 0 d0 α = α , when d ≥ d 0 0 where the enhancement factor α 0 is a constant and α 0 > 1, the constant d 0 is the transition width, and d denotes the distance between the current saturated pixel and the closest unsaturated pixel in the surrounding region. 20 / 27
Smooth
enhancement
– Small jumps of enhance values between adjacent 1, 2, and 3-channel saturated regions • Use of different strategies – Calculating saturated RGB channel
– Reduction of disturbing contours – Obtainment of natural looking enhanced images
Fig. 8. Illustration of the smoothing process between 1, 2, and 3channel saturated regions
21 / 27
– Adjusted saturated channel value ω ω 1 (x= + 1 − × × ∑ P ( x, y ) P P( x0 , y0 ) × 0 , y0 ) ω0 ω0 N ( x , y )∈Ax0 , y0
(5)
where ω0 is the width of the transition band, ω is the distance between ( x0 , y0 ) and ( x1 , y1 ) , and N is the number of pixels in the area Ax0 , y0
• Choosing ω0 to adjust the amount of the smoothing – Reasonable range of ω0 » 3 ~ 10 pixels
Fig. 9. Example of the smoothing effect. (a) The saturation category map, (b) enhanced image before smoothing, and (c) enhanced image after smoothing.
22 / 27
Experimental Results Experiment
of proposed method
– Objective and subjective evaluation
Fig. 10. Thumbnails of our test images. In reading order: girl, landscape, baby_girl, mountain, shoes, sunset, kodim03(caps), kodim05(motor-cycles), kodim06(boat), kodim12(beach), kodim16(lake), kodim21(light-house), and kodim23(parrots). 23 / 27
– Objective evaluation Table 1. Quality comparison between different algorithms
24 / 27
– Result image • (1)
Fig. 11. Results of clipped pixel enhancement. For each row, we show (from left to right) the original image, clipped image, clipped areas superimposed on the image luma, enhanced image using Bayesian algorithm, and enhanced image using our proposed algorithm.
25 / 27
• (2)
Fig. 12. Two virtual exposures of HDR images, The four column of images correspond to (from left to right) the original unclipped LDR image, the clipped LDR image, the enhanced image obtained by the Bayesian algorithm, and the enhanced image produced by our 26 / 27 proposed algorithm, respectively.
Conclusion Proposed
method
– Effective method for enhancing clipped pixels in color image • Use of the strong correlation between the chroma of the clipped pixels and surrounding unclipped pixels
– Greatly reducing color distortion caused by clipping – Objective and subjective evaluation • Visually more plausible than clipped images • More enhanced images produced by previous algorithm
27 / 27
