Color Image Sharpening Based on Nonlinear ... - Semantic Scholar
COLOR IMAGE SHARPENING BASED ON NONLINEAR REACTION-DIFFUSION Takahiro Saito, Hiroyuki Harada, Jun Satsumabayashi, Takashi Komatsu Dept. of Electrical, Electronics and Information Engineering, Kanagawa University e-mail: {saitot01, R200370108, satsuj01, komatt01}@kanagawa-u.ac.jp
ABSTRACT Previously we have presented a selective sharpening method for monochrome images. Our method is based on the simultaneous nonlinear reaction-diffusion time-evolution equipped with a nonlinear diffusion term, a reaction term and an overshooting term, and can sharpen degraded edges blurred by several causes without increasing the visibility of random noise. This paper extends our method to selective sharpening of color images. As to the how to extend it, we take into accounts two variations about the treatment of three color components and the selection of the color space. By experiments, we quantitatively evaluate performance of these variations. The best performance is achieved by the collective treatment of color components based on the simultaneous full-nonlinear reaction-diffusion time-evolution, and blurred color edges are sharpened selectively much better than by the existing methods such as the adaptive peaking method.
consider other color spaces. Taking these possibilities into accounts, we study some variations about the treatment of color components and the selection of the color space. By experiments, we quantitatively compare performance of these variations. 2. SHARPENING OF MONOCHROME IMAGE [5] 2.1. Simultaneous time-evolution equations Previously we have derived the simultaneous nonlinear reactiondiffusion time-evolution equations: ∂f σ2 = div g ( ∇f ) ⋅∇f − s ⋅ (u x + v y ) − ⋅( f − I ) ∂τ 2λ ∂u 1 = ⋅ div g ( ∇u ) ⋅∇u − (u − f x ) ∂τ λ ∂v 1 = ⋅ div g ( ∇v ) ⋅∇v − (v − f y ) ∂τ λ , I : input image , f : time-evolution image ,
(1)
where the two auxiliary functions u, v approximate the first spatial derivatives of the function f, as follows: 1. INTRODUCTION As an image sharpening method, the peaking technique [1],[2] have been popular and used for practical applications. Recently another type of sharpening method using nonlinear filters such as a Volterra filter has been proposed [3]. However, these existing methods have their own limitations; they cannot work well for blurred image corrupted by random noise, and will produce the side effect that the noise visibility are augmented to some extent. Previously, extending a prototype of the simultaneous nonlinear reaction-diffusion time-evolution, proposed by M. Proesmans et al. [4], to selective monochrome-image sharpening, we have formed a new selective sharpening method using the simultaneous nonlinear reaction-diffusion time-evolution , and shown that our method has an ideal capability to sharpen blurred edges without increasing the visibility of random noise [5]. Its discrete expression is defined as the iterative nonlinear operations, and by experiments we have demonstrated that the discrete expression has a desirable property that computing a certain decision criterion halts its iteration when it achieves the best selective sharpness enhancement [5]. This paper extends our method to selective sharpening of color images. As to the extension, there are some options. One possible extension is to treat three color components separately, whereas another possible extension is to treat three color components collectively, where, instead of the magnitude of the intensity gradient [6], we introduce a certain measure quantifying the magnitude of a color edge to control the time-evolution [7]. Moreover, in addition to the primary RGB color space, we
0-7803-7750-8/03/$17.00 ©2003 IEEE.
u=fx +δ , v= fy +δ′ ,
(2)
and the function g means the nonlinear diffusivity function and in this paper is given by
{
g ( z) = 1 1+ ( z K )
2
}
.
(3)
2.2. Discrete form of the time-evolution equations The discrete expression for equation 1 is defined by
{(
fiτ, j+1 = fiτ, j + ε ⋅ ∑ g ∇d fiτ, j ⋅∇d fiτ, j d =N ,S ,E ,W − ,
∑
s 2
d = N , S , E ,W
⋅
{(u
τ +1 i +1, j
)
} − σ2λ ⋅ ( f
− uiτ−+1,1 j ) + ( viτ,+j1+1 − viτ,+j1−1 )
{g ( ∇ f ) ⋅∇ f } τ d i, j
(
}
)
2
τ i, j
− I i , j )
τ d i, j
(
)
= g ∇ N fiτ, j ⋅∇ N fiτ, j + g ∇S fiτ, j ⋅∇S fiτ, j
(
τ E i, j
+g ∇ f
) ⋅∇
τ E i, j
(
τ W i, j
f +g ∇ f
) ⋅∇
τ W i, j
f
, ∇N fiτ, j = fiτ, j −1 − fiτ, j , ∇S fiτ, j = fiτ, j +1 − fiτ, j ∇E fiτ, j = fiτ+1, j − fiτ, j , ∇W fiτ, j = fiτ−1, j − fiτ, j ε
uiτ,+j1 = uiτ, j + ⋅ λ
∑
d = N , S , E ,W
{
{g ( ∇ u ) ⋅∇ u } τ d i, j
τ d i, j
(4)
}
− ε ⋅ uiτ, j − 12 ⋅ ( fiτ+1, j − fiτ−1, j ) ε
viτ,+j1 = viτ, j + ⋅ λ
∑
d = N , S , E ,W
{
{g ( ∇ v ) ⋅∇ v } τ d i, j
τ d i, j
}
− ε ⋅ viτ, j − 12 ⋅ ( fiτ, j +1 − fiτ, j −1 ) ,
τ i, j
f ; intensity of the (i, j ) pixel at the τ -th iteration Ii , j ; intensity of the (i, j ) pixel in the input image I .
ICIP 2003
The initial setting is given by f i ,0j = I i , j ; ui0, j = 12 ⋅ ( I i +1, j − I i −1, j ) ; vi0, j = 12 ⋅ ( I i , j +1 − I i , j −1 ) .
(5)
As a decision scheme to halt the iteration, we have proposed if δ τ −1,τ − δ τ ,τ +1 ≤ δT , then stop the iteration, , δ τ ,τ +1 = δ τf ,τ +1 δ wτ +1
δ τf ,τ +1 = ∑ f iτ, j+1 − f iτ, j
∑1
i, j
δ
τ +1 w
{
=∑ u i, j
τ +1 i, j
i, j
τ +1 i, j
+v
(6)
2 ⋅ ∑1 . i, j
}
Equation 4 has a desirable property that the above decision scheme halt the iteration almost at the ideal moment when it achieves the best selective sharpness enhancement. In most cases, equation 4 is stopped within 100 iterations. 3. EXTENSION TO SHARPENING OF COLOR IMAGES In this paper, we consider three different color spaces: the primary color space (R,G,B), the color-difference space (G,R-G,B-G) and the CIE LAB color space (L*,a*,b*). The CIE LAB color space is a kind of uniform lightness-chromaticity scale color space. In each case, a color vector is defined by C=
( R, G , B ) T
; primary color space,
(G , R − G , B − G ) T
; color-difference space,
( L*, a*, b*) T
; CIE LAB color space.
(7)
The time-evolving color vector and the input color vector are denoted by C = ( C(1) , C(2) , C(3) )T
; time-evolving color vector,
I = ( I (1) , I ( 2) , I (3) )T
; input color vector.
(8)
3.2. Collective full-nonlinear reaction-diffusion scheme (C-FN-Scheme) The C-FN-Scheme collectively treats all the color components; it uses the magnitude of a color edge to control the nonlinear diffusion term of the time-evolution. The magnitude of the color edge is defined from the Riemannian metric [7] defined by dx 2 dC = ( dx dy ) ⋅ [H] ⋅ ; Riemannian metric dy E F , [H] = F G 2
The color edge appears along the eigenvector corresponding to the maximum eigenvalue λ+ of the matrix H, and the magnitude of the color edge γ is defined by using the maximum eigenvalue λ+ and the minimum eigenvalue λ- , as follows: γ = λ+ − λ − =
(( E − G ) + 4 F )
∂C( n ) ∂τ ∂u( n ) ∂τ ∂v( n ) ∂τ
(
)
= div g ( ∇C( n ) ) ⋅ ∇C( n ) − s ⋅ (u( n ) )x + ( v( n ) )y − = =
1
λ 1
λ
⋅ div g ( ∇u ( n )
) ⋅ ∇u
⋅ div g ( ∇v( n )
) ⋅ ∇v
(n )
( n)
σ2 2λ
⋅ ( C( n ) − I ( n ) )
− (u( n ) − (C( n ) )x )
(
− v( n ) − (C( n ) ) y
)
(9)
, n = 1, 2, 3 , I ( n ) : input color component, C( n ) : time-evolution color component.
∂C( n )
; ξ τ +1 = max { ξ (τn+)1 n =1,2,3
1
4
∂τ
γ =
(( E − G ) + 4 F ) 2
)
3
2
1
4
2
3
3
n =1
n =1
, n = 1, 2, 3 .
(
E(τN+1) i , j = 1 + ∑ min mod u(τn+)1i , j −1 , u (τn+)1i , j n =1
3
)
2
(
F(τN+)1i , j = ∑ min mod uτ( n+)1i , j −1 , u τ( n+)1i , j ⋅ min mod vτ( n+)1i , j −1 , v(τn+)1i , j
n =1
(
, min mod [α , β ] =
δ (τn,τ) +1 = δ Cτ ,(τn+)1 δ wτ +( n1)
(13)
2
However, as the time-evolution for the two auxiliary functions u, v approximating the first spatial derivatives of each color component, the C-FN-Scheme employs the same equations as the ID-N-Scheme. Moreover, we employ the simultaneous stopping scheme. In the discrete expression of equation 13, the magnitude of the color edge γ is estimated at an intermediate point between samples. Thus we need specify the half-point values for the scalar values of E, F and G. For example, in the case of the half-point in the north, those values are estimated by taking the minmod [8] between the neighboring samples:
G(τN+1) i , j = 1 + ∑ min mod vτ( n+)1i , j −1 , vτ( n+)1i , j
ξ(τn+)1 = δ (τn−)1,τ − δ (τn,τ) +1
σ2 ⋅ (C ( n ) − I ( n ) ) 2λ
E = 1 + ∑ (u( n ) ) , F = ∑ (u( n ) ⋅ v( n ) ) , G = 1 + ∑ (v( n ) )
3
}
(12)
.
(
n =1
if ξ τ +1 ≤ δ T , then stop the iteration for all the color components,
2
= div g (γ ) ⋅∇C( n ) − s ⋅ (u( n ) )x + (v( n ) ) y −
3
As the decision scheme to halt its iteration, we employ the following simultaneous stopping scheme:
2
In the one dimensional case, namely n = 1, the definition of the magnitude of the color edge γ reduces to the magnitude of the intensity gradient. Unlike the ID-N-Scheme, as the time-evolution for each color component, the C-FN-Scheme employs the following fullnonlinear reaction-diffusion time-evolution:
n =1
3.1. Independent nonlinear reaction-diffusion scheme (IDN-Scheme) The ID-N-Scheme applies the time-evolution of equation 1 independently to each color component, as follows:
(11)
2
3 3 ∂C ∂C ∂C ∂C E = 1 + ∑ ( n) , F = ∑ ( n) ⋅ (n) , G = 1 + ∑ (n) . x x y ∂ ∂ ∂ n =1 n =1 n =1 ∂y 3
sign (α ) + sign ( β ) 2
)
)
2
⋅ min ( α , β
(14)
).
(10)
In equation 14, instead of the minmod function, as an alternative we may adopt the average function, but the use of the minmod function has the advantage that it sharpens color edges better than the use of the average function.
The simultaneous stopping scheme can halt the iteration almost at the ideal moment.
4. PERFORMANCE EVALUATIONS 4.1. Test color images and performance measures We evaluate performance of our schemes using artificially blurred
δ Cτ ,(τn+)1 = ∑ C(τn+)1i , j − C(τn) i , j
∑1
i, j
{
δ wτ +( n1) = ∑ uτ(n+)1i , j + vτ( n+)1i , j i, j
i, j
}
2 ⋅ ∑1 . i j ,
test color images. First we blur an original sharp color image h(x,y), given in RGB components, h = ( h(1) , h(2) , h(3) )T ,
(15)
with the Gaussian filter having the impulse response G(x,y ; ζ), ( x2 + y2 ) 1 − , G ( x, y;ς ) = ⋅ exp 2πς 2 2ς 2
(16)
and then add random Gaussian noise to the blurred color image G*h(x,y); thus we generate an artificially blurred test color image I(x,y), given in RGB components: I = ( I (1) , I ( 2) , I (3) )T .
(17)
Let C(x,y) denote the sharpness-enhanced color image reproduced from the blurred test color image I(x,y), and it is expressed in the primary RGB color space: C = ( C(1) , C(2) , C(3) )T .
(18)
We evaluate the performance of our schemes in the RGB color space even though the color vector is processed in the other color space, because existing display devices produce a RGB color image. For the performance evaluation, we define the quantitative measures in the RGB color space as follows. (1) SNR(n) of the sharpened color image: SNR is computed between the original sharp color image h and the sharpened color image C: SNR( n ) = 20 ⋅ log10 ( 255 D( n ) ) [ dB]
2 , D( n ) = E( x , y ) {C( n ) ( x, y ) − h( n) ( x, y )} .
(19)
The simultaneous stopping scheme of equation 10 can halt the iteration of our time-evolution schemes of equations 9 and 13 almost at the ideal moment when it attains its maximum SNR. (2) Blur-Removal Ratio Br(n) & Noise-Removal Ratio Nr(n): Let the vectors, b(n), n(n), s(n), be Gaussian-blurs artificially added to the original color image component h (n) , random Gaussian noise components added to the Gaussian blurred color image component G*h(n), and deviations of the artificially blurred test color image component I (n) from the sharpened color image component C(n), i.e. I(n) - C(n), respectively. We define those vectors by arranging their respective values at all the pixels in one column. Then, we define the blur-removal ratio Br(n) and the noise-removal ratio Nr(n) as follows: Br( n ) = ( b ( n ) , s( n )
)
b( n )
2
, Nr( n ) = ( n( n ) , s( n )
)
n( n )
2
. (20)
The positive value of Br (n) / Nr(n) means that the blur / noise removal is successfully achieved for the n-th color component. If blur / noise is perfectly removed for the n-th color component, then the value of Br(n) / Nr(n) will be 1; but the reverse is not necessarily true. On the contrary, the negative value means that the blur / noise is augmented far from being removed. (3) Artifact Component Ratio Ar (n): We define the artifactcomponent ratio Ar(n) as follows: Ar( n ) =
( Id − P
(n)
⋅ ( P(Tn ) ⋅ P( n )
, P( n ) = b ( n ) , n ( n )
)
−1
)
⋅ P(Tn ) ⋅ s ( n )
s(n)
, Id ; identity matrix .
(21)
This measure is given by the ratio of the norm of the projection of s (n) onto the orthogonal complement of the linear subspace spanned by b(n) and n(n), to the norm of s(n), and it quantifies how far the enhancement signal component -s (n), produced by the sharpening algorithm, contains undesirable artifact components
irrelevant to the compensation for the blur b(n) and the noise n(n). If no artifact occurs, then the value of the artifact-component ratio Ar(n) will be 0. 4.2. Evaluation results and further investigations In the production of artificially blurred test color images, we generate additive Gaussian noise with a zero mean and a standard deviation of 5.0. Figure 1(a) shows an artificially blurred test color image, where we set the blurring parameter ζ of the Gaussian filter defined in equation 16 to 1.0. As for the sharpness enhancement parameters of λ, σ and K, their proper values chiefly depend on the noise variance. We experimentally investigate their proper values. The variance dependence of the parameters of λ and σ is much weaker than that of the parameter K. If we set the parameter K close to the standard deviation of noise, the ID-N-Scheme and the C-FNScheme provide suboptimal selective sharpening performance. We set the parameters of ε, δT, λ, σ and K to 0.05, 10 - 4, 1.0, 1.0 and 5.0, respectively; and these suboptimal parameter values are found out by experiments. On the other hand, the shooting parameter s strongly depends on the blurring parameter ζ, and its proper value is different not only for every scheme but also for every color space. Hence, in this paper, for every case, we optimize the shooting parameter s. We apply the ID-N-Scheme and the C-FN-Scheme, to the selective sharpness enhancement of figure 1(a) in the three different color spaces: (R,G,B), (G,R-G,B-G) and (L*,a*,b*). For every scheme we optimize the shooting parameter s. Table 1 shows the values of SNR(n), Br(n), Nr(n) and Ar(n). As shown in table 1, the C-FN-Scheme outperforms the ID-N-Scheme, and the CFN-Scheme achieves almost equally excellent performance irrespective of the selection of the color space. Table 1 also shows the results of the quantitative performance evaluations for the adaptive peaking method [2], one of the typical existing schemes. In the case of the adaptive peaking method, the values of Br(n) are very close to zeros, and the values of Nr(n) are negative; these results means that the adaptive peaking method cannot attain selective sharpness enhancement at all. On the other hand, our two schemes successfully achieve selective sharpness enhancement. Figure 1(b) shows a sharpness-enhanced color image produced by the C-FN-Scheme in the primary color space (R,G,B). The C-FN-Scheme gives a sharpened image of subjectively superior quality. Next, setting the sharpness enhancement parameters λ, σ, K to 1.0, 1.0, 5.0, respectively, and changing the blurring parameter ζ of the Gaussian filter, we apply the C-FN-Scheme to the selective sharpness enhancement in the primary color space (R,G,B). In addition, for every blurred test color image, we optimize the shooting parameter s. Figure 2 shows the values of SNR(G), Br(G), Nr(G) and Ar(G) versus the blurring parameter ζ, for the green color component. Figure 2 also shows the SNR(TEST) of each blurred test image. If the blurring parameter ζ is less than 1.2, the C-FN-Scheme will achieve selective sharpening of blurred color edges; but if the blurring is more intensive, the C-FN-Scheme will not necessarily work well. 5. CONCLUSIONS Our C-FN-Scheme achieves the best performance, and sharpens blurred color edges selectively much better than the existing
methods such as the adaptive peaking method. The C-FN-Scheme can be applied to several image processing tasks such as the suppression of focus variations, the motion de-blurring, the sharpening of mosaicked color images acquired with a single solidstate color camera, etc. For each application, we should introduce a certain additional technique. For instance, in the motion deblurring case, we need to incorporate directional selectivity according to estimated image motion into the overshooting term of the underlying time-evolution. At present, we study these applications. 6. REFERENCES [1]A. Rosenfeld and A.C. Kak,"Digital picture processing," Ch.6.4.2., Academic Press, Inc., New York, 1982. [2]E.G.T. Jaspers and P.H.N. de With,"A generic 2D sharpness enhancement algorithm for luminance signals," Proc. IEE Int’l. Conf. IPA97, pp.263-273, 1997. [3]S. Thurnhofer and S.K. Mitra,"A general framework for quadratic Volterra filters for edge enhancement," IEEE Trans.
Image Process., 5, pp.950-963, 1996. [4]M. Proesmans, E.J. Pauwels and L.J. Van Gool,"Coupled geometry driven diffusion equations for low-level vision," Geometry-Driven Diffusion in Computer Vision, B.M. ter Haar Romeny (Ed.), pp.191-228, Kluwer, Dordrecht, 1994. [5]T. Saito, et al.,"Selective image sharpness enhancement by coupled nonlinear reaction-diffusion time-evolution and its practical application," Proc. EUSIPCO 2002, vol.II, pp.445-448, 2002. [6]P. Perona and J. Malik,"Scale-space and edge detection using anisotropic diffusion," IEEE Trans. Pattern. Anal. & Mach. Intell., 12, pp.629-639, 1990. [7]G. Sapiro and D. L. Ringach,"Anisotropic diffusion of multivalued images with application to color filtering," IEEE Trans. Image Process., 5, pp.1582-1586, 1996. [8]S. Osher and L. Rudin,"Feature-oriented image enhancement using shock filters," SIAM J. Num. Anal., 27, pp.919-940, 1990.
Table 1. Quantitative performance evaluations of our six different schemes for selective sharpness enhancement of color images and the existing adaptive sharpening method. Nr * * * 0.559 0.555 0.592 0.479 0.645 0.476 0.701 0.718 0.684 0.719 0.722 0.727 0.738 0.727 0.745 0.742 0.738 0.743 -0.003 -0.002 -0.002 -0.024 -0.001 -0.006 -0.002 0.000 0.002
Ar * * * 0.722 0.707 0.671 0.819 0.655 0.821 0.684 0.590 0.651 0.616 0.608 0.600 0.600 0.608 0.591 0.583 0.578 0.559 0.981 0.983 0.981 0.868 0.999 0.980 0.997 0.998 0.996
(a) Test color image (b) Sharpness-enhanced image Figure 1. Artificially blurred test color image and a sharpnessenhanced color image reproduced by our C-FN-Scheme in the primary color space (R,G,B).
45
SNR [dB]
Br * * * 0.327 0.424 0.421 0.344 0.424 0.377 0.367 0.369 0.281 0.423 0.475 0.392 0.422 0.466 0.326 0.346 0.357 0.207 0.003 0.003 0.004 0.096 0.001 0.045 0.001 0.000 0.001
SNR(TEST) SNR(G)
Br(G) Nr(G) Ar(G)
1
40
0.8
35
0.6
30
0.4
25
0.2
20 0.6
0.8 1 1.2 1.4 Gaussian blurring parameter ζ
Br, Nr, Ar
Artificially blurred R test color image G B ID-N-Scheme R (R,G,B) G B ID-N-Scheme R (G,R-G,B-G) G B ID-N-Scheme R (L*,a*,b*) G B C-FN-Scheme R (R,G,B) G B C-FN-Scheme R (G,R-G,B-G) G B C-FN-Scheme R (L*,a*,b*) G B Adaptive Peaking R Method G (R,G,B) B Adaptive Peaking R Method G (G,R-G,B-G) B Adaptive Peaking R Method G (L*,a*,b*) B
PSNR [dB] 32.712 33.644 34.046 35.288 36.580 37.385 34.245 37.103 35.459 35.548 37.585 37.125 36.471 37.693 38.001 36.621 37.663 37.843 36.425 37.636 37.617 32.714 33.644 34.048 32.983 33.644 34.071 32.699 33.645 34.039
0
Figure 2. SNR(G), Br(G), Nr(G) and Ar(G) versus the blurring parameter ζ, for the green color component of the color image sharpened by the C-FN-Scheme in the primary color space (R,G,B); SNR(TEST) means SNR of each blurred test image.
ABSTRACT Previously we have presented a selective sharpening method for monochrome images. Our method is based on the simultaneous nonlinear reaction-diffusion time-evolution equipped with a nonlinear diffusion term, a reaction term and an overshooting term, and can sharpen degraded edges blurred by several causes without increasing the visibility of random noise. This paper extends our method to selective sharpening of color images. As to the how to extend it, we take into accounts two variations about the treatment of three color components and the selection of the color space. By experiments, we quantitatively evaluate performance of these variations. The best performance is achieved by the collective treatment of color components based on the simultaneous full-nonlinear reaction-diffusion time-evolution, and blurred color edges are sharpened selectively much better than by the existing methods such as the adaptive peaking method.
consider other color spaces. Taking these possibilities into accounts, we study some variations about the treatment of color components and the selection of the color space. By experiments, we quantitatively compare performance of these variations. 2. SHARPENING OF MONOCHROME IMAGE [5] 2.1. Simultaneous time-evolution equations Previously we have derived the simultaneous nonlinear reactiondiffusion time-evolution equations: ∂f σ2 = div g ( ∇f ) ⋅∇f − s ⋅ (u x + v y ) − ⋅( f − I ) ∂τ 2λ ∂u 1 = ⋅ div g ( ∇u ) ⋅∇u − (u − f x ) ∂τ λ ∂v 1 = ⋅ div g ( ∇v ) ⋅∇v − (v − f y ) ∂τ λ , I : input image , f : time-evolution image ,
(1)
where the two auxiliary functions u, v approximate the first spatial derivatives of the function f, as follows: 1. INTRODUCTION As an image sharpening method, the peaking technique [1],[2] have been popular and used for practical applications. Recently another type of sharpening method using nonlinear filters such as a Volterra filter has been proposed [3]. However, these existing methods have their own limitations; they cannot work well for blurred image corrupted by random noise, and will produce the side effect that the noise visibility are augmented to some extent. Previously, extending a prototype of the simultaneous nonlinear reaction-diffusion time-evolution, proposed by M. Proesmans et al. [4], to selective monochrome-image sharpening, we have formed a new selective sharpening method using the simultaneous nonlinear reaction-diffusion time-evolution , and shown that our method has an ideal capability to sharpen blurred edges without increasing the visibility of random noise [5]. Its discrete expression is defined as the iterative nonlinear operations, and by experiments we have demonstrated that the discrete expression has a desirable property that computing a certain decision criterion halts its iteration when it achieves the best selective sharpness enhancement [5]. This paper extends our method to selective sharpening of color images. As to the extension, there are some options. One possible extension is to treat three color components separately, whereas another possible extension is to treat three color components collectively, where, instead of the magnitude of the intensity gradient [6], we introduce a certain measure quantifying the magnitude of a color edge to control the time-evolution [7]. Moreover, in addition to the primary RGB color space, we
0-7803-7750-8/03/$17.00 ©2003 IEEE.
u=fx +δ , v= fy +δ′ ,
(2)
and the function g means the nonlinear diffusivity function and in this paper is given by
{
g ( z) = 1 1+ ( z K )
2
}
.
(3)
2.2. Discrete form of the time-evolution equations The discrete expression for equation 1 is defined by
{(
fiτ, j+1 = fiτ, j + ε ⋅ ∑ g ∇d fiτ, j ⋅∇d fiτ, j d =N ,S ,E ,W − ,
∑
s 2
d = N , S , E ,W
⋅
{(u
τ +1 i +1, j
)
} − σ2λ ⋅ ( f
− uiτ−+1,1 j ) + ( viτ,+j1+1 − viτ,+j1−1 )
{g ( ∇ f ) ⋅∇ f } τ d i, j
(
}
)
2
τ i, j
− I i , j )
τ d i, j
(
)
= g ∇ N fiτ, j ⋅∇ N fiτ, j + g ∇S fiτ, j ⋅∇S fiτ, j
(
τ E i, j
+g ∇ f
) ⋅∇
τ E i, j
(
τ W i, j
f +g ∇ f
) ⋅∇
τ W i, j
f
, ∇N fiτ, j = fiτ, j −1 − fiτ, j , ∇S fiτ, j = fiτ, j +1 − fiτ, j ∇E fiτ, j = fiτ+1, j − fiτ, j , ∇W fiτ, j = fiτ−1, j − fiτ, j ε
uiτ,+j1 = uiτ, j + ⋅ λ
∑
d = N , S , E ,W
{
{g ( ∇ u ) ⋅∇ u } τ d i, j
τ d i, j
(4)
}
− ε ⋅ uiτ, j − 12 ⋅ ( fiτ+1, j − fiτ−1, j ) ε
viτ,+j1 = viτ, j + ⋅ λ
∑
d = N , S , E ,W
{
{g ( ∇ v ) ⋅∇ v } τ d i, j
τ d i, j
}
− ε ⋅ viτ, j − 12 ⋅ ( fiτ, j +1 − fiτ, j −1 ) ,
τ i, j
f ; intensity of the (i, j ) pixel at the τ -th iteration Ii , j ; intensity of the (i, j ) pixel in the input image I .
ICIP 2003
The initial setting is given by f i ,0j = I i , j ; ui0, j = 12 ⋅ ( I i +1, j − I i −1, j ) ; vi0, j = 12 ⋅ ( I i , j +1 − I i , j −1 ) .
(5)
As a decision scheme to halt the iteration, we have proposed if δ τ −1,τ − δ τ ,τ +1 ≤ δT , then stop the iteration, , δ τ ,τ +1 = δ τf ,τ +1 δ wτ +1
δ τf ,τ +1 = ∑ f iτ, j+1 − f iτ, j
∑1
i, j
δ
τ +1 w
{
=∑ u i, j
τ +1 i, j
i, j
τ +1 i, j
+v
(6)
2 ⋅ ∑1 . i, j
}
Equation 4 has a desirable property that the above decision scheme halt the iteration almost at the ideal moment when it achieves the best selective sharpness enhancement. In most cases, equation 4 is stopped within 100 iterations. 3. EXTENSION TO SHARPENING OF COLOR IMAGES In this paper, we consider three different color spaces: the primary color space (R,G,B), the color-difference space (G,R-G,B-G) and the CIE LAB color space (L*,a*,b*). The CIE LAB color space is a kind of uniform lightness-chromaticity scale color space. In each case, a color vector is defined by C=
( R, G , B ) T
; primary color space,
(G , R − G , B − G ) T
; color-difference space,
( L*, a*, b*) T
; CIE LAB color space.
(7)
The time-evolving color vector and the input color vector are denoted by C = ( C(1) , C(2) , C(3) )T
; time-evolving color vector,
I = ( I (1) , I ( 2) , I (3) )T
; input color vector.
(8)
3.2. Collective full-nonlinear reaction-diffusion scheme (C-FN-Scheme) The C-FN-Scheme collectively treats all the color components; it uses the magnitude of a color edge to control the nonlinear diffusion term of the time-evolution. The magnitude of the color edge is defined from the Riemannian metric [7] defined by dx 2 dC = ( dx dy ) ⋅ [H] ⋅ ; Riemannian metric dy E F , [H] = F G 2
The color edge appears along the eigenvector corresponding to the maximum eigenvalue λ+ of the matrix H, and the magnitude of the color edge γ is defined by using the maximum eigenvalue λ+ and the minimum eigenvalue λ- , as follows: γ = λ+ − λ − =
(( E − G ) + 4 F )
∂C( n ) ∂τ ∂u( n ) ∂τ ∂v( n ) ∂τ
(
)
= div g ( ∇C( n ) ) ⋅ ∇C( n ) − s ⋅ (u( n ) )x + ( v( n ) )y − = =
1
λ 1
λ
⋅ div g ( ∇u ( n )
) ⋅ ∇u
⋅ div g ( ∇v( n )
) ⋅ ∇v
(n )
( n)
σ2 2λ
⋅ ( C( n ) − I ( n ) )
− (u( n ) − (C( n ) )x )
(
− v( n ) − (C( n ) ) y
)
(9)
, n = 1, 2, 3 , I ( n ) : input color component, C( n ) : time-evolution color component.
∂C( n )
; ξ τ +1 = max { ξ (τn+)1 n =1,2,3
1
4
∂τ
γ =
(( E − G ) + 4 F ) 2
)
3
2
1
4
2
3
3
n =1
n =1
, n = 1, 2, 3 .
(
E(τN+1) i , j = 1 + ∑ min mod u(τn+)1i , j −1 , u (τn+)1i , j n =1
3
)
2
(
F(τN+)1i , j = ∑ min mod uτ( n+)1i , j −1 , u τ( n+)1i , j ⋅ min mod vτ( n+)1i , j −1 , v(τn+)1i , j
n =1
(
, min mod [α , β ] =
δ (τn,τ) +1 = δ Cτ ,(τn+)1 δ wτ +( n1)
(13)
2
However, as the time-evolution for the two auxiliary functions u, v approximating the first spatial derivatives of each color component, the C-FN-Scheme employs the same equations as the ID-N-Scheme. Moreover, we employ the simultaneous stopping scheme. In the discrete expression of equation 13, the magnitude of the color edge γ is estimated at an intermediate point between samples. Thus we need specify the half-point values for the scalar values of E, F and G. For example, in the case of the half-point in the north, those values are estimated by taking the minmod [8] between the neighboring samples:
G(τN+1) i , j = 1 + ∑ min mod vτ( n+)1i , j −1 , vτ( n+)1i , j
ξ(τn+)1 = δ (τn−)1,τ − δ (τn,τ) +1
σ2 ⋅ (C ( n ) − I ( n ) ) 2λ
E = 1 + ∑ (u( n ) ) , F = ∑ (u( n ) ⋅ v( n ) ) , G = 1 + ∑ (v( n ) )
3
}
(12)
.
(
n =1
if ξ τ +1 ≤ δ T , then stop the iteration for all the color components,
2
= div g (γ ) ⋅∇C( n ) − s ⋅ (u( n ) )x + (v( n ) ) y −
3
As the decision scheme to halt its iteration, we employ the following simultaneous stopping scheme:
2
In the one dimensional case, namely n = 1, the definition of the magnitude of the color edge γ reduces to the magnitude of the intensity gradient. Unlike the ID-N-Scheme, as the time-evolution for each color component, the C-FN-Scheme employs the following fullnonlinear reaction-diffusion time-evolution:
n =1
3.1. Independent nonlinear reaction-diffusion scheme (IDN-Scheme) The ID-N-Scheme applies the time-evolution of equation 1 independently to each color component, as follows:
(11)
2
3 3 ∂C ∂C ∂C ∂C E = 1 + ∑ ( n) , F = ∑ ( n) ⋅ (n) , G = 1 + ∑ (n) . x x y ∂ ∂ ∂ n =1 n =1 n =1 ∂y 3
sign (α ) + sign ( β ) 2
)
)
2
⋅ min ( α , β
(14)
).
(10)
In equation 14, instead of the minmod function, as an alternative we may adopt the average function, but the use of the minmod function has the advantage that it sharpens color edges better than the use of the average function.
The simultaneous stopping scheme can halt the iteration almost at the ideal moment.
4. PERFORMANCE EVALUATIONS 4.1. Test color images and performance measures We evaluate performance of our schemes using artificially blurred
δ Cτ ,(τn+)1 = ∑ C(τn+)1i , j − C(τn) i , j
∑1
i, j
{
δ wτ +( n1) = ∑ uτ(n+)1i , j + vτ( n+)1i , j i, j
i, j
}
2 ⋅ ∑1 . i j ,
test color images. First we blur an original sharp color image h(x,y), given in RGB components, h = ( h(1) , h(2) , h(3) )T ,
(15)
with the Gaussian filter having the impulse response G(x,y ; ζ), ( x2 + y2 ) 1 − , G ( x, y;ς ) = ⋅ exp 2πς 2 2ς 2
(16)
and then add random Gaussian noise to the blurred color image G*h(x,y); thus we generate an artificially blurred test color image I(x,y), given in RGB components: I = ( I (1) , I ( 2) , I (3) )T .
(17)
Let C(x,y) denote the sharpness-enhanced color image reproduced from the blurred test color image I(x,y), and it is expressed in the primary RGB color space: C = ( C(1) , C(2) , C(3) )T .
(18)
We evaluate the performance of our schemes in the RGB color space even though the color vector is processed in the other color space, because existing display devices produce a RGB color image. For the performance evaluation, we define the quantitative measures in the RGB color space as follows. (1) SNR(n) of the sharpened color image: SNR is computed between the original sharp color image h and the sharpened color image C: SNR( n ) = 20 ⋅ log10 ( 255 D( n ) ) [ dB]
2 , D( n ) = E( x , y ) {C( n ) ( x, y ) − h( n) ( x, y )} .
(19)
The simultaneous stopping scheme of equation 10 can halt the iteration of our time-evolution schemes of equations 9 and 13 almost at the ideal moment when it attains its maximum SNR. (2) Blur-Removal Ratio Br(n) & Noise-Removal Ratio Nr(n): Let the vectors, b(n), n(n), s(n), be Gaussian-blurs artificially added to the original color image component h (n) , random Gaussian noise components added to the Gaussian blurred color image component G*h(n), and deviations of the artificially blurred test color image component I (n) from the sharpened color image component C(n), i.e. I(n) - C(n), respectively. We define those vectors by arranging their respective values at all the pixels in one column. Then, we define the blur-removal ratio Br(n) and the noise-removal ratio Nr(n) as follows: Br( n ) = ( b ( n ) , s( n )
)
b( n )
2
, Nr( n ) = ( n( n ) , s( n )
)
n( n )
2
. (20)
The positive value of Br (n) / Nr(n) means that the blur / noise removal is successfully achieved for the n-th color component. If blur / noise is perfectly removed for the n-th color component, then the value of Br(n) / Nr(n) will be 1; but the reverse is not necessarily true. On the contrary, the negative value means that the blur / noise is augmented far from being removed. (3) Artifact Component Ratio Ar (n): We define the artifactcomponent ratio Ar(n) as follows: Ar( n ) =
( Id − P
(n)
⋅ ( P(Tn ) ⋅ P( n )
, P( n ) = b ( n ) , n ( n )
)
−1
)
⋅ P(Tn ) ⋅ s ( n )
s(n)
, Id ; identity matrix .
(21)
This measure is given by the ratio of the norm of the projection of s (n) onto the orthogonal complement of the linear subspace spanned by b(n) and n(n), to the norm of s(n), and it quantifies how far the enhancement signal component -s (n), produced by the sharpening algorithm, contains undesirable artifact components
irrelevant to the compensation for the blur b(n) and the noise n(n). If no artifact occurs, then the value of the artifact-component ratio Ar(n) will be 0. 4.2. Evaluation results and further investigations In the production of artificially blurred test color images, we generate additive Gaussian noise with a zero mean and a standard deviation of 5.0. Figure 1(a) shows an artificially blurred test color image, where we set the blurring parameter ζ of the Gaussian filter defined in equation 16 to 1.0. As for the sharpness enhancement parameters of λ, σ and K, their proper values chiefly depend on the noise variance. We experimentally investigate their proper values. The variance dependence of the parameters of λ and σ is much weaker than that of the parameter K. If we set the parameter K close to the standard deviation of noise, the ID-N-Scheme and the C-FNScheme provide suboptimal selective sharpening performance. We set the parameters of ε, δT, λ, σ and K to 0.05, 10 - 4, 1.0, 1.0 and 5.0, respectively; and these suboptimal parameter values are found out by experiments. On the other hand, the shooting parameter s strongly depends on the blurring parameter ζ, and its proper value is different not only for every scheme but also for every color space. Hence, in this paper, for every case, we optimize the shooting parameter s. We apply the ID-N-Scheme and the C-FN-Scheme, to the selective sharpness enhancement of figure 1(a) in the three different color spaces: (R,G,B), (G,R-G,B-G) and (L*,a*,b*). For every scheme we optimize the shooting parameter s. Table 1 shows the values of SNR(n), Br(n), Nr(n) and Ar(n). As shown in table 1, the C-FN-Scheme outperforms the ID-N-Scheme, and the CFN-Scheme achieves almost equally excellent performance irrespective of the selection of the color space. Table 1 also shows the results of the quantitative performance evaluations for the adaptive peaking method [2], one of the typical existing schemes. In the case of the adaptive peaking method, the values of Br(n) are very close to zeros, and the values of Nr(n) are negative; these results means that the adaptive peaking method cannot attain selective sharpness enhancement at all. On the other hand, our two schemes successfully achieve selective sharpness enhancement. Figure 1(b) shows a sharpness-enhanced color image produced by the C-FN-Scheme in the primary color space (R,G,B). The C-FN-Scheme gives a sharpened image of subjectively superior quality. Next, setting the sharpness enhancement parameters λ, σ, K to 1.0, 1.0, 5.0, respectively, and changing the blurring parameter ζ of the Gaussian filter, we apply the C-FN-Scheme to the selective sharpness enhancement in the primary color space (R,G,B). In addition, for every blurred test color image, we optimize the shooting parameter s. Figure 2 shows the values of SNR(G), Br(G), Nr(G) and Ar(G) versus the blurring parameter ζ, for the green color component. Figure 2 also shows the SNR(TEST) of each blurred test image. If the blurring parameter ζ is less than 1.2, the C-FN-Scheme will achieve selective sharpening of blurred color edges; but if the blurring is more intensive, the C-FN-Scheme will not necessarily work well. 5. CONCLUSIONS Our C-FN-Scheme achieves the best performance, and sharpens blurred color edges selectively much better than the existing
methods such as the adaptive peaking method. The C-FN-Scheme can be applied to several image processing tasks such as the suppression of focus variations, the motion de-blurring, the sharpening of mosaicked color images acquired with a single solidstate color camera, etc. For each application, we should introduce a certain additional technique. For instance, in the motion deblurring case, we need to incorporate directional selectivity according to estimated image motion into the overshooting term of the underlying time-evolution. At present, we study these applications. 6. REFERENCES [1]A. Rosenfeld and A.C. Kak,"Digital picture processing," Ch.6.4.2., Academic Press, Inc., New York, 1982. [2]E.G.T. Jaspers and P.H.N. de With,"A generic 2D sharpness enhancement algorithm for luminance signals," Proc. IEE Int’l. Conf. IPA97, pp.263-273, 1997. [3]S. Thurnhofer and S.K. Mitra,"A general framework for quadratic Volterra filters for edge enhancement," IEEE Trans.
Image Process., 5, pp.950-963, 1996. [4]M. Proesmans, E.J. Pauwels and L.J. Van Gool,"Coupled geometry driven diffusion equations for low-level vision," Geometry-Driven Diffusion in Computer Vision, B.M. ter Haar Romeny (Ed.), pp.191-228, Kluwer, Dordrecht, 1994. [5]T. Saito, et al.,"Selective image sharpness enhancement by coupled nonlinear reaction-diffusion time-evolution and its practical application," Proc. EUSIPCO 2002, vol.II, pp.445-448, 2002. [6]P. Perona and J. Malik,"Scale-space and edge detection using anisotropic diffusion," IEEE Trans. Pattern. Anal. & Mach. Intell., 12, pp.629-639, 1990. [7]G. Sapiro and D. L. Ringach,"Anisotropic diffusion of multivalued images with application to color filtering," IEEE Trans. Image Process., 5, pp.1582-1586, 1996. [8]S. Osher and L. Rudin,"Feature-oriented image enhancement using shock filters," SIAM J. Num. Anal., 27, pp.919-940, 1990.
Table 1. Quantitative performance evaluations of our six different schemes for selective sharpness enhancement of color images and the existing adaptive sharpening method. Nr * * * 0.559 0.555 0.592 0.479 0.645 0.476 0.701 0.718 0.684 0.719 0.722 0.727 0.738 0.727 0.745 0.742 0.738 0.743 -0.003 -0.002 -0.002 -0.024 -0.001 -0.006 -0.002 0.000 0.002
Ar * * * 0.722 0.707 0.671 0.819 0.655 0.821 0.684 0.590 0.651 0.616 0.608 0.600 0.600 0.608 0.591 0.583 0.578 0.559 0.981 0.983 0.981 0.868 0.999 0.980 0.997 0.998 0.996
(a) Test color image (b) Sharpness-enhanced image Figure 1. Artificially blurred test color image and a sharpnessenhanced color image reproduced by our C-FN-Scheme in the primary color space (R,G,B).
45
SNR [dB]
Br * * * 0.327 0.424 0.421 0.344 0.424 0.377 0.367 0.369 0.281 0.423 0.475 0.392 0.422 0.466 0.326 0.346 0.357 0.207 0.003 0.003 0.004 0.096 0.001 0.045 0.001 0.000 0.001
SNR(TEST) SNR(G)
Br(G) Nr(G) Ar(G)
1
40
0.8
35
0.6
30
0.4
25
0.2
20 0.6
0.8 1 1.2 1.4 Gaussian blurring parameter ζ
Br, Nr, Ar
Artificially blurred R test color image G B ID-N-Scheme R (R,G,B) G B ID-N-Scheme R (G,R-G,B-G) G B ID-N-Scheme R (L*,a*,b*) G B C-FN-Scheme R (R,G,B) G B C-FN-Scheme R (G,R-G,B-G) G B C-FN-Scheme R (L*,a*,b*) G B Adaptive Peaking R Method G (R,G,B) B Adaptive Peaking R Method G (G,R-G,B-G) B Adaptive Peaking R Method G (L*,a*,b*) B
PSNR [dB] 32.712 33.644 34.046 35.288 36.580 37.385 34.245 37.103 35.459 35.548 37.585 37.125 36.471 37.693 38.001 36.621 37.663 37.843 36.425 37.636 37.617 32.714 33.644 34.048 32.983 33.644 34.071 32.699 33.645 34.039
0
Figure 2. SNR(G), Br(G), Nr(G) and Ar(G) versus the blurring parameter ζ, for the green color component of the color image sharpened by the C-FN-Scheme in the primary color space (R,G,B); SNR(TEST) means SNR of each blurred test image.
