Image restoration using digital inpainting and noise removal
Image and Vision Computing 25 (2007) 61–69 www.elsevier.com/locate/imavis
Image restoration using digital inpainting and noise removal Celia A. Zorzo Barcelos a, Marcos Aure´lio Batista b,* b
a Faculty of Mathematics, Federal University of Uberlaˆndia, Caixa Postal 593, CEP 38.400-902, Uberlaˆndia, MG, Brazil. Department of Computing, Federal University of Goia´s, Campus Catala˜o, Caixa Postal 56, CEP 75704-020, Catala˜o, GO, Brazil.
Received 26 May 2004; received in revised form 4 August 2005; accepted 18 December 2005
Abstract Inpainting and denoising are two important tasks in the field of image processing with broad applications in image and vision analysis. In this paper, we present a new approach for image restoration. Our method simultaneously fills in missing, corrupted, or undesirable information while it removes noise. The denoising is performed by the smoothing equation working inside and outside of the inpainting domain but in completely different ways. Inside the inpainting domain, the smoothing is carried out by the Mean Curvature Flow, while the smoothing of the outside of the inpainting domain is carried out in a way as to encourage smoothing within a region and discourage smoothing across boundaries. Besides smoothing, the approach here presented permits the transportation of available information from the outside towards the inside of the inpainting domain. This combination permits the simultaneous use of filling-in and differentiated smoothing of different regions of an image. The experimental results show the effective performance of the combination of these two procedures in restoring scratched photos, disocclusion (or removal of entire objects from the image) in vision analysis and text removal from images. q 2006 Elsevier B.V. All rights reserved. Keywords: Inpaint; Image processing; Noise removal; Edge detection; Diffusion equation; Transport equation
1. Introduction Inpainting is a practice carried out by artists when modifying a picture, in such a way so that an observer is unable to detect any changes. The terminology for digital inpainting was first introduced by Bertalmı´o, Sapiro, Caseles and Ballester [7] where these authors introduced a partial differential equation model based on the transport theory. The inpainting process consists of, in general, the filling-in of missing information within a domain D or to replace this domain with a different kind of information, based upon the image information available outside of the domain D. This domain is referred to as the inpainting domain and is where the original image has been damaged due to age action or also the region that we desire to add or remove information. The fillingin of missing information and the removal of noise are two very important topics in image processing, with several applications such as image coding and wireless image transmission (e.g. * Corresponding author. Address: Department of Computing, Federal University of Goia´s, Campus Catala˜o, Caixa Postal 56, CEP 75704-020, Catala˜o, GO, Brazil. Tel.: C55 64 34411500 E-mail addresses: [email protected] (C.A.Z. Barcelos), [email protected]. br (M.A. Batista).
0262-8856/$ - see front matter q 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2005.12.008
recovering lost blocks), special effects (e.g. removal of objects) and image restoration (e.g. fold lines, scratches and noise removal). The basic idea of inpainting algorithms is to fill-in regions with available information from their surroundings. In most cases, the available data of the original image is noisy which makes it necessary to eliminate the noise and fill-in the blank spaces (those without information). The basic idea of our algorithm is to complete these spaces which hold no information and eliminate noise (if exists) while preserving the edges, and the goal of this work is to recover the entire clean image u(x) from a given incomplete noisy image I(x) observed only outside of an inpainting domain D, performing the inpainting and denoising action simultaneously. The fill-in and the denoising procedures are carried out automatically, and are independent of the topology of the objects that compose the image or of the inpainting domain D, which could be a union of several disconnected regions. In Section 2, we briefly describe the inpainting scheme given in [6,7] and the Mean Curvature Flow model. The proposed model for inpainting and denoising is described in Section 3. The Euler discretization of the proposed model and the numerical implementation are discussed in Section 4 as well as showing the application of our model in various examples, including denoising and restoring scratched pictures, texts and object removal from images. A note emphasizing the necessity of the diffusion process is presented
62
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
in Section 5 and the concluding remarks are presented in Section 6. Throughout the paper, U denotes the entire domain of the image, D the inpainting domain, DC the available part of the image I on U and u the restored image. 2. Image inpainting and image denoising Inpainting is the art of rebuilding the basics of visual art and consists of filling-in unknown data in a known region of an image, with the principal objective of restoring harmony to the given damaged picture with parts worn by time, overexposed objects or objects we want to remove from the image. To fill-in the inpainting domain, in a given picture, professionals follow these basic rules: (1) Observe the image as a whole. (2) Paint the inpainting domain in the same tones found in the surrounding background regions. (3) Using, as a basis, similar regions of the same image or known images of the same theme to carry out repairs in damaged regions and then add texture. In [6,7], Bertalmı´o et al. introduced the digital inpainting concept and proposed the Partial Differential Equations (PDE’s) model to simulate the digital work of the artisan’s art of inpainting. As the work of these artisans, is highly subjective, they limit the digital simulation to the automatization of the item (2), and for this, they proposed to transport the information from the surrounding inpainting domain into its interior always following the direction of the isophotes, arriving at vD. In order to make the transport of information follow this proposal, Bertalmio uses the inner product between two vectors, one indicates the inpainting’s normal direction (P(Du)) and the other indicates the isophote direction (Ptu). Following this procedure, the inpainting domain receives information and thus decreases. The goal of the inpainting model introduced in [6,7] is to transport as smoothly as possible (along the isophotes) the information from the surrounding inpainting domain. The transport of information is performed by solving the following PDE ut Z VðDuÞ$Vt u;
x 2D; tO 0
(1)
where t is a scale space parameter as in [2,3,12]. Note that this evolution equation runs only inside the region to be inpainted D. In Eq. (1), Ptu (Fig. 1a), is a vector which indicates the direction of u(x) variation of least intensity. The absolute value of Ptu is numerically equal to the instant variation rate of u(x) along the isophotes. The vector P(Du) indicates the direction where the value of u(x) varies abruptly (indicating a border or edge) and its absolute value is numerically equal to the Laplacian instant variation rate in this direction (Fig. 1b). The inner product of these two vectors (Fig. 1c) gives us the value to be transported to the pixel in question. We observe that in the case where the two vectors are perpendicular the transport will not be carried out. In this case, we have a strong
Fig. 1. (a) The perpendicular vector to Pu, (b) the P(Du) vector and (c) the two vectors showing where the inpainting will be performed, one notices that from the lowest lying pixel there will not be any transport of information once the vector P(Du) and Ptu are perpendicular.
indication that the pixel in question is localized on a border. In other cases where the transport does not happen P(Du), or Ptu, is a void vector. For more details of this approach, see [6,7]. The inpainting method proposed in [6,7] consists of intercalating the Eq. (1) with a diffusion equation whose objective is ‘to ensure a correct evolution of the direction field’ [6]. They used the following anisotropic diffusion equation ut Z g3 KjVuj;
x 2D3
(2)
where D3 is a dilation of D with radius 3, K is the Euclidean curvature of u, and g3 is a smooth function in D3 such that g3Z0 in vD3 and g3Z1 in D3.1 We will refer to this model as the BSCB model. In this paper, we propose a modification of the BSCB model, which consists of using different diffusion processes at points whether they belong to the inpainting domain or not. The equation used in DC makes a differentiated diffusion at points whether or not they belong to the edges of the objects that compose the image. In this way, the noise will be eliminated and the edges preserved. Such an equation enriches the inpainting process eliminating any eventual noise in the image, at the same time a correct evolution is possible in the direction of the transportation and stabilizes the process, eliminating the necessity to carry out a diffusion in D3 which in practice creates the ‘frame effect’, this means around the vicinity of the inpainting domain, within the dilation of D, the data is smoothed. Several PDE-based techniques have been proposed for the smoothing of an image, some obtained on the direct derivation of the evolution equations, and others from energy approaches such as the L2 norm dependent model or from the total variation model of Rudin, Osher and Fatemi [16]. Here, we will smooth the data inside the inpainting domain using the Mean Curvature Flow (MCF) equation (see [1,2] for further details), which is given by ut Z jVujdiv
Vu : jVuj
This equation diffuses u in the direction orthogonal to its gradient Pu and does not diffuse in the direction of Pu. The level sets of the solution of the MCF equation move in the 1 In the numerical implementations, Eq. (2) was not implemented in D3, a non-linear scaling was used to stabilize the discrete equation (see [6], p. 66).
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
63
Fig. 2. Top to bottom, left to right: Lenna image degraded by black and white spots and by blank region (flower format), inpainting domain, three intermediate steps and final results of the proposed model.
64
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
Fig. 3. (a) Degraded image with Poisson noise and a blank domain. (b and c) Partial results and (d) final result of proposed model.
Fig. 5. (a) Degraded image corrupted Gaussian noise (0 dB) and a blank domain. (b and c) Partial results and (d) final result of proposed model.
normal direction with a speed proportional to their mean curvature.
The proposed model in this work will deal with two tasks: denoising and inpainting. Our problem is to reconstruct a damaged image or to remove an object from an image. Our proposed method will inpaint the domain D, which has scratches or missing patches and will denoise the image in DC. Both tasks will be performed simultaneously. The goal of this work is to develop the idea of removing noise without losing the boundaries or edges and inpaint the
damaged regions. Our approach presents different procedures inside and outside of the inpainting domain. The inpainting process consists of filling-in the missing information in the empty domain D, based upon the available image information found outside of the domain D as in [7,8], and will be performed only inside the inpaint domain D while the smoothing procedure will be performed in all x2U but will act differently depending on if point x belongs, or not, to the inpainting domain D. Inpainting. The algorithm will execute, for each x belonging to the inpainting domain D, the transportation of neighboring pixel information belonging to D C, where the image information is available, using the Eq. (1) as in [7].
Fig. 4. (a) Degraded image with Gaussian noise (10 dB) and a blank domain. (b and c) Partial results and (d) final result of proposed model obtained after 100 iterations and using as parameter kZ0.05, DtiZ0.1, DtsZ0.2 and AZBZ10.
Fig. 6. (a) Degraded image by Gaussian noise (0 dB) and by a blank domain. (b and c) Partial results and (d) final result of proposed model.
3. The proposed scheme
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
65
Fig. 7. Plots of the 80th line of Fig. 6. Left to right: degraded image, partial and final results.
Smoothing. A smoothing procedure will be applied for each x belonging to the domain U. If x2D, the smoothed u(x) will be the solution of the Mean Curvature Flow equation, otherwise, i.e. if x lies outside of the inpainting domain a selective diffusion equation will be used. The smoothed version u(x) will be the solution to the following parabolic equation Vu ut Z gjVujdiv Kð1KgÞðuKIÞ; x 2Dc ; tO 0 (3) jVuj
Vu ut Z jVujdiv ; jVuj
x 2D
uðx;0Þ Z IðxÞ; x 2U vu Z 0; x 2vU; vt vU!RC
The non-linear space scale scheme (3), (5) and (6) for data smoothing was introduced in [4] following the ideas given in [2,3,12,15], with the objective of eliminating unnecessary parameters, where in many cases they are correlated. This equation allows one to perform selective smoothing in accordance with the value of the image gradient at each point x. We would also like to mention some other interesting works on inpainting. In [10,13], the authors present a technique for removing occlusion solving the following variational formulation
(4) (5)
tO 0
(6)
where gZg(jPuj). I(x, y) is an image to be processed, u(x, y, t) is its smoothed version on the scale t. The function g(s)R0 is a non-increasing function, satisfying g(0)Z1 and g(s)/0 when s/N. The term jVujdivðVu=jVujÞZ DuKV2 uðVu;VuÞ=jVuj2 diffuses u in the orthogonal direction to its gradient Pu and does not diffuse it in any other direction. The goal is to allow smoothing in the image u in a way that it is performed on both sides of an edge with minimal smoothing on the edge itself. The term g(jPuj) is used for edge detection and controls the diffusion speed: if Pu has a small value at the point x, it will be considered an interior point, and the diffusion will be stronger; on the other hand, that is, if Pu has a large value at the point x, then this point x will be considered an edge point, and the diffusion will be low since g(s) always assumes values for large values of s. The balance between the forcing term and the diffusion term is made by (1Kg), which works as a moderate selector of the diffusion process. Thus, the proposed model consists of selectively applying the Alvarez, Lions and Morel model’s [2] in areas of the image that demand a larger suavization. This model also consists of forcing, in an incisive way, the smoothed image u to remain close to the initial image I in the boundary areas which have gw0. On the other hand, in homogeneous areas gw1, and therefore, the forcing term will have an inexpressive effect, which allows for a better suavization of the image.
Fig. 8. Top to bottom, left to right: degraded image by characters, inpainting domain, three intermediate steps and final results of the proposed model. Parameters: 50 iterations, AZ10, BZ1, kZ3, DtiZ0.1 and DtsZ0.2.
66
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
Vu p EðuÞ Z jVuj a C bdiv ; pR 1;aO 0;bO 0; jVuj ð
(7)
D
to perform disocclusion and the practical algorithm for disocclusion, connecting the T-junctions associated with the same gray level using elastica minimizing curves. In [13,14], the authors used dynamic programming to solve the variational approach numerically. As reported by the authors, the regions to be inpainted are limited to having simple topologies, e.g. holes are not possible taking into account the specific numerical technique proposed. In [10], the authors work on the minimization of the elastica energy functional (7) obtaining the Euler–Lagrange equation for this functional. The choice of values for constants a and b is an issue of numerical implementation. There has been no investigation about the range of these two constants, which permits convergence and/or stability of the numerical algorithm. 4. Numerical approximations and experimental results
Fig. 9. Reconstruction of the shadow area after the removal of the bag. From top to bottom and left to right: the initial image, the inpainting domain, partial and final results.
In this section, we present the results by applying the proposed model on images of varying difficulty. Our tested images are represented by 256!256 matrices of intensity values, where each matrix element ui,j is a real value correspondent to the grayscale level of the image u(x, y) at the point xZxiZiDx and yZyjZjDy. We denote u(xi, yj, tn) by uni;j , where tnZnDt. The derivative of u in relation to the time t (space scale parameter), that is, ut calculated in (xi, yj, tn) is approximated by n Euler’s method, i.e. ut wðunC1 i;j Kui;j Þ=Dt. Numerical implementation using central difference techniques works quite well in the approximation of the diffusion term u2x uyy K2ux uy uxy C u2y uxx Vu jVuj div ; Z jVuj u2x C u2y and the transport term VðDuÞ$Vt u Z ux ðuxxy C uyyy ÞKuy ðuxxx C uxyy Þ: Using Neumann’s boundary conditions we calculate unC1 ij ; nZ 1;2;.;N, by the discretization of the Eqs. (1), (3) and (4) vijnC1 Z unij C Dti Li ðunij Þ
ðx;yÞ 2D
vijnC1 Z unij (8)
ðx;yÞ 2DC unC1 Z vnC1 C Dts Ls ðvijnC1 Þ ij ij
Fig. 10. Landscape reconstruction: (a) the initial image; (b) the inpainting domain; (c) partial result; and (d) the result obtained by the proposed model running 300 iterations and parameters kZ0.5, AZBZ10, DtiZ0.02 and DtsZ0.2.
ðx;yÞ 2U
(9)
with u0ij Z Iðxi ;yi Þ, Li ðuÞZ VðDuÞ$Vt u and Vv Ls ðvÞ Z gjVvjdiv Kð1KgÞðvKIÞ; if ðx;yÞ 2DC ; jVvj Vv or Ls ðvÞ Z jVvjdiv if ðx;yÞ 2D: jVvj here, gZ gðsÞZ 1=ð1C ks2 Þ, where k is a constant and sZjPvj.
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
67
Fig. 11. (a) Initial image, (b) inpainting domain, (c–e) partial results and (f) final result obtained by the proposed model running 200 iterations using AZ3, BZ10, kZ1, DtiZ0.02 and DtsZ0.2; (g) final result obtained by the proposed model running using different parameters; (h and i) results obtained by the BSCB model using two different choice to parameter values.
As in the BSCB model, inside every inpainting domain each restoration loop will perform A steps of transport with (1) intercalated with B steps of data diffusion with (2). The following results illustrate the performance obtained by the proposed model. We performed inpainting on five typical experiments and as the principal contribution of the proposed model is the case where the initial image I(x) was damaged by noise and also has a region with lost or no information data, the first experiment presents five images corrupted by a blank domain and noise. Lenna image corrupted by black and white spots; a synthetic image corrupted by Poisson noise; the same synthetic image corrupted by Gaussian noise with signal-tonoise ratio (i.e. the ratio between the standard deviation of a noiseless image and the standard deviation of the noise) being 10 and 0 dB; another synthetic image with different structures and corrupted by Gaussian noise with SNRZ0 dB. We can see in the Fig. 2, the perfect reconstruction by the proposed model in the Lenna image degraded by black and white spots and a flower superimposed. Fig. 3 shows a synthetic image with Poisson noise and the region to be inpainted in white, the partial and the final results. Hundred iterations were performed with AZ5 steps for transport and BZ10 for diffusion and the parameters used are: kZ0.05, DtiZ0.02 and DtsZ0.2. In Fig. 4, one can see the results of the proposed model used to restore the degraded synthetic image with Gaussian noise SNRZ10 dB. In Fig. 5, 100 iterations were run, each with AZ5 iterations of transport and BZ10 iterations of diffusion. The parameters used were: DtiZ0.02, DtsZ0.2 and kZ0.004. In Fig. 6 again, 100 iterations were run, each one with AZ BZ10 iterations. As one can observe, the final result is an
image free of noise without loss of edges and with damaged parts restored. The parameters used were: DtiZ0.1, DtsZ0.2 and kZ0.05. One can easily observe in the plots of the 80th Fig. 6 lines (Fig. 7), that it was totally reconstructed eliminating the noise and inpainting the blank pixels belonging to the image (columns 50–53, 191–207 and 217–222). The second experiment the text removal from an image is shown. Fig. 8 shows the restoration of a peppers image with a superimposed text. Several intermediate steps of the reconstruction using the proposed model are shown. The third experiment shows the special effect of removing an object from a real scene. Fig. 9 shows the total removal of an object. The bag on the floor was removed as desired. This result gives us a situation where the Connectivity Principle of human perception—‘humans mostly prefer having the two disjoint parts connected, even when they are far apart’ [9,11] is satisfied, where the shadow on the floor is properly reconstructed. AZBZ10 (100 iterations). In the fourth experiment (Fig. 10), we have the performance of the proposed model applied in a textured natural image. In (a), (b), (c) and (d) we have: the natural damaged image, the inpainting domain, the partial and the final results, respectively. After 300 iterations, we can see the perfect reconstruction on the left hand part of the inpainting domain where the level line between the sky and the land (horizontal line) is strong, however, on the right hand part of the inpainting domain, we can see that the textured mountain was not very well restored. The last experiment (Fig. 11) shows that the choice for the constant’s value A and B of steps for data transportation which should be intercalated with the steps for data diffusion,
68
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
respectively, is very important. In Fig. 11f, one can see that the result carried out by our proposed model running 1000 iterations using AZ3 and BZ10 is quite good, managing to fill-in the cat and the tower simultaneously, while in Fig. 11g with another choice for A and B we achieved good results in the cat but the tower was not restored. Similar results are obtained with the BSCB model as we can see in Fig. 11h, where the tower was not restored and in Fig. 11i, where the perfect reconstruction was obtained with different values for the constants A and B. The choice of the parameter k, and the number of iterations depends on the amount of noise presented in the image and the size of the inpainting domain. In general they are chosen as being those that give better results from a visual perception point of view. Following the ideas developed in [5] we are working on the optimization of the choice of those parameters seeking better results with smaller computational effort. The computational code was written in C language. The results were obtained by using a Pentium III PC (512 MB RAM–1.2 GHz). The running time for an image of 256!256 pixels size is about 15 s for each 100 iterations with AZ10 and BZ5 and less than 2 min in all tested images. The temporal and spacial algorithm complexities are linear with the size of the data input. 5. The necessity of the diffusion process The BSCB model as well as the proposed model work combining A steps of the transport process with B steps of the diffusion process. Bertalmio, in his doctoral thesis [6] justified the need for the diffusion process to be intercalated with the transport process to ensure ‘the correct evolution of the directional field’. However, one can observe by applying the transport equation without diffusion the iterative process generated by the Euler discretization of the Eq. (1) can be nonconvergence as can be easily seen considering, for example, the initial image u0 as the function u0(x, y)Zx3Cy. The iterative process un Z unK1 C Vt unK1 $VðDunK1 Þ;
V uk $VðDuk Þ ZK6;
c k 2N;
we have that un Z unK1 C Vt unK1 $VðDunK1 Þ Z x3 C yK6n;
Acknowledgements This work is partially supported by CNPq grants.
References
generated by the transport equation is divergence. In fact, as t
denoising. The restoration is carried out by the adapted action of equations, which perform diffusion and transport. The model is a combination of three equations, one to perform the transport of the data and two to perform the smoothing inside and outside of the inpainting domain D. The results presented in our examples demonstrate the high performance of the proposed model which has demonstrated great efficiency in dealing with the inpainting of damaged images and denoising when the initial image is noisy. The nature of the noise corruption and the topology of the inpainting domain are not an issue, we have tested images corrupted by Poisson noise, Gaussian noise and black and white spots and as well as using different forms for inpaint domain D in other situations all showing satisfactory results. The size of the inpainting domain can interfere with the results. The drawback of the proposed model can be seen when we analyze the image, which is characterized by texture, a great part of which is eliminated as one can see in Fig. 10, because the inpainting domain is quite large. The proposed model and the BSCB model work in the same way and produce analogous results when applied in non-noisy images, being the first more robust as it has a wider application allowing for the restoration of an image corrupted by noise as well as with domains with no useful data. The outside smoothing of the inpainting domain gives one the stabilization of the numerical technique and there are no frame effects. From the numerical point of view, the number of intercalated steps of inpainting and diffusion should be investigated mainly when the image to be restored has several different inpainting domains. Some choices for these parameters can produce partial reconstruction as we can see in Fig. 11g and h.
c n 2N
which goes to infinity when n goes to infinity. 6. Concluding remarks In this paper, we present a mathematical model for image restoration, which consists of the recovery of missing parts in an image and also eliminating the corruption caused by noise. The model presented in this work was inspired on the BSCB model for inpainting and a balanced diffusion equation for
[1] L. Alvarez, J.M. Morel, Formalization and computational aspects of image analysis, Acta Numer. (1994) 1–61. [2] L. Alvarez, P.L. Lions, J.M. Morel, Image selective smoothing and edge detection by nonlinear diffusion, SIAM J. Numer. Anal. 29 (3) (1992) 845–866. [3] C.A.Z. Barcelos, Edge-preserving regularization in image restoration, Boletim (SBMAC) (in press). [4] C.A.Z. Barcelos, M. Boaventura, E.C. Silva Jr., A well-balanced flow equation for noise removal and edge detection, IEEE Trans. Image Process. 12 (7) (2003) 751–763. [5] C.A.Z. Barcelos, M. Boaventura, E.C. Silva Jr., Edge Detection and Noise Removal by use of a Partial Differential Equation with Automatic Selection of Parameters, Comp. Appl. Math., 24(1), pp. 131–150, 2005. [6] M. Bertalmio, Processing of Flat and non-Flat Image Information on Arbitrary Manifolds Using Partial Differential Equations, Thesis, University of Minnesota, 2001. [7] M. Bertalmio, G. Sapiro, V. Caselles, C. Ballester, Image inpainting, Computer Graphics, SIGGRAPH 2000, 2000, pp. 417–424.
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69 [8] T. Chan, J. Shen, Local inpainting models and TV inpainting, SIAM J. Appl. Math. 62 (3) (2001) 1019–1043. [9] T. Chan, J. Shen, Non-texture inpainting by curvature-driven diffusions (CDD), J. Vis. Commun. Image Rep. 12 (4) (2001) 436–449. [10] T. Chan, S. Kang, J. Shen, Euler’s elastica and curvature-based inpainting, SIAM J. Appl. Math. 63 (2) (2002) 564–592. [11] G. Kanizsa, Organization in Vision, Praeger, New York, 1979. [12] J. Malik, P. Perona, Scale-space and edge detection using anisotropic diffusion, IEEE TPAMI 12 (7) (1990) 629–639.
69
[13] S. Masnou and J.M. Morel, Level lines based disocclusion, IEEE Int. Conf. Image Processing, v.3, pp. 259–263, 1998. [14] S. Masnou, Disocclusion: a variational approach using level lines, IEEE Trans. Image Process. 11 (2) (2002) 68–76. [15] K.N. Nordstro¨m, Biased anisotropic diffusion: a unified regularization and diffusion approach to edge detection, Image Vis. Comput. 8 (1990) 318– 327. [16] L. Rudin, S. Osher, E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D 60 (1992) 259–268.
Image restoration using digital inpainting and noise removal Celia A. Zorzo Barcelos a, Marcos Aure´lio Batista b,* b
a Faculty of Mathematics, Federal University of Uberlaˆndia, Caixa Postal 593, CEP 38.400-902, Uberlaˆndia, MG, Brazil. Department of Computing, Federal University of Goia´s, Campus Catala˜o, Caixa Postal 56, CEP 75704-020, Catala˜o, GO, Brazil.
Received 26 May 2004; received in revised form 4 August 2005; accepted 18 December 2005
Abstract Inpainting and denoising are two important tasks in the field of image processing with broad applications in image and vision analysis. In this paper, we present a new approach for image restoration. Our method simultaneously fills in missing, corrupted, or undesirable information while it removes noise. The denoising is performed by the smoothing equation working inside and outside of the inpainting domain but in completely different ways. Inside the inpainting domain, the smoothing is carried out by the Mean Curvature Flow, while the smoothing of the outside of the inpainting domain is carried out in a way as to encourage smoothing within a region and discourage smoothing across boundaries. Besides smoothing, the approach here presented permits the transportation of available information from the outside towards the inside of the inpainting domain. This combination permits the simultaneous use of filling-in and differentiated smoothing of different regions of an image. The experimental results show the effective performance of the combination of these two procedures in restoring scratched photos, disocclusion (or removal of entire objects from the image) in vision analysis and text removal from images. q 2006 Elsevier B.V. All rights reserved. Keywords: Inpaint; Image processing; Noise removal; Edge detection; Diffusion equation; Transport equation
1. Introduction Inpainting is a practice carried out by artists when modifying a picture, in such a way so that an observer is unable to detect any changes. The terminology for digital inpainting was first introduced by Bertalmı´o, Sapiro, Caseles and Ballester [7] where these authors introduced a partial differential equation model based on the transport theory. The inpainting process consists of, in general, the filling-in of missing information within a domain D or to replace this domain with a different kind of information, based upon the image information available outside of the domain D. This domain is referred to as the inpainting domain and is where the original image has been damaged due to age action or also the region that we desire to add or remove information. The fillingin of missing information and the removal of noise are two very important topics in image processing, with several applications such as image coding and wireless image transmission (e.g. * Corresponding author. Address: Department of Computing, Federal University of Goia´s, Campus Catala˜o, Caixa Postal 56, CEP 75704-020, Catala˜o, GO, Brazil. Tel.: C55 64 34411500 E-mail addresses: [email protected] (C.A.Z. Barcelos), [email protected]. br (M.A. Batista).
0262-8856/$ - see front matter q 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2005.12.008
recovering lost blocks), special effects (e.g. removal of objects) and image restoration (e.g. fold lines, scratches and noise removal). The basic idea of inpainting algorithms is to fill-in regions with available information from their surroundings. In most cases, the available data of the original image is noisy which makes it necessary to eliminate the noise and fill-in the blank spaces (those without information). The basic idea of our algorithm is to complete these spaces which hold no information and eliminate noise (if exists) while preserving the edges, and the goal of this work is to recover the entire clean image u(x) from a given incomplete noisy image I(x) observed only outside of an inpainting domain D, performing the inpainting and denoising action simultaneously. The fill-in and the denoising procedures are carried out automatically, and are independent of the topology of the objects that compose the image or of the inpainting domain D, which could be a union of several disconnected regions. In Section 2, we briefly describe the inpainting scheme given in [6,7] and the Mean Curvature Flow model. The proposed model for inpainting and denoising is described in Section 3. The Euler discretization of the proposed model and the numerical implementation are discussed in Section 4 as well as showing the application of our model in various examples, including denoising and restoring scratched pictures, texts and object removal from images. A note emphasizing the necessity of the diffusion process is presented
62
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
in Section 5 and the concluding remarks are presented in Section 6. Throughout the paper, U denotes the entire domain of the image, D the inpainting domain, DC the available part of the image I on U and u the restored image. 2. Image inpainting and image denoising Inpainting is the art of rebuilding the basics of visual art and consists of filling-in unknown data in a known region of an image, with the principal objective of restoring harmony to the given damaged picture with parts worn by time, overexposed objects or objects we want to remove from the image. To fill-in the inpainting domain, in a given picture, professionals follow these basic rules: (1) Observe the image as a whole. (2) Paint the inpainting domain in the same tones found in the surrounding background regions. (3) Using, as a basis, similar regions of the same image or known images of the same theme to carry out repairs in damaged regions and then add texture. In [6,7], Bertalmı´o et al. introduced the digital inpainting concept and proposed the Partial Differential Equations (PDE’s) model to simulate the digital work of the artisan’s art of inpainting. As the work of these artisans, is highly subjective, they limit the digital simulation to the automatization of the item (2), and for this, they proposed to transport the information from the surrounding inpainting domain into its interior always following the direction of the isophotes, arriving at vD. In order to make the transport of information follow this proposal, Bertalmio uses the inner product between two vectors, one indicates the inpainting’s normal direction (P(Du)) and the other indicates the isophote direction (Ptu). Following this procedure, the inpainting domain receives information and thus decreases. The goal of the inpainting model introduced in [6,7] is to transport as smoothly as possible (along the isophotes) the information from the surrounding inpainting domain. The transport of information is performed by solving the following PDE ut Z VðDuÞ$Vt u;
x 2D; tO 0
(1)
where t is a scale space parameter as in [2,3,12]. Note that this evolution equation runs only inside the region to be inpainted D. In Eq. (1), Ptu (Fig. 1a), is a vector which indicates the direction of u(x) variation of least intensity. The absolute value of Ptu is numerically equal to the instant variation rate of u(x) along the isophotes. The vector P(Du) indicates the direction where the value of u(x) varies abruptly (indicating a border or edge) and its absolute value is numerically equal to the Laplacian instant variation rate in this direction (Fig. 1b). The inner product of these two vectors (Fig. 1c) gives us the value to be transported to the pixel in question. We observe that in the case where the two vectors are perpendicular the transport will not be carried out. In this case, we have a strong
Fig. 1. (a) The perpendicular vector to Pu, (b) the P(Du) vector and (c) the two vectors showing where the inpainting will be performed, one notices that from the lowest lying pixel there will not be any transport of information once the vector P(Du) and Ptu are perpendicular.
indication that the pixel in question is localized on a border. In other cases where the transport does not happen P(Du), or Ptu, is a void vector. For more details of this approach, see [6,7]. The inpainting method proposed in [6,7] consists of intercalating the Eq. (1) with a diffusion equation whose objective is ‘to ensure a correct evolution of the direction field’ [6]. They used the following anisotropic diffusion equation ut Z g3 KjVuj;
x 2D3
(2)
where D3 is a dilation of D with radius 3, K is the Euclidean curvature of u, and g3 is a smooth function in D3 such that g3Z0 in vD3 and g3Z1 in D3.1 We will refer to this model as the BSCB model. In this paper, we propose a modification of the BSCB model, which consists of using different diffusion processes at points whether they belong to the inpainting domain or not. The equation used in DC makes a differentiated diffusion at points whether or not they belong to the edges of the objects that compose the image. In this way, the noise will be eliminated and the edges preserved. Such an equation enriches the inpainting process eliminating any eventual noise in the image, at the same time a correct evolution is possible in the direction of the transportation and stabilizes the process, eliminating the necessity to carry out a diffusion in D3 which in practice creates the ‘frame effect’, this means around the vicinity of the inpainting domain, within the dilation of D, the data is smoothed. Several PDE-based techniques have been proposed for the smoothing of an image, some obtained on the direct derivation of the evolution equations, and others from energy approaches such as the L2 norm dependent model or from the total variation model of Rudin, Osher and Fatemi [16]. Here, we will smooth the data inside the inpainting domain using the Mean Curvature Flow (MCF) equation (see [1,2] for further details), which is given by ut Z jVujdiv
Vu : jVuj
This equation diffuses u in the direction orthogonal to its gradient Pu and does not diffuse in the direction of Pu. The level sets of the solution of the MCF equation move in the 1 In the numerical implementations, Eq. (2) was not implemented in D3, a non-linear scaling was used to stabilize the discrete equation (see [6], p. 66).
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
63
Fig. 2. Top to bottom, left to right: Lenna image degraded by black and white spots and by blank region (flower format), inpainting domain, three intermediate steps and final results of the proposed model.
64
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
Fig. 3. (a) Degraded image with Poisson noise and a blank domain. (b and c) Partial results and (d) final result of proposed model.
Fig. 5. (a) Degraded image corrupted Gaussian noise (0 dB) and a blank domain. (b and c) Partial results and (d) final result of proposed model.
normal direction with a speed proportional to their mean curvature.
The proposed model in this work will deal with two tasks: denoising and inpainting. Our problem is to reconstruct a damaged image or to remove an object from an image. Our proposed method will inpaint the domain D, which has scratches or missing patches and will denoise the image in DC. Both tasks will be performed simultaneously. The goal of this work is to develop the idea of removing noise without losing the boundaries or edges and inpaint the
damaged regions. Our approach presents different procedures inside and outside of the inpainting domain. The inpainting process consists of filling-in the missing information in the empty domain D, based upon the available image information found outside of the domain D as in [7,8], and will be performed only inside the inpaint domain D while the smoothing procedure will be performed in all x2U but will act differently depending on if point x belongs, or not, to the inpainting domain D. Inpainting. The algorithm will execute, for each x belonging to the inpainting domain D, the transportation of neighboring pixel information belonging to D C, where the image information is available, using the Eq. (1) as in [7].
Fig. 4. (a) Degraded image with Gaussian noise (10 dB) and a blank domain. (b and c) Partial results and (d) final result of proposed model obtained after 100 iterations and using as parameter kZ0.05, DtiZ0.1, DtsZ0.2 and AZBZ10.
Fig. 6. (a) Degraded image by Gaussian noise (0 dB) and by a blank domain. (b and c) Partial results and (d) final result of proposed model.
3. The proposed scheme
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
65
Fig. 7. Plots of the 80th line of Fig. 6. Left to right: degraded image, partial and final results.
Smoothing. A smoothing procedure will be applied for each x belonging to the domain U. If x2D, the smoothed u(x) will be the solution of the Mean Curvature Flow equation, otherwise, i.e. if x lies outside of the inpainting domain a selective diffusion equation will be used. The smoothed version u(x) will be the solution to the following parabolic equation Vu ut Z gjVujdiv Kð1KgÞðuKIÞ; x 2Dc ; tO 0 (3) jVuj
Vu ut Z jVujdiv ; jVuj
x 2D
uðx;0Þ Z IðxÞ; x 2U vu Z 0; x 2vU; vt vU!RC
The non-linear space scale scheme (3), (5) and (6) for data smoothing was introduced in [4] following the ideas given in [2,3,12,15], with the objective of eliminating unnecessary parameters, where in many cases they are correlated. This equation allows one to perform selective smoothing in accordance with the value of the image gradient at each point x. We would also like to mention some other interesting works on inpainting. In [10,13], the authors present a technique for removing occlusion solving the following variational formulation
(4) (5)
tO 0
(6)
where gZg(jPuj). I(x, y) is an image to be processed, u(x, y, t) is its smoothed version on the scale t. The function g(s)R0 is a non-increasing function, satisfying g(0)Z1 and g(s)/0 when s/N. The term jVujdivðVu=jVujÞZ DuKV2 uðVu;VuÞ=jVuj2 diffuses u in the orthogonal direction to its gradient Pu and does not diffuse it in any other direction. The goal is to allow smoothing in the image u in a way that it is performed on both sides of an edge with minimal smoothing on the edge itself. The term g(jPuj) is used for edge detection and controls the diffusion speed: if Pu has a small value at the point x, it will be considered an interior point, and the diffusion will be stronger; on the other hand, that is, if Pu has a large value at the point x, then this point x will be considered an edge point, and the diffusion will be low since g(s) always assumes values for large values of s. The balance between the forcing term and the diffusion term is made by (1Kg), which works as a moderate selector of the diffusion process. Thus, the proposed model consists of selectively applying the Alvarez, Lions and Morel model’s [2] in areas of the image that demand a larger suavization. This model also consists of forcing, in an incisive way, the smoothed image u to remain close to the initial image I in the boundary areas which have gw0. On the other hand, in homogeneous areas gw1, and therefore, the forcing term will have an inexpressive effect, which allows for a better suavization of the image.
Fig. 8. Top to bottom, left to right: degraded image by characters, inpainting domain, three intermediate steps and final results of the proposed model. Parameters: 50 iterations, AZ10, BZ1, kZ3, DtiZ0.1 and DtsZ0.2.
66
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
Vu p EðuÞ Z jVuj a C bdiv ; pR 1;aO 0;bO 0; jVuj ð
(7)
D
to perform disocclusion and the practical algorithm for disocclusion, connecting the T-junctions associated with the same gray level using elastica minimizing curves. In [13,14], the authors used dynamic programming to solve the variational approach numerically. As reported by the authors, the regions to be inpainted are limited to having simple topologies, e.g. holes are not possible taking into account the specific numerical technique proposed. In [10], the authors work on the minimization of the elastica energy functional (7) obtaining the Euler–Lagrange equation for this functional. The choice of values for constants a and b is an issue of numerical implementation. There has been no investigation about the range of these two constants, which permits convergence and/or stability of the numerical algorithm. 4. Numerical approximations and experimental results
Fig. 9. Reconstruction of the shadow area after the removal of the bag. From top to bottom and left to right: the initial image, the inpainting domain, partial and final results.
In this section, we present the results by applying the proposed model on images of varying difficulty. Our tested images are represented by 256!256 matrices of intensity values, where each matrix element ui,j is a real value correspondent to the grayscale level of the image u(x, y) at the point xZxiZiDx and yZyjZjDy. We denote u(xi, yj, tn) by uni;j , where tnZnDt. The derivative of u in relation to the time t (space scale parameter), that is, ut calculated in (xi, yj, tn) is approximated by n Euler’s method, i.e. ut wðunC1 i;j Kui;j Þ=Dt. Numerical implementation using central difference techniques works quite well in the approximation of the diffusion term u2x uyy K2ux uy uxy C u2y uxx Vu jVuj div ; Z jVuj u2x C u2y and the transport term VðDuÞ$Vt u Z ux ðuxxy C uyyy ÞKuy ðuxxx C uxyy Þ: Using Neumann’s boundary conditions we calculate unC1 ij ; nZ 1;2;.;N, by the discretization of the Eqs. (1), (3) and (4) vijnC1 Z unij C Dti Li ðunij Þ
ðx;yÞ 2D
vijnC1 Z unij (8)
ðx;yÞ 2DC unC1 Z vnC1 C Dts Ls ðvijnC1 Þ ij ij
Fig. 10. Landscape reconstruction: (a) the initial image; (b) the inpainting domain; (c) partial result; and (d) the result obtained by the proposed model running 300 iterations and parameters kZ0.5, AZBZ10, DtiZ0.02 and DtsZ0.2.
ðx;yÞ 2U
(9)
with u0ij Z Iðxi ;yi Þ, Li ðuÞZ VðDuÞ$Vt u and Vv Ls ðvÞ Z gjVvjdiv Kð1KgÞðvKIÞ; if ðx;yÞ 2DC ; jVvj Vv or Ls ðvÞ Z jVvjdiv if ðx;yÞ 2D: jVvj here, gZ gðsÞZ 1=ð1C ks2 Þ, where k is a constant and sZjPvj.
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
67
Fig. 11. (a) Initial image, (b) inpainting domain, (c–e) partial results and (f) final result obtained by the proposed model running 200 iterations using AZ3, BZ10, kZ1, DtiZ0.02 and DtsZ0.2; (g) final result obtained by the proposed model running using different parameters; (h and i) results obtained by the BSCB model using two different choice to parameter values.
As in the BSCB model, inside every inpainting domain each restoration loop will perform A steps of transport with (1) intercalated with B steps of data diffusion with (2). The following results illustrate the performance obtained by the proposed model. We performed inpainting on five typical experiments and as the principal contribution of the proposed model is the case where the initial image I(x) was damaged by noise and also has a region with lost or no information data, the first experiment presents five images corrupted by a blank domain and noise. Lenna image corrupted by black and white spots; a synthetic image corrupted by Poisson noise; the same synthetic image corrupted by Gaussian noise with signal-tonoise ratio (i.e. the ratio between the standard deviation of a noiseless image and the standard deviation of the noise) being 10 and 0 dB; another synthetic image with different structures and corrupted by Gaussian noise with SNRZ0 dB. We can see in the Fig. 2, the perfect reconstruction by the proposed model in the Lenna image degraded by black and white spots and a flower superimposed. Fig. 3 shows a synthetic image with Poisson noise and the region to be inpainted in white, the partial and the final results. Hundred iterations were performed with AZ5 steps for transport and BZ10 for diffusion and the parameters used are: kZ0.05, DtiZ0.02 and DtsZ0.2. In Fig. 4, one can see the results of the proposed model used to restore the degraded synthetic image with Gaussian noise SNRZ10 dB. In Fig. 5, 100 iterations were run, each with AZ5 iterations of transport and BZ10 iterations of diffusion. The parameters used were: DtiZ0.02, DtsZ0.2 and kZ0.004. In Fig. 6 again, 100 iterations were run, each one with AZ BZ10 iterations. As one can observe, the final result is an
image free of noise without loss of edges and with damaged parts restored. The parameters used were: DtiZ0.1, DtsZ0.2 and kZ0.05. One can easily observe in the plots of the 80th Fig. 6 lines (Fig. 7), that it was totally reconstructed eliminating the noise and inpainting the blank pixels belonging to the image (columns 50–53, 191–207 and 217–222). The second experiment the text removal from an image is shown. Fig. 8 shows the restoration of a peppers image with a superimposed text. Several intermediate steps of the reconstruction using the proposed model are shown. The third experiment shows the special effect of removing an object from a real scene. Fig. 9 shows the total removal of an object. The bag on the floor was removed as desired. This result gives us a situation where the Connectivity Principle of human perception—‘humans mostly prefer having the two disjoint parts connected, even when they are far apart’ [9,11] is satisfied, where the shadow on the floor is properly reconstructed. AZBZ10 (100 iterations). In the fourth experiment (Fig. 10), we have the performance of the proposed model applied in a textured natural image. In (a), (b), (c) and (d) we have: the natural damaged image, the inpainting domain, the partial and the final results, respectively. After 300 iterations, we can see the perfect reconstruction on the left hand part of the inpainting domain where the level line between the sky and the land (horizontal line) is strong, however, on the right hand part of the inpainting domain, we can see that the textured mountain was not very well restored. The last experiment (Fig. 11) shows that the choice for the constant’s value A and B of steps for data transportation which should be intercalated with the steps for data diffusion,
68
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69
respectively, is very important. In Fig. 11f, one can see that the result carried out by our proposed model running 1000 iterations using AZ3 and BZ10 is quite good, managing to fill-in the cat and the tower simultaneously, while in Fig. 11g with another choice for A and B we achieved good results in the cat but the tower was not restored. Similar results are obtained with the BSCB model as we can see in Fig. 11h, where the tower was not restored and in Fig. 11i, where the perfect reconstruction was obtained with different values for the constants A and B. The choice of the parameter k, and the number of iterations depends on the amount of noise presented in the image and the size of the inpainting domain. In general they are chosen as being those that give better results from a visual perception point of view. Following the ideas developed in [5] we are working on the optimization of the choice of those parameters seeking better results with smaller computational effort. The computational code was written in C language. The results were obtained by using a Pentium III PC (512 MB RAM–1.2 GHz). The running time for an image of 256!256 pixels size is about 15 s for each 100 iterations with AZ10 and BZ5 and less than 2 min in all tested images. The temporal and spacial algorithm complexities are linear with the size of the data input. 5. The necessity of the diffusion process The BSCB model as well as the proposed model work combining A steps of the transport process with B steps of the diffusion process. Bertalmio, in his doctoral thesis [6] justified the need for the diffusion process to be intercalated with the transport process to ensure ‘the correct evolution of the directional field’. However, one can observe by applying the transport equation without diffusion the iterative process generated by the Euler discretization of the Eq. (1) can be nonconvergence as can be easily seen considering, for example, the initial image u0 as the function u0(x, y)Zx3Cy. The iterative process un Z unK1 C Vt unK1 $VðDunK1 Þ;
V uk $VðDuk Þ ZK6;
c k 2N;
we have that un Z unK1 C Vt unK1 $VðDunK1 Þ Z x3 C yK6n;
Acknowledgements This work is partially supported by CNPq grants.
References
generated by the transport equation is divergence. In fact, as t
denoising. The restoration is carried out by the adapted action of equations, which perform diffusion and transport. The model is a combination of three equations, one to perform the transport of the data and two to perform the smoothing inside and outside of the inpainting domain D. The results presented in our examples demonstrate the high performance of the proposed model which has demonstrated great efficiency in dealing with the inpainting of damaged images and denoising when the initial image is noisy. The nature of the noise corruption and the topology of the inpainting domain are not an issue, we have tested images corrupted by Poisson noise, Gaussian noise and black and white spots and as well as using different forms for inpaint domain D in other situations all showing satisfactory results. The size of the inpainting domain can interfere with the results. The drawback of the proposed model can be seen when we analyze the image, which is characterized by texture, a great part of which is eliminated as one can see in Fig. 10, because the inpainting domain is quite large. The proposed model and the BSCB model work in the same way and produce analogous results when applied in non-noisy images, being the first more robust as it has a wider application allowing for the restoration of an image corrupted by noise as well as with domains with no useful data. The outside smoothing of the inpainting domain gives one the stabilization of the numerical technique and there are no frame effects. From the numerical point of view, the number of intercalated steps of inpainting and diffusion should be investigated mainly when the image to be restored has several different inpainting domains. Some choices for these parameters can produce partial reconstruction as we can see in Fig. 11g and h.
c n 2N
which goes to infinity when n goes to infinity. 6. Concluding remarks In this paper, we present a mathematical model for image restoration, which consists of the recovery of missing parts in an image and also eliminating the corruption caused by noise. The model presented in this work was inspired on the BSCB model for inpainting and a balanced diffusion equation for
[1] L. Alvarez, J.M. Morel, Formalization and computational aspects of image analysis, Acta Numer. (1994) 1–61. [2] L. Alvarez, P.L. Lions, J.M. Morel, Image selective smoothing and edge detection by nonlinear diffusion, SIAM J. Numer. Anal. 29 (3) (1992) 845–866. [3] C.A.Z. Barcelos, Edge-preserving regularization in image restoration, Boletim (SBMAC) (in press). [4] C.A.Z. Barcelos, M. Boaventura, E.C. Silva Jr., A well-balanced flow equation for noise removal and edge detection, IEEE Trans. Image Process. 12 (7) (2003) 751–763. [5] C.A.Z. Barcelos, M. Boaventura, E.C. Silva Jr., Edge Detection and Noise Removal by use of a Partial Differential Equation with Automatic Selection of Parameters, Comp. Appl. Math., 24(1), pp. 131–150, 2005. [6] M. Bertalmio, Processing of Flat and non-Flat Image Information on Arbitrary Manifolds Using Partial Differential Equations, Thesis, University of Minnesota, 2001. [7] M. Bertalmio, G. Sapiro, V. Caselles, C. Ballester, Image inpainting, Computer Graphics, SIGGRAPH 2000, 2000, pp. 417–424.
C.A.Z. Barcelos, M.A. Batista / Image and Vision Computing 25 (2007) 61–69 [8] T. Chan, J. Shen, Local inpainting models and TV inpainting, SIAM J. Appl. Math. 62 (3) (2001) 1019–1043. [9] T. Chan, J. Shen, Non-texture inpainting by curvature-driven diffusions (CDD), J. Vis. Commun. Image Rep. 12 (4) (2001) 436–449. [10] T. Chan, S. Kang, J. Shen, Euler’s elastica and curvature-based inpainting, SIAM J. Appl. Math. 63 (2) (2002) 564–592. [11] G. Kanizsa, Organization in Vision, Praeger, New York, 1979. [12] J. Malik, P. Perona, Scale-space and edge detection using anisotropic diffusion, IEEE TPAMI 12 (7) (1990) 629–639.
69
[13] S. Masnou and J.M. Morel, Level lines based disocclusion, IEEE Int. Conf. Image Processing, v.3, pp. 259–263, 1998. [14] S. Masnou, Disocclusion: a variational approach using level lines, IEEE Trans. Image Process. 11 (2) (2002) 68–76. [15] K.N. Nordstro¨m, Biased anisotropic diffusion: a unified regularization and diffusion approach to edge detection, Image Vis. Comput. 8 (1990) 318– 327. [16] L. Rudin, S. Osher, E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D 60 (1992) 259–268.
