Color Correction for the Virtual Recomposition of Fragmented Frescos F. Renna, G. Carlomagno, N. Mosca, G. Attolico, A. Distante Istituto di Studi sui Sistemi Intelligenti per l’Automazione Via Amendola, 122/D-O – 70126 – Bari Tel. +39 080 5929441 – Fax. +39 080 5929460 E-Mail: [renna,carlomagno,nmosca,attolico,distante]@ba.issia.cnr.it
Abstract The paper describes the solution to the color correction problem used in the digital system we have designed for the reconstruction of the S. Matthew fresco, painted by Cimabue for the Upper Church of S. Francis in Assisi. The characteristics of the problem make difficult to evaluate correspondences between colors and require specific corrections to be applied to different parts of the fresco. The obtained improvement in terms of color similarity and retrieval results is shown.
rotations and translations across the region of interest using a special mouse. On the right monitor a virtual container used for organizing the fragments is shown. It is the digital counterpart of the boxes used in the real lab to collect logically related fragments.
1. Introduction This work describes part of a system for the virtual aided recomposition of fragmented frescos. The interest in such a system arises from the need of recomposing the S. Matthew fresco, painted by Cimabue for the Upper Church of S. Francis in Assisi and broken into more than 140.000 fragments by the earthquake in September 1997. The large extension of the fresco (about 35 squared meters), the huge number of fragments and the particular technique used by Cimabue (that makes the pictorial film very sensitive to the physical manipulation required by the traditional recomposition) have suggested the application of digital tools to this challenging problem. Moreover, fragments do not cover the whole fresco and could partially belong to a neighbour fresco broken during the same event; in addition their contours do not always match exactly. The designed system transposes the traditional recomposition process in a digital way, by offering to the operators the central and critical role of managing and applying new tools and flexible algorithms of image analysis to increase the efficacy of their work [1]. The multi-monitor graphical station (fig. 1) allows the selection of a part of the image of the whole fresco (visualized in a scaled version on the central monitor) as background for the working area that is shown at full resolution on the left-side monitor. The operator looks for the best place for each fragment by applying simultaneous
Figure 1. The developed workstation for the virtual aided recomposition of fragmented frescos. A fundamental improvement to the recomposition process is the support to the retrieval of fragments of interest from the database (digital images of every single fragment) using a query-by-example modality that is incremental and iterative. The operator picks-up a set of examples (images of fragments or details of the reference image) and the system selects in the database the fragments more similar to them. If the results are not satisfying, the set can be modified by adding, removing or changing the examples: the process can be repeated until the operator’s needs are fulfilled. Color and texture are the most important components of similarity evaluation [2], especially when shapes, damaged during the fragmentation process, do not necessarily match perfectly. Unfortunately, the colors of the reference image, a picture taken several years before the earthquake under unknown illumination conditions, are very different from those of the real fragments. Moreover, the fresco was painted on a vaulted ceiling, inducing different lighting conditions (and appearance of colors) in different areas of
0-7695-2128-2/04 $20.00 (C) 2004 IEEE
the fresco. Furthermore, after this acquisition numerous causes may have caused different deteriorations across the large surface of the fresco. In order to indexing the database using details of the reference image, it is necessary to correct colors of the whole fresco image. Moreover, this simplify the work of the restorers when, to correctly place the fragments returned from the system onto the working area, they compare the fragment image with the reference image.
2. Color correction The aim is to evaluate a transformation matrix that allows the roto-translation of the RGB components of the reference image so that they match the corresponding colors of the fragments. The imposed system is:
r a11 a12 g a21 a22 b = a a32 31 1 0 0
a13 a23 a33 0
a14 r0 a24 g 0 a34 b0 1 1
The ro bo go values of each color of the reference image are multiplied by the transformation matrix to obtain the r g b components of the corresponding color of fragments. To calculate the unknown twelve elements ( aij ) of the roto-translation matrix, it is necessary to solve:
Ax = b where the matrix A holds the RGB components of colors of reference image, the matrix x is the unknown transformation and the b vector holds the RGB components of the fragments colors. To solve the system four colors at least (3x4) are needed; to obtain a more reliable solution an overdetemined set of m linear equations, solved using the least-squares method [3], has been used, where m is three times the number of colors considered. The solution x minimizes the distance: Ax − b
2
where A ∈ ℜ mxn and b ∈ ℜ m
The algorithm we have used, proposed in [4], solves Ax = b using the singular value decomposition technique. The RGB components in A and b are the mean color of patches, mostly monochromatic, extracted by hand from both the reference image and the corresponding areas of fragments. After its evaluation, the roto-translation matrix cannot correct the whole reference image but only parts of it having almost the same chromatic characteristics.
3. Experimental results As an example, in this work we compare the results of
two of the found correction matrices:
0.621915 0.040462 − 0.112838 44.119537 X 1 = − 0.157467 0.679967 − 0.082440 55.964485 − 0.089917 0.214128 0.255315 53.000317 0.503821 0.110221 − 0.036876 51.439159 X 2 = − 0.101329 0.793839 − 0.226638 48.415569 − 0.070683 0.199633 0.304800 47.469845 These matrices increase the color similarity between the reference image and the real fragments. Moreover, they allow the retrieval of fragments from the database using details extracted from the corrected reference image obtaining very consistent results. Table 1 shows the mean RGB components of some of the 47 patches used to calculate the matrix X1 and their Euclidean distances: RNCGNCBNC refer to the not corrected reference image; RCGCBC refer to the corrected reference image; RFGFBF refer to the corresponding fragments; RGBNC-RGBF (RGBC-RGBF) are Euclidean distances between the mean value of colors of not corrected (corrected) patches of the reference image and the corresponding parts of fragments. Correction generally increases color similarity between the reference image and the corresponding fragments, improving the perception of restorers, apart from some exceptions due to the least mean square method (figure 2). RNC GNC BNC RC GC BC RF GF BF RGBNC-RGBF RGBC-RGBF 183 147 91 151117 89 152119 95 42.06 6.29 147 103 43 132 97 71 138109 85 43.56 19.34 … 189 163 136 151123103152123 99 66.17 3.62 164 134 84 140112 86 141116 89 28.71 5.64 206 189 161 159136113161143 112 80.51 7.82 30 1 3 61 49 48 38 21 20 27.51 45.31 157 93 67 138 87 73 151105 92 28.58 29.46 205 203 195 155143125156146 131 98.8 7.08 213 212 209 160146129162151 134 108.78 7.76 213 212 208 159147129159150 136 109.24 7.76
Table 1. Some colors extracted from the patches used to evaluate the color correction matrices.
The image on the left in figure 3 shows a patch extracted from the reference image. The central image shows the coverage (grey pixels) of this not corrected patch by the color palette including only the colors detected in the whole collection of fragments. The image on the right shows the coverage of the same patch corrected using the transformation matrix X1. The white pixels correspond to colors absent in the color palette.
0-7695-2128-2/04 $20.00 (C) 2004 IEEE
reference image while in the third one the palette comes from the corrected patch. The almost total coverage of the fragments show that the use of details of the reference image after the color correction process can really make meaningful the query to the database, because the two color spaces have been made close enough.
Figure 2. Euclidean distances between colors of patches from the reference image and of fragments before (dark) and after (light) the color correction.
Detail 1 Not corrected Corrected (X1) Detail 2 Not corrected Corrected (X1) Detail 3 Not corrected Corrected (X1) Detail 4 Not corrected Corrected (X1) Detail 5 Not corrected Corrected (X2)
W [%] 79.73 5.82
D.G.[%] 20.27 94.18
75.04 6.32
24.96 93.68
36.93 0.13
63.07 99.87
74.87 7.86
25.13 92.14
82.53 8.07
17.47 91.93
Table 2. Color correction improves the percentage of colors of the reference image present in the palette extracted from the collection of fragments.
Figure 3. Grey pixels show the colors, in the patch on the left, present in the palette defined by the whole collection of fragments before (center) and after (right) the color correction.
Table 2 quantifies this coverage for five patches extracted from different areas of the reference image and corrected using X1 and X2 transformation matrices. The values have been computed dividing the number of white (respectively dark grey) pixels by the total number of image pixels. The correction always induces a coverage higher than 90%. The first image of figure 4 is a real fragment. The second and the third images use only three colors: White for pixels (very few) whose color is present neither in the palette of the corresponding patch of the reference image nor in the palette of fragments; Dark Grey for pixels whose color is present in the palette extracted from the corresponding patch of the reference image; Light Grey for pixels whose color is not present in the palette extracted from the corresponding patch of the reference image. In the second image, the palette has been extracted from the corresponding not corrected patch of the
Figure 4. Dark (light) grey pixels have their color present (absent) in the palette extracted from the corresponding part of the reference image before (center) and after (right) the color correction.
Table 3 quantifies this result using percentages computed as in table 2. In this case the number of white, dark grey and light grey pixels have been divided by the number of pixels belonging only to the fragment (not to the black background). Again the calculated transformation matrices improve color similarity making meaningful the use of details from the reference image for indexing the database. The advantages of color correction on fragments retrieval from the database are evident also comparing the results of
0-7695-2128-2/04 $20.00 (C) 2004 IEEE
searches carried out from a detail of the corrected reference image with those obtained using a real fragment as example, assumed as reference. Table 4 shows the number of fragments, returned by the system starting from a patch extracted from the reference image respectively corrected (column 2) and not corrected (column 3), that are present in the list returned using the real fragment as example. Fragment 1 Not corrected detail Corrected detail (X1) Fragment 2 Not corrected detail Corrected detail (X1) Fragment 3 Not corrected detail Corrected detail (X1) Fragment 4 Not corrected detail Corrected detail (X1) Fragment 5 Not corrected detail Corrected detail (X2)
W [%] 0.24 0.24
D.G.[%] 52.9 82.73
L.G.[%] 46.85 17.03
0.33 0.33
55.58 84.4
44.09 15.27
0.12 0.12
88.78 88.79
11.1 11.09
0.28 0.28
57.09 94.68
42.62 5.03
0.78 0.78
78.94 83.19
20.28 16.03
Table 3. Color correction improves the percentage of colors of fragments present in the palette extracted from corresponding areas of the reference image.
Detail 1 Detail 2 Detail 3 Detail 4 Detail 5
Corrected 224 46 85 121 99
Not Corrected 26 9 0 109 52
Table 4. Number of valid fragments, retrieved using corrected and not corrected details of the reference image. The validity of a fragment is judged by its presence in the results of a query based on the real fragment corresponding to the detail at hand.
At the moment, the system returns, in decreasing order, the 500 fragments of the database that are the most similar to the given example in terms of color. Color similarity is measured using algorithms, based on a modified version of the “histogram intersection” technique [5], working on color histograms computed on the example image and on each candidate fragment in the database. The comparison generates a similarity value in the range [0;1] that is related to the candidate fragment with respect to the current sample images. Table 4 emphasizes that color correction improves
the similarity between the palettes of the reference image and of the whole collection of fragments, a crucial requirement for satisfactory performance of the system.
4. Conclusions Color correction of the reference image with respect to fragments inside a system for virtual aided recomposition of fragmented frescos has been modelled as a roto-translation in the RGB space. The obtained roto-translation matrices, evaluated using the least squares method applied to the RGB components of the almost monochromatic patches extracted from the reference image and from real fragments, are very useful for the effectiveness of the system. The increased similarity of chromatic characteristics between reference image and fragments makes easier the visual perception of the operators that must find the right position of fragments. Moreover it makes more meaningful the retrieval process performed starting from details extracted from the corrected reference image: the results show a larger number of returned fragments that are really related to the example of interest.
5. Acknowledgments Thanks to Marco Malavasi (Dip. Rapporti con le Regioni – CNR) for its important support and to Giuseppe Basile, Lidia Rissotto, Angelo Rubino (ICR), Manuela Viscontini (Università della Tuscia) and Laura Cacchi for many valuable discussions during the development of the project.
6. References [1] G. Attolico, A. Distante, “Aided virtual recomposition of fragmented frescos”, Dall’utopia alla realtà, Note dal cantiere dei frammenti della Basilica Superiore di S. Francesco in Assisi, n. 2, April 2001. [2] A. Branca, G. Attolico, S. Tornesello, A.Distante, “Fragment similarity estimation for the restoration of frescos”, Proceeding of the Irish Machine Vision & Image Processing Conference, September 2001, Maynooth. [3] G.H. Golub, C.F. Van Loan, Matrix Computations, The Johns Hopkins University Press, 1996. [4] W.H. Press, S.A. Teukolsky, W.T. Vetterling, B.P. Flannery, Numerical Recipes in C, Cambridge University Press. [5] M.J. Swain, D.H. Ballard, “Color Indexing”, International Journal of Computer Vision, 7, N.1, pp. 11-32, 1991.
0-7695-2128-2/04 $20.00 (C) 2004 IEEE