A METHOD FOR REASSEMBLING FRAGMENTS ... - Semantic Scholar
A METHOD FOR REASSEMBLING FRAGMENTS IN IMAGE RECONSTRUCTION Francesco Amigoni, Stefano Gazzani, Simone Podico Artificial Intelligence and Robotics Laboratory; Dipartimento di Elettronica e Informazione Politecnico di Milano; Piazza Leonardo da Vinci 32, 20133 Milano, Italy ABSTRACT The reassembly of fragments to reconstruct images and objects is a problem often encountered in several applications, ranging from archeology to medicine. In this paper we present a novel method for reassembling two 2D fragments. The proposed method is based on the information extracted from the outlines and from the color contents of the fragments, without relying on any knowledge of the final image. The techniques illustrated in this paper constitute the core of a more general method for reassembling n fragments we are developing. However, here we concentrate mainly on the description of the algorithms for matching two fragments and on their experimental analysis and validation. 1. INTRODUCTION The reassembly of fragments to recompose images and objects is a problem occurring in a number of fields: in archeology [1], in art restoration [2], and in other disciplines including forensics, computer-aided design, chemistry, and medicine [3]. In this paper we present a novel method for reassembling two 2D fragments. We give only few preliminary considerations on the more general problem (on which we are currently working) of integrating n fragments to reconstruct a complete image. The outline of the proposed method for reassembling two fragments follows. (1) Analysis of the two fragments and extraction of their representations. A fragment is represented by the sequence of its border pixels with associated information about coordinates, color, and local curvature. (2) Comparison of the two sequences to identify the common subsequences (with similar local curvature and satisfying constraints on color contents). The common subsequences represent the candidate matching portions of the borders. (3) Calculation of the transformation (translation and rotation) that brings the first fragment to join the second one along the best (longest) matching portion of the borders. This paper is structured as follows. In Section 2 we detail our method for reassembling two fragments of a 2D image. Section 3 shows some of the results obtained from the experimental analysis that validated the proposed method. In Section 4 we critically compare our method with other similar techniques. Finally, Section 5 concludes the paper. This paper is dedicated to the memory of Marco Somalvico, who has been both teacher and friend.
2. REASSEMBLY OF TWO FRAGMENTS 2.1. Fragments Analysis In this subsection we describe the step (1) above. The input to our method is represented by two rectangular bitmap images of the fragments (having closed boundary curves) over a background. The method requires two images with the same scale; this means that they must be digitally acquired from the same distance and viewing angle. Given the two images, we preliminary separate the fragments from the background: we use a floodfill based algorithm [4] that labels every pixel as fragment pixel or background pixel. The pixels that compose the border of a fragment are then located as those belonging to the fragment that are adjacent to a background pixel. We employ a simple algorithm that clockwise follows the border of the fragment by moving from one border pixel to another: the next border pixel is located by searching among the eight neighbors of the current border pixel. The result is a border array containing the coordinates and the color of every pixel belonging to the fragment’s border. To obtain a rotation- and translation-independent model of the border, we use the local curvature (as in [5] and [6]). The local curvature of the border in a pixel p is calculated as the magnitude of the derivative of the local tangent in p. To make the computed curvature independent from the images resolution, we introduce the parameter D that represents the distance from p to the two neighboring border pixels used to calculate the local tangent in p. The local curvature of the border pixels of the first and the second fragment are stored in real-values curvature arrays, called Shape1 and Shape2, respectively. Since the number of pixels in two segments with the same physical length is different if the segments are not parallel, we use an adjusting algorithm that stretches the border and curvature arrays adding interpolated pixels depending on the slope of the local tangent. 2.2. Fragments Match In this subsection we describe the steps (2) and (3) above. In order to find the longest common portion of the borders, we developed an algorithm called tank algorithm, listed in Fig. 2, that represents the foundational part of our method and the main contribution of this paper. We make two preliminary considerations before describing the core of the tank algorithm. First, one of the two curvature arrays has to be flipped because the border pixels in both images have been found in clockwise order. Second, since the fragments have close boundary curves, we consider the curvature arrays as circular by duplicating them.
Fig. 1. Various placements of the arrays
The tank algorithm requires that Shape1 is longer than Shape2. The outer loop cycles the different placements of the curvature arrays, as shown in Fig. 1. Initially they are positioned as in A, then Shape2 is shifted by one position (B), and so on until the end of the loop (C). The number of evaluated placements is equal to the length l1 of Shape1. For each single placement, the curvature arrays are compared element by element (pixel by pixel) from left to right (the double arrows of Fig. 1). The total number of element comparisons for each placement is equal to the double of length l2 of Shape2. This means that the worst-case complexity of the tank algorithm (not considering the color test, see below) is O(l1 l2 ). For each element comparison the algorithm calculates the absolute value e of the difference of the two corresponding curvatures. When e is minor than the parameter ε, the curvatures are very similar and a common portion of the borders starts. When a common portion has started, the e computed for the following elements is added to ErrorTank (initially set to 0). When ErrorTank exceeds the parameter ErrorTankCapacity, the current common portion is ended. While examining a common portion, e is not only added to ErrorTank but also contributes to reduce the ErrorTank value in particular cases. We use two parameters called R1 and R2 . When e < ε the ErrorTank is reduced by being multiplied by R1 (0 ≤ R1 ≤ 1), otherwise if e < 4 · ε, ErrorTank is reduced by being multiplied with R2 (0 ≤ R1 ≤ R2 ≤ 1). Typical values for the two paraments are 0.5 for R1 (ErrorTank becomes the 50% of its old value) and 0.8 (ErrorTank becomes the 80% of its old value) for R2 . R1 and R2 give a sort of reward when the curvature of two pixels are almost equal (R1 ) or enough similar (R2 ). This makes the tank algorithm very robust to sporadic differences in the common portion of the borders. Finally a refinement of the common portion is done: the elements from the end are trimmed until the error e is less than ε (this is necessary because, at the end of the search for the common portion, the curvatures were different and ErrorTank overflowed). Once a common portion has been found, we perform a color test by applying a modified version of the above part of the tank algorithm to the border arrays containing information about the RGB color values of the border pixels. The variable e now is the Euclidean distance (in three-dimensional space RGB) between the points that represent the colors of the corresponding pixels (in the current common portion). If, during the color test, ErrorTank exceeds the ErrorTankCapacity the common portion is discarded due to incompatibility in color of adjacent pixels. Otherwise the common portion is accepted and, if it is the longest for the current placement, it is labelled with its length (in pixel) and with a weight that represents the color diversity, given by ErrorTank/ErrorTankCapacity.
Require: Length(Shape1)>Length(Shape2) Shape1Dup ← Duplicate(Shape1) Shape2Dup ← Duplicate(Flip(Shape2)) for i = 0 to Length(Shape1) do MaxCommonLength ← 0; Counting ← FALSE; ErrorTank ← 0 for j = 0 to Length(Shape2Dup) do e ← |Shape1[(i+j) mod Length(Shape1Dup)].Curvature− Shape2Dup[j].Curvature| {compute the error} if (e < ε) ∨ (Counting) then ErrorTank ← ErrorTank+e Counting ← TRUE if ErrorTank < ErrorTankCapacity then if e < ε then ErrorTank ← ErrorTank ·R1 else if e < 4 · ε then ErrorTank ← ErrorTank ·R2 end if {increase the common portion length} CommonLength ← CommonLength+1 else {ErrorTank overflowed} if CommonLength > MaxCommonLength then {refine the candidate common portion} CommonLength ← Trim(CommonPortion) end if if CommonLength > MaxCommonLength then if ColorTest=TRUE then {color check ok: new longest common portion found} MaxCommonLength ← CommonLength Result[i] ← CommonPortionData end if end if CommonLength ← 0; Counting ← FALSE; ErrorTank ← 0 end if end if end for end for
Fig. 2. The tank algorithm
The tank algorithm produces the array Result that contains information about the longest common portion found for each placement of the borders. At the top of Fig. 3 is shown the normalized graph of the length of the common portions for all possible placements of two fragments. For each common portion, call ϕ the angle by which the first fragment must be counterclockwise rotated to align with the second one along the common portion of the borders. This angle is calculated by comparing the slopes of the segments that join the start and end points of the common portion in the two fragments. The length of the common portions for each angle ϕ forms a new array of 360 elements (it is possible to have approximations more accurate than one degree by using more elements). An example of this (normalized) array is shown at the bottom of Fig. 3; it is possible to notice that the best rotation angle is 198◦ . The translation is simply calculated in order to superimpose the common portions of the borders. 3. EXPERIMENTAL RESULTS In this section we present some of the experimental results obtained from the implementation of the method described above. The first prototypes of the algorithms have been coded in Delphi to preliminary assess the effectiveness of the method and to analyze the role of the different parameters involved, working in a GUI environment that could be quickly modified. Given this non-
Fig. 6. Two fragments of a jigsaw puzzle taken from the Web site of the movie “Le fabuleux destin d’Am´elie Poulain” and the reassembled puzzle
Fig. 3. Length of the common portions for the placements (top) and for the rotation angles (bottom) of the fragments of Fig. 4
Fig. 4. Two fragments of an image of Piazza dei Miracoli at Pisa (Italy)
optimized implementation, we did not make a deep complexity analysis of our method. The integration of the two fragments of Fig. 4 is reported in Fig. 5. Although it represents only a particular application, our method has proven to be effective also in reassembling fragments of jigsaw puzzles as shown in Fig. 6. Fig. 7 shows an example in which our method fails. In this case, the two fragments clearly cannot be correctly matched and the method returns the best match. However, the result appears absolutely reasonable when the shape and the color content of the fragments are considered (see the clouds!). Only humans can recognize the erroneous reassembly because of the semantic content of the fragments. The values of the parameters used in the above reassembling examples have been: D = 10, ε = 10, R1 = 0.5, R2 = 0.8, and ErrorTankCapacity= 0.5l2 (remember that l2 is the length of the second, shorter, fragment). The time required (on a 300MHz Pentium II computer) for matching the previous fragments ranges from 1s to 5s. It is impossible to report here the complete experimental analysis on the role of parameters in our method, but some main facts are worth to be discussed. The tank algorithm correctly finds the
Fig. 5. The reassembled image of Pisa (Italy)
Fig. 7. Two fragments of an image of Pisa (Italy) that are not correctly reassembled
common portion of the borders even when R1 = R2 = 1 (namely, when the tank level is never reduced) given that ε is large enough. For example, when ε > 5 for the fragments of Fig. 4. Conversely, if the tank level can be reduced (0 ≤ R1 ≤ R2 < 1), then the correct common portion is found also for small ε. For example, for ε = 1, R1 = 0.5, and R2 = 0.8 for the fragments of Fig. 4. Experimentally, it results that the longest correct common portions of the borders (namely the best matches) are often found when 10 ≤ ε ≤ 30 and that, in these cases, the influence of the values of R1 and R2 is small. Finally, the value of D depends on the resolution (the larger the resolution the larger D) and on the irregularities in the borders. In conclusion, the proposed method for reassembling fragments has shown to be robust in producing the correct result also when some little pieces of the boundaries are missing. 4. RELATED WORKS A number of methods have been developed to reassembly fragments in order to recompose both two-dimensional images and three-dimensional objects. In the case of 2D images, when the final result of integration is not known, the only information available to the methods is obtained from the fragments themselves. Roughly, this information is related to the boundaries (edges or fracture lines) or to the content of the fragments (their colors or drawings). The method presented in this paper exploits information about both boundaries and color content of fragments. Another effec-
tive method but based only on boundary information is presented in [6]. The outlines of the fragments are represented by curvature strings (as in [5]), namely by a circular sequence of samples, where each sample is the local curvature of the outline at the corresponding point. This representation is used also in our method. The matches between fragments are found by a multiscale procedure; while the refinement of matches is performed by a dynamic programming algorithm. In our method we do not employ a multiscale procedure for matching the fragments and, in the current version, we match only two fragments; thus we cannot directly compare the two methods. Although the representation is similar, our method for matching the outlines of the fragments differs from the one presented in [5]. The latter manages noise by considering two curvature values as equal if there is at most an ²-difference between them (where ² is a “local” measure of accuracy), while the former considers a “global” measure of accuracy as implemented in the tank algorithm. Our method deals more robustly with noise since it allows scattered pairs of significantly different curvature points within a sequence of pairs of significantly equal curvature points. (Note that our method essentially reduces to that in [5] when R1 = R2 = 1.) The worst-case complexity of the method in [5] is O(n2 ) (where n is the length of the longer border representation), which is comparable to that of our method. Hori et al. [7] propose a method for detecting the joint among two 2D potsherds. Their method uses a segmentation of the contours of the fragments and, then, finds match between pairs of segments. Its applicability (and that of similar techniques) is limited to the cases in which the fragments possess critical points (i.e., corners) on which the segmentation can be based. Our method is, in this respect, more general and a direct comparison is difficult. The object recognition techniques (see [8] for a survey) can provide hints to represent the boundaries and to develop algorithms for their matching. However, the object recognition problem is structurally different from the fragment reassembly problem. In the former it is usually the case when an object is known (the model) and the goal is to find, by matching, the occurrences of the object in a scene. The fragment reassembly problem is more “symmetric”, since the fragments to be matched are not subject to any model-instance relation. Finally, although addressing a similar problem, the most effective techniques for automatic assembly of jigsaw puzzles (like [9]) cannot be directly applied to fragment reassembly since they usually heavily rely on the specific shape of jigsaw puzzle fragments from which invariants are extracted. 5. CONCLUSIONS In this paper we have presented some results of a novel method for reassembling two fragments of a 2D image. The method has proved to be effective and robust in several experimental settings. Since our goal was to assess the validity of the proposed approach, we made no attempt to optimize the algorithms but we aim to improve their performance by implementing a more efficient C++ version of the code. Another direction of future work is to employ the method presented in this paper to address the reassembly of n fragments. Some preliminary results are shown in Fig. 8. The reassembled images have been obtained by a quite simple algorithm: given a pool of fragments, we look for the two fragments with the best match (the longest common portion of borders), we add to the pool the new fragment obtained by joining the two fragments (which are removed from the pool), and we continue until a single
Fig. 8. Two images of the Villa Contarini-Camerini at Piazzola sul Brenta (Italy) reassembled from two sets of 5 (left) and 6 (right) fragments
fragment is left. We aim to improve this elementary method for reassembling n fragments by pre-classifying the fragments in the pool (for example according to their color contents) and to experimentally test the method in real-case applications. 6. REFERENCES [1] M. Levoy, “Forma urbis romae project,” 2003, http: //formaurbis.stanford.edu/, last accessed January 8th, 2003. [2] Guide to the Recovery, Recomposition, and Restoration of Shattered Wall Paintings: Experience Gained at the Basilica di San Francesco in Assisi, ICR, Rome, Italy, 2001. [3] G. Barequet and M. Sharir, “Partial surface and volume matching in three dimensions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 9, pp. 929– 948, September 1997. “Intro to computer [4] MIT Computer Graphics Group, graphics,” 2003, http://graphics.lcs.mit.edu/ classes/6.837/F01/Lecture03/index.html, last accessed January 8th, 2003. [5] H. Wolfson, “On curve matching,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 5, pp. 483–489, May 1990. [6] H. C. de Gama Leit˜ao and J. Stolfi, “A multiscale method for the reassembly of two-dimensional fragmented objects,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 9, pp. 1239–1251, September 2002. [7] K. Hori, M. Imai, and T. Ogasawara, “Joint detection for potsherds of broken earthenware,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 1999, vol. 2, pp. 440–445. [8] A. Pope, “Model-based object recognition: A survey of recent research,” Tech. Rep. TR-94-04, University of California at Berkeley, Berkeley, CA, USA, January 1994. [9] G. Budea and H. Wolfson, “Solving jigsaw puzzles by a robot,” IEEE Transactions on Robotics and Automation, vol. 5, no. 6, pp. 752–764, December 1989.
2. REASSEMBLY OF TWO FRAGMENTS 2.1. Fragments Analysis In this subsection we describe the step (1) above. The input to our method is represented by two rectangular bitmap images of the fragments (having closed boundary curves) over a background. The method requires two images with the same scale; this means that they must be digitally acquired from the same distance and viewing angle. Given the two images, we preliminary separate the fragments from the background: we use a floodfill based algorithm [4] that labels every pixel as fragment pixel or background pixel. The pixels that compose the border of a fragment are then located as those belonging to the fragment that are adjacent to a background pixel. We employ a simple algorithm that clockwise follows the border of the fragment by moving from one border pixel to another: the next border pixel is located by searching among the eight neighbors of the current border pixel. The result is a border array containing the coordinates and the color of every pixel belonging to the fragment’s border. To obtain a rotation- and translation-independent model of the border, we use the local curvature (as in [5] and [6]). The local curvature of the border in a pixel p is calculated as the magnitude of the derivative of the local tangent in p. To make the computed curvature independent from the images resolution, we introduce the parameter D that represents the distance from p to the two neighboring border pixels used to calculate the local tangent in p. The local curvature of the border pixels of the first and the second fragment are stored in real-values curvature arrays, called Shape1 and Shape2, respectively. Since the number of pixels in two segments with the same physical length is different if the segments are not parallel, we use an adjusting algorithm that stretches the border and curvature arrays adding interpolated pixels depending on the slope of the local tangent. 2.2. Fragments Match In this subsection we describe the steps (2) and (3) above. In order to find the longest common portion of the borders, we developed an algorithm called tank algorithm, listed in Fig. 2, that represents the foundational part of our method and the main contribution of this paper. We make two preliminary considerations before describing the core of the tank algorithm. First, one of the two curvature arrays has to be flipped because the border pixels in both images have been found in clockwise order. Second, since the fragments have close boundary curves, we consider the curvature arrays as circular by duplicating them.
Fig. 1. Various placements of the arrays
The tank algorithm requires that Shape1 is longer than Shape2. The outer loop cycles the different placements of the curvature arrays, as shown in Fig. 1. Initially they are positioned as in A, then Shape2 is shifted by one position (B), and so on until the end of the loop (C). The number of evaluated placements is equal to the length l1 of Shape1. For each single placement, the curvature arrays are compared element by element (pixel by pixel) from left to right (the double arrows of Fig. 1). The total number of element comparisons for each placement is equal to the double of length l2 of Shape2. This means that the worst-case complexity of the tank algorithm (not considering the color test, see below) is O(l1 l2 ). For each element comparison the algorithm calculates the absolute value e of the difference of the two corresponding curvatures. When e is minor than the parameter ε, the curvatures are very similar and a common portion of the borders starts. When a common portion has started, the e computed for the following elements is added to ErrorTank (initially set to 0). When ErrorTank exceeds the parameter ErrorTankCapacity, the current common portion is ended. While examining a common portion, e is not only added to ErrorTank but also contributes to reduce the ErrorTank value in particular cases. We use two parameters called R1 and R2 . When e < ε the ErrorTank is reduced by being multiplied by R1 (0 ≤ R1 ≤ 1), otherwise if e < 4 · ε, ErrorTank is reduced by being multiplied with R2 (0 ≤ R1 ≤ R2 ≤ 1). Typical values for the two paraments are 0.5 for R1 (ErrorTank becomes the 50% of its old value) and 0.8 (ErrorTank becomes the 80% of its old value) for R2 . R1 and R2 give a sort of reward when the curvature of two pixels are almost equal (R1 ) or enough similar (R2 ). This makes the tank algorithm very robust to sporadic differences in the common portion of the borders. Finally a refinement of the common portion is done: the elements from the end are trimmed until the error e is less than ε (this is necessary because, at the end of the search for the common portion, the curvatures were different and ErrorTank overflowed). Once a common portion has been found, we perform a color test by applying a modified version of the above part of the tank algorithm to the border arrays containing information about the RGB color values of the border pixels. The variable e now is the Euclidean distance (in three-dimensional space RGB) between the points that represent the colors of the corresponding pixels (in the current common portion). If, during the color test, ErrorTank exceeds the ErrorTankCapacity the common portion is discarded due to incompatibility in color of adjacent pixels. Otherwise the common portion is accepted and, if it is the longest for the current placement, it is labelled with its length (in pixel) and with a weight that represents the color diversity, given by ErrorTank/ErrorTankCapacity.
Require: Length(Shape1)>Length(Shape2) Shape1Dup ← Duplicate(Shape1) Shape2Dup ← Duplicate(Flip(Shape2)) for i = 0 to Length(Shape1) do MaxCommonLength ← 0; Counting ← FALSE; ErrorTank ← 0 for j = 0 to Length(Shape2Dup) do e ← |Shape1[(i+j) mod Length(Shape1Dup)].Curvature− Shape2Dup[j].Curvature| {compute the error} if (e < ε) ∨ (Counting) then ErrorTank ← ErrorTank+e Counting ← TRUE if ErrorTank < ErrorTankCapacity then if e < ε then ErrorTank ← ErrorTank ·R1 else if e < 4 · ε then ErrorTank ← ErrorTank ·R2 end if {increase the common portion length} CommonLength ← CommonLength+1 else {ErrorTank overflowed} if CommonLength > MaxCommonLength then {refine the candidate common portion} CommonLength ← Trim(CommonPortion) end if if CommonLength > MaxCommonLength then if ColorTest=TRUE then {color check ok: new longest common portion found} MaxCommonLength ← CommonLength Result[i] ← CommonPortionData end if end if CommonLength ← 0; Counting ← FALSE; ErrorTank ← 0 end if end if end for end for
Fig. 2. The tank algorithm
The tank algorithm produces the array Result that contains information about the longest common portion found for each placement of the borders. At the top of Fig. 3 is shown the normalized graph of the length of the common portions for all possible placements of two fragments. For each common portion, call ϕ the angle by which the first fragment must be counterclockwise rotated to align with the second one along the common portion of the borders. This angle is calculated by comparing the slopes of the segments that join the start and end points of the common portion in the two fragments. The length of the common portions for each angle ϕ forms a new array of 360 elements (it is possible to have approximations more accurate than one degree by using more elements). An example of this (normalized) array is shown at the bottom of Fig. 3; it is possible to notice that the best rotation angle is 198◦ . The translation is simply calculated in order to superimpose the common portions of the borders. 3. EXPERIMENTAL RESULTS In this section we present some of the experimental results obtained from the implementation of the method described above. The first prototypes of the algorithms have been coded in Delphi to preliminary assess the effectiveness of the method and to analyze the role of the different parameters involved, working in a GUI environment that could be quickly modified. Given this non-
Fig. 6. Two fragments of a jigsaw puzzle taken from the Web site of the movie “Le fabuleux destin d’Am´elie Poulain” and the reassembled puzzle
Fig. 3. Length of the common portions for the placements (top) and for the rotation angles (bottom) of the fragments of Fig. 4
Fig. 4. Two fragments of an image of Piazza dei Miracoli at Pisa (Italy)
optimized implementation, we did not make a deep complexity analysis of our method. The integration of the two fragments of Fig. 4 is reported in Fig. 5. Although it represents only a particular application, our method has proven to be effective also in reassembling fragments of jigsaw puzzles as shown in Fig. 6. Fig. 7 shows an example in which our method fails. In this case, the two fragments clearly cannot be correctly matched and the method returns the best match. However, the result appears absolutely reasonable when the shape and the color content of the fragments are considered (see the clouds!). Only humans can recognize the erroneous reassembly because of the semantic content of the fragments. The values of the parameters used in the above reassembling examples have been: D = 10, ε = 10, R1 = 0.5, R2 = 0.8, and ErrorTankCapacity= 0.5l2 (remember that l2 is the length of the second, shorter, fragment). The time required (on a 300MHz Pentium II computer) for matching the previous fragments ranges from 1s to 5s. It is impossible to report here the complete experimental analysis on the role of parameters in our method, but some main facts are worth to be discussed. The tank algorithm correctly finds the
Fig. 5. The reassembled image of Pisa (Italy)
Fig. 7. Two fragments of an image of Pisa (Italy) that are not correctly reassembled
common portion of the borders even when R1 = R2 = 1 (namely, when the tank level is never reduced) given that ε is large enough. For example, when ε > 5 for the fragments of Fig. 4. Conversely, if the tank level can be reduced (0 ≤ R1 ≤ R2 < 1), then the correct common portion is found also for small ε. For example, for ε = 1, R1 = 0.5, and R2 = 0.8 for the fragments of Fig. 4. Experimentally, it results that the longest correct common portions of the borders (namely the best matches) are often found when 10 ≤ ε ≤ 30 and that, in these cases, the influence of the values of R1 and R2 is small. Finally, the value of D depends on the resolution (the larger the resolution the larger D) and on the irregularities in the borders. In conclusion, the proposed method for reassembling fragments has shown to be robust in producing the correct result also when some little pieces of the boundaries are missing. 4. RELATED WORKS A number of methods have been developed to reassembly fragments in order to recompose both two-dimensional images and three-dimensional objects. In the case of 2D images, when the final result of integration is not known, the only information available to the methods is obtained from the fragments themselves. Roughly, this information is related to the boundaries (edges or fracture lines) or to the content of the fragments (their colors or drawings). The method presented in this paper exploits information about both boundaries and color content of fragments. Another effec-
tive method but based only on boundary information is presented in [6]. The outlines of the fragments are represented by curvature strings (as in [5]), namely by a circular sequence of samples, where each sample is the local curvature of the outline at the corresponding point. This representation is used also in our method. The matches between fragments are found by a multiscale procedure; while the refinement of matches is performed by a dynamic programming algorithm. In our method we do not employ a multiscale procedure for matching the fragments and, in the current version, we match only two fragments; thus we cannot directly compare the two methods. Although the representation is similar, our method for matching the outlines of the fragments differs from the one presented in [5]. The latter manages noise by considering two curvature values as equal if there is at most an ²-difference between them (where ² is a “local” measure of accuracy), while the former considers a “global” measure of accuracy as implemented in the tank algorithm. Our method deals more robustly with noise since it allows scattered pairs of significantly different curvature points within a sequence of pairs of significantly equal curvature points. (Note that our method essentially reduces to that in [5] when R1 = R2 = 1.) The worst-case complexity of the method in [5] is O(n2 ) (where n is the length of the longer border representation), which is comparable to that of our method. Hori et al. [7] propose a method for detecting the joint among two 2D potsherds. Their method uses a segmentation of the contours of the fragments and, then, finds match between pairs of segments. Its applicability (and that of similar techniques) is limited to the cases in which the fragments possess critical points (i.e., corners) on which the segmentation can be based. Our method is, in this respect, more general and a direct comparison is difficult. The object recognition techniques (see [8] for a survey) can provide hints to represent the boundaries and to develop algorithms for their matching. However, the object recognition problem is structurally different from the fragment reassembly problem. In the former it is usually the case when an object is known (the model) and the goal is to find, by matching, the occurrences of the object in a scene. The fragment reassembly problem is more “symmetric”, since the fragments to be matched are not subject to any model-instance relation. Finally, although addressing a similar problem, the most effective techniques for automatic assembly of jigsaw puzzles (like [9]) cannot be directly applied to fragment reassembly since they usually heavily rely on the specific shape of jigsaw puzzle fragments from which invariants are extracted. 5. CONCLUSIONS In this paper we have presented some results of a novel method for reassembling two fragments of a 2D image. The method has proved to be effective and robust in several experimental settings. Since our goal was to assess the validity of the proposed approach, we made no attempt to optimize the algorithms but we aim to improve their performance by implementing a more efficient C++ version of the code. Another direction of future work is to employ the method presented in this paper to address the reassembly of n fragments. Some preliminary results are shown in Fig. 8. The reassembled images have been obtained by a quite simple algorithm: given a pool of fragments, we look for the two fragments with the best match (the longest common portion of borders), we add to the pool the new fragment obtained by joining the two fragments (which are removed from the pool), and we continue until a single
Fig. 8. Two images of the Villa Contarini-Camerini at Piazzola sul Brenta (Italy) reassembled from two sets of 5 (left) and 6 (right) fragments
fragment is left. We aim to improve this elementary method for reassembling n fragments by pre-classifying the fragments in the pool (for example according to their color contents) and to experimentally test the method in real-case applications. 6. REFERENCES [1] M. Levoy, “Forma urbis romae project,” 2003, http: //formaurbis.stanford.edu/, last accessed January 8th, 2003. [2] Guide to the Recovery, Recomposition, and Restoration of Shattered Wall Paintings: Experience Gained at the Basilica di San Francesco in Assisi, ICR, Rome, Italy, 2001. [3] G. Barequet and M. Sharir, “Partial surface and volume matching in three dimensions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 9, pp. 929– 948, September 1997. “Intro to computer [4] MIT Computer Graphics Group, graphics,” 2003, http://graphics.lcs.mit.edu/ classes/6.837/F01/Lecture03/index.html, last accessed January 8th, 2003. [5] H. Wolfson, “On curve matching,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 5, pp. 483–489, May 1990. [6] H. C. de Gama Leit˜ao and J. Stolfi, “A multiscale method for the reassembly of two-dimensional fragmented objects,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 9, pp. 1239–1251, September 2002. [7] K. Hori, M. Imai, and T. Ogasawara, “Joint detection for potsherds of broken earthenware,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 1999, vol. 2, pp. 440–445. [8] A. Pope, “Model-based object recognition: A survey of recent research,” Tech. Rep. TR-94-04, University of California at Berkeley, Berkeley, CA, USA, January 1994. [9] G. Budea and H. Wolfson, “Solving jigsaw puzzles by a robot,” IEEE Transactions on Robotics and Automation, vol. 5, no. 6, pp. 752–764, December 1989.
