On the Consistency of Line-Drawings, Obtained by ... - Semantic Scholar

On the Consistency of Line-Drawings, Obtained by Projections of Piecewise Planar Objects

Anders Heyden

Dept of Mathematics, Lund Institute of Technology Box 118, S-221 00 Lund, Sweden

Abstract

This paper deals with line-drawings, obtained from images of piecewise planar objects after edge detection. Such images are used e.g. for navigation and recognition. In order to be a possible image of a three dimensional piecewise planar object, it has to obey some projective conditions. Criteria for a line-drawing to be correct is given in this paper, along with methods to nd possible interpretations. In real life situations, due to digitization errors and noise, a line-drawing in general does not obey the geometric conditions imposed by the projective imaging process. Under various optimality conditions, algorithms are presented for the correction of such distorted line-drawings.

1

Contents

1 Introduction 2 Depth and Shape 3 Criteria for Correctness

3 3 7

4 The Degree of Freedom

12

5 Objects with Occlusions 6 Overcoming Superstrictness 7 A Correction Method 8 A Comparison with Sugiharas Method 9 Taking Noise into Account 10 A New Correction Algorithm 11 Conclusions

15 19 22 24 26 29 31

3.1 De nitions and S -matrix Conditions : : : : : : : : : : : : : : : : : : : : : : : : : 7 3.2 Combinatorial Conditions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 10

4.1 Brief Review of Matroid Theory : : : : : : : : : : : : : : : : : : : : : : : : : : : 12 4.2 A Matroid Describing the Degree of Freedom : : : : : : : : : : : : : : : : : : : : 14

2

1 Introduction A central problem in scene analysis is the reconstruction of 3D-objects from 2D-images, obtained by projections. Two important classes of objects are polyhedral and origami ('papermade') ones, where the dierence is that in the latter case the objects need not be solid, but may partly consist of laminas. Projective images of such objects will, after edge detection, be called line-drawings. Conditions are imposed on the line-drawing by the projective image formation process. There exist line-drawings which are impossible as images of truly three dimensional objects. The impossibility may have dierent explanations. One well-known example is the Reutersward-Penrose tribar, see [Penrose]. An image of a real three dimensional object may be considered impossible because of e.g. noise; some examples are shown below. If the impossibility is due to noise there is an apparent need for correction methods. One such method has been proposed by [Sugihara 86]. Still better would be if some sort of optimality condition is ful lled, so that only small deformations are needed in the correction. Such a method was presented by the author in [Persson 91], and will also be outlined in this paper. It is also desirable to have some model of how noise in the image formation process in uences the correctness of the image, and to take this information into account when correcting the image. Such a model and correction method is presented below, see also [Heyden]. Another conceivable application for the correction methods is in computer aided design (CAD), where a careless line drawing may serve as input, and a projectively correct line drawing is obtained as output.

2 Depth and Shape In this chapter we will present some basic properties of the concepts of depth and shape. For more detailed treatments, see [Sparr 91], [Sparr 92b]. We start with the de nition of shape.

De nition 2.1. A con guration is an ordered set of points in 3-space, X = (X 1; : : : ; X n). Let xi denote their coordinates in some basis, i = 1; : : : ; n. Then the shape of X is de ned as the linear space (1)

s(X ) = f  j

" n X i ix = 0; i = 0 g = N x11 i=1 i=1

n X

#

1 ::: 1 : x2 : : : xn



An important property of s(X ) is that it is independent of the coordinate representation of the points, or (equivalently) that it is unchanged under ane coordinate transformations. Another property is that it can be used to determine if the points are collinear, coplanar or not coplanar. In fact, the following holds.

Lemma 2.1. Let X be a con guration of n points. Then dim s(X ) = n ? 2 if the points are collinear, dim s(X ) = n ? 3 if the points are coplanar, but not collinear, dim s(X ) = n ? 4 if the points are not coplanar. De nition 2.2. A perspective transformation (or perspectivity) with center Z and image plane  , Z 2=  , is a mapping with the property that every point on a line through Z is mapped 3

onto the intersection of the line with  , see Figure 1. If one con guration X is mapped onto another Y , where Y is planar, then for some  = (1; : : : ; n) with i 6= 0; i = 1; : : : ; n, holds

ZX i = iZY i; i = 1; : : : ; n: The vector  is called the depth of X with respect to Y . A projectivity is a composition of perspective transformations.  Remark. The center Z may be an in nity point and then the perspective transformation is just a parallel projection in the direction given by Z . 

Z

Y

X

Figure 1: A perspective transformation The importance of these concepts is illustrated in the following theorems and de nitions (for more details see [Sparr 92b]).

Theorem 2.2. If X and Y are planar con gurations, then the following statements are equiv-

alent:

(1) There exists a perspectivity P , such that P (X ) and Y have equal shape, and X has depth  with respect to P (X ) (2) diag()s(X ) = s(Y )

This theorem says that whenever an X -con guration, with a given shape s(X ), can be mapped by a perspectivity onto a Y -con guration, with a given shape s(Y ), that mapping must have the depth  given by the theorem, independently of Z and  . We will now consider point con gurations de ned by the vertices of piecewise planar objects. Each of the f polygonal faces of the object contributes by its vertices with a planar subcon guration. Xi = (Xi1; : : : ; Xini); i = 1; : : : ; f: Thus the whole con guration can be considered as an ordered set of such sub-con gurations, that is as a con guration of con gurations. 4

De nition 2.3. A con guration of con gurations will be called a composite con guration and will be denoted by (X ) = (X1 ; : : : ; Xf ): The subcon gurations Xi will be called the faces of the composite con guration. By X is meant the con guration consisting of the union of the points of the faces of (X ), with some ordering.



In the sequel, these concepts will be used to analyze piecewise planar objects and their projective images. Such an object consists of planar polygonal regions, also called faces. For each such polygon the vertices form a planar subcon guration. The object will be identi ed with the composite con guration built up by these subcon gurations. Concerning the image of a piecewise planar object, it is assumed that for each pair of adjacent faces in the object their intersection give rise to a linear arc in the image plane. These arcs together form what is called a line-drawing, consisting of a number of coplanar polygonal regions. Below we deal with the problem of reconstruction of the (class of) objects that are possible to project on a given line-drawing. The objects are assumed to be non-transparent. Since the line-drawing only contains information about the visible parts of the object, the word 'object' will be used synonymously with 'visible parts of the object'. We will also consider the situation of a scene consisting of several objects, where occlusions cause severe complications.

De nition 2.4.  If X is a point con guration, by an S -matrix for X is meant a matrix S X whose columns are a basis for s(X ).  If (X ) = (X1; : : : ; Xf ) is a composite con guration, by an S -matrix for (X ) is meant a matrix

S (X) = [S1 : : : Sf ]

such that, for i = 1; : : : ; f , { each point of X corresponds to a row of S (X), { Si corresponds to the face Xi, { Si has zeros on all rows corresponding to points not in Xi, and { Si can be obtained from " #

S Xi 0

by a permutation of the rows.  By the composite shape space s((X )) is meant the column space of S (X).



Example 2.1. Consider the composite con guration in Figure 2, consisting of four faces. Only

three of them contribute to the S -matrix because the triangular face has a trivial shape space. If a basis for the shape space of face I is [1 2 3 4 ]T , for face II is [1 2 3 4 ]T and for face

5

1

4

I 5

II 6

III 2

3

Figure 2: A composite con guration III is [1 2 3 4 ]T , then

2

1 1 0 3 6 7 6 2 0 1 7 6 7 S (X) = 6660 2 02777 3 6 3 7 44 0 35 0 4 4



Note that the S -matrix is not uniquely de ned. Also note that the composite con guration (X ) contains more structure than the con guration X , because in (X ) the points are ordered in subcon gurations. From the de nition follows that there are more columns in S X than in S (X), because every column in S (X) belongs to s(X ), but together they don't necessarily span s(X ). Hence s((X ))  s(X ): The S -matrices tell us a lot about the original con guration, which the next theorem shows.

Theorem 2.3. Let (X ) be a composite con guration and let (Y ) be its image under a projective

transformation with known point correspondences. Then to every choice of S (X) there exists an S -matrix S (Y) with the same distribution of zeroes as S (X). For these S -matrices holds for some  and c,

diag()S (X) = S (Y) diag(c):

(2) An equivalent formulation is that

diag()s((X )) = s((Y )): Proof:

See [Sparr 92b].

De nition 2.5. Given a planar composite con guration (Y ), we say that a composite con guration (possibly three-dimensional), (X ), is a pre-image of (Y ) if there is a projective transformation taking the points in (X ) to the points in (Y ).  6

According to the previous theorem (X ) is a preimage of (Y ) if and only if there exists S (X) and S (Y) ful lling (2) for some  and c. The next theorem tells us, for a given planar composite con guration (Y ), which depths that can occur for a perspective transformation of a composite con guration with image (Y ).

Theorem 2.4. Given a planar composite con guration, (Y ), and an S -matrix, S (Y). Then the set of possible depth vectors  = [1 : : :n ]T for Y with respect to some preimage is given by the solution to

?1 T (3) T S (Y) = 0; = [?1 1 : : :n ] ; In other words the possible inverse depths are given by the left nullspace to S (Y) , NL (S (Y) ). Proof: According to Theorem 2.3 diag()S (X) = S (Y) where S (X) is an S -matrix of some preimage. The fact that S (X) is an S -matrix means that each of its column sums vanishes, which is the same as eT S (X) = 0 ) eT diag( )S (Y) = 0 ) T S (Y) = 0 where e = [1 1 : : : 1]T . Remark. Let eT = [1 1 : : : 1] and let x and y be the vectors containing the x- and y-coordinates of the points in the composite con guration, respectively. Then e, x and y belong to the left nullspace of S (Y) by the de nition of shape. The depth e gives just a preimage identical to (Y ). The vectors x and y correspond to preimages obtained by tilting the image plane around the y-axis and the x-axis, respectively. All of them give a planar preimage and are therefore in general not interesting. The interesting question is whether there exists another depth vector, consistent with Theorem 2.3, which corresponds to a truly three dimensional preimage. 

3 Criteria for Correctness In this section we rst discuss what is meant by a correct line-drawing and how to use the S matrix to determine if a composite con guration is correct. Then we present some combinatorial properties of correctness.

3.1 De nitions and S -matrix Conditions

Next we have to de ne what to mean by a possible/impossible line-drawing of a piecewise planar object. As is discussed above such a line-drawing is obtained by projection from a piecewise planar composite con guration where every pair of adjacent faces intersect in a line. A human person would in general interpret the lines in a line-drawing as caused by such intersections of planes in the scene. Then a line-drawing is correct if there exists a preimage where no pair of planes corresponding to opposite faces of a line coincide. Another way to say this is that every pair of adjacent faces in the line-drawing corresponds to non-parallel planes in the preimage. Sugihara has used a slightly stronger condition requiring no planes at all in the preimage to be parallel, see [Sugihara 86]. We restate these comments as formal de nitions.

De nition 3.1. A planar composite con guration will be called truly possible or truly correct if there exists a preimage where no two adjacent faces are coplanar. Otherwise the planar composite con guration is said to be truly impossible or truly incorrect.  7

De nition 3.2. A planar composite con guration will be called strictly possible or strictly correct if there exists a preimage where no two faces are coplanar. Otherwise the planar composite con guration is said to be strictly impossible or strictly incorrect.  These de nitions of correctness have an intuitively good interpretation but they are hard to work with. Therefore we give a third de nition of correctness.

De nition 3.3. A planar composite con guration, (Y ), will be called weakly impossible or weakly incorrect if for some connected collection of faces (A) = (Yi ; : : : ; Yik ), k  2, holds P : (X ) ?! (A); P a projectivity =) (X ) is a planar con guration: Otherwise the planar composite con guration is said to be weakly possible or weakly correct.  1

This is a slightly weaker condition than the two given above. The relations between them are given in the following proposition.

Proposition 3.1. Let (Y ) be a planar composite con guration. Then (Y ) is weakly impossible ) (Y ) is truly impossible ) (Y ) is strictly impossible or equivalently

(Y ) is strictly possible ) (Y ) is truly possible ) (Y ) is weakly possible : Proof: If (Y ) is weakly impossible then there exists some connected collection, (A), of faces that are coplanar in every preimage of (A). Thus (Y ) is truly impossible, because there exist adjacent faces that are coplanar in every preimage of (Y ). If (Y ) is truly impossible then there exists a pair of adjacent faces that are coplanar in every preimage of (Y ) and thus (Y ) is strictly impossible.

Theorem 3.2. (Y ) is a weakly impossible planar composite con guration if and only if (4) rank S (A)  jV (A)j ? 3 for some collection of faces (A) of (Y ), where jV (A)j is the number of points in A. Proof: For any preimage (X ) of the planar composite con guration (A) holds diag()s((X )) = s((A)); or equivalently,

s((X )) = diag()?1s((A)):

Hence (4) holds for (A) if and only if it holds for (X ), which in turn, by Lemma 2.1 is the same as that (X ) is planar. This proves the assertion. Theorem 3.2 shows that it is possible to use the S -matrix S (Y) to determine if the planar composite con guration (Y ) is weakly possible.

De nition 3.4. We say that (Y ) obeys the rank condition if rank S (A)  jV (A)j? 4 for every collection of faces, (A), of (Y ). Here again jV (A)j denotes the number of points in A.  8

According to De nition 3.1 and 3.2 we want to determine whether or not two faces in a planar composite con guration correspond to coplanar faces in every preimage. The following de nition is convenient.

De nition 3.5. Given a planar composite con guration, (Y ). The points Y 1; : : : ; Y l in Y are said to be gravely coplanar if their corresponding points are coplanar in every preimage of (Y ).  According to the previous de nition, a planar composite con guration is truly incorrect if and only if there exists four points on two adjacent faces (not all contained in one face) that are gravely coplanar. Analogously a planar composite con guration is strictly incorrect if and only if there exists four points (not all contained in one face) that are gravely coplanar. The following lemma can be used to check if four given points are gravely coplanar.

Lemma 3.3. Given a planar composite con guration, (Y ). Then four points, Y i; Y j ; Y k; Y l in Y are gravely coplanar if and only if s((Y )) contains a vector with vanishing components except for positions i, j , k, l. Proof: For every preimage (X ) of (Y ) holds

diag()s((X )) = s((Y )); or equivalently, for some . Hence

s((X )) = diag()?1s((Y ));

v 2 s((X )) () w = diag()v 2 s((Y )) for some . Here the components of v and w vanish at the same positions. In particular, if v has non-vanishing components only at positions i, j , k, l, the same holds for w, which in turn, by Lemma 2.1, means that the con guration (X i; X j ; X k; X l) is planar. Since this holds for

every preimage of (Y ), the lemma is proved. An equivalent formulation of the condition "s((Y )) contains a vector with vanishing components except for positions i, j , k, l" can be formulated as follows. There exists some linear combination of the columns in S (Y) which has zero entries except for the positions i, j , k, l. That is, there exists a vector  such that S (Y)  = s~, where s~ is a vector built up by the shape vector for the four point combination Y i ; Y j ; Y k ; Y l at positions i, j , k, l and zeros elsewhere. To nd gravely coplanar point sets, Lemma 3.3 can be combined with the following proposition.

Proposition 3.4. Let Sijkl denote the S -matrix, S (Y), for the planar composite con guration (Y ) with the four rows i, j , k and l deleted and let n denote the number of points in Y . If S (Y) has n ? 4 columns and rank equal to n ? 4 the following holds. Y i; Y j ; Y k and Y l are gravely coplanar () det(Sijkl) = 0 Proof:

It is obvious that

det(Sijkl) = 0

() 9 6= 0 j Sijkl = 0 () 9

9 6= 0 j S (Y) = 0 except in position i, j , k and l

The positions i, j , k, l can not all be zero because the rank of the S -matrix is n ? 4. By the previous lemma this is equivalent to the statement in the proposition. Remark. If the S -matrix has rank > n ? 4, we can remove columns of the S -matrix that are linear combinations of other columns in the S -matrix until a basis for s((X )) is left. The S -matrix can at most have rank n ? 3, because there are three independent vectors in the left nullspace: e, x and y . If the rank is n ? 3 every preimage must be planar, according to Lemma 2.1, and every subset of four points must be planar in every preimage. If the rank is less than n ? 4, say n ? k (k > 4), after the removal of linearly dependent columns as above, the condition det(Sijkl) = 0 can be replaced by rank(S~ijkl) < n ? k or det(Q) = 0 for some square submatrix Q of S~ijkl, where S~ijkl is the resulting matrix after the removal of linearly dependent columns as above and removal of the rows i, j , k and l. The proof of this is almost the same as the previous proof.  Remark. By Proposition 3.4 we can examine if a given composite con guration is truly possible or strictly possible. We can check whether or not all adjacent faces respective all faces are gravely coplanar. If no adjacent faces respective no faces are coplanar we can nd a solution to (3) where these two faces are not coplanar. Then we can make a suitable linear combination of these solutions to make a preimage where no pair of adjacent faces respective no pair of faces are coplanar. 

3.2 Combinatorial Conditions

Below conditions for a composite con guration to be correct will be formulated in terms of its incidence structure. The following notations are needed.

De nition 3.6. Given a composite con guration, (Y ), we introduce the following sets: V = fpoints in (Y )g F = ffaces in (Y )g R = frelations in (Y )g  V  F where (r; f ) 2 R if the point r is incident with the face f . If A  F , then let V (A) be the set of points incident with some face in A and let R(A) be the set of relations (r; f ) where f 2 A. Further let (v ) denote the multiplicity of v 2 V , which by de nition means the number of

faces incident with v . If S = S (Y) is an S -matrix for the composite con guration (Y ) and A is a subset of F we let S (A) denote the matrix consisting of the columns in S corresponding to the faces belonging to A.  For a subset, A, of faces in a composite con guration, jR(A)j is the number of 'relations' on A, which means the number of vertices counted with multiplicities, and jV (A)j is the number of distinct vertices in A. Here jAj means the cardinality of the set A. By a combinatorial condition is meant a condition expressed in terms of jV j, jF j, and jRj. Since these values are stable under deformations of the points of the con guration, a stability property has to be imposed on the con guration itself in order to formulate a combinatorial condition for correctness.

De nition 3.7. The planar composite con guration (Y ) is in S -generic position if the columns of an S -matrix, S (Y) , are linearly independent.  10

Remark. When the planar composite con guration (Y ) is in S -generic position it has the following property. For every deformed planar composite con guration (Y +
Y ) with vertices Y i +
Y i, i; : : : ; f , holds

rank(S (Y+
Y) ) = rank(S (Y) ) for every deformation
Y i with jj
Y i jj < .  Remark. Observe that if the columns are linearly independent in one S -matrix then they are linearly independent in every S -matrix of (Y ). Observe also that the condition that the planar composite con guration is in S -generic position depends both on the incidence structure, (V; F; R) and the positions of the points.  P

Example 3.1. Consider the planar composite con guration in Figure 2. If the three dividing

lines between faces I-II, II-III and III-I meet in a common point the rank of the S -matrix is 2, else the rank is 3. In the rst case there is a linear dependency among the columns. Thus the composite con guration is not in S -generic position in this case. In the second case, where the lines do not meet in a common point, the columns are linearly independent. Thus the composite con guration is in S -generic position in this case. The con guration has 6 points and it can easily be seen that it obeys the rank condition in the rst case but not in the second. 

Lemma 3.5. Given a planar composite con guration (Y ). Then for every subset of faces, A, the number of columns in S (Y) (A) is equal to jR(A)j ? 3jAj. Proof: The number of columns in S (Y) (A) is X

f 2A

(jV (f )j ? 3) =

X

v2V (A)

((v )) ? 3jAj = jR(A)j ? 3jAj

because the number of shape vectors from each face is equal to the number of vertices minus 3 and X X (v): jV (f )j = f 2A

v2V (A)

When the planar composite con guration is in S -generic position there is a criterion, involving only the number of edges and faces in the image, telling us if the image is weakly correct.

Theorem 3.6. A planar composite con guration in S -generic position is weakly correct if and only if for every subset, A, of faces, with jAj > 1 holds jR(A)j ? 3jAj < jV (A)j ? 3 Proof: If the inequality above is ful lled for every subset, A, of faces, then according to the previous Lemma rank(S (Y) (A)) = jR(A)j ? 3jAj < jV (A)j ? 3; for every subset, A, of faces. Thus the rank condition is ful lled and the planar composite con guration is weakly correct according to Theorem 3.2.

We conjecture that Theorem 3.6 is true under the weaker condition of the Remark after De nition 3.7. Sugihara, [Sugihara 86], and Whiteley, [Whiteley], use another condition to achieve stability. In order to forbid special dependencies among the points of the con guration, considering them as indeterminates, they introduce the following concept. 11

De nition 3.8. A set of points in R2 is said to be in generic position if the coordinates are algebraically independent over the rational numbers.  Note that this condition is stronger than S -generic position.

Theorem 3.7 (Sugihara, Whiteley) A planar composite con guration where the points are in general position is strictly correct if and only if for every subset, A, of faces, with jAj > 1 holds jR(A)j ? 3jAj < jV (A)j ? 3: Proof: See [Sugihara 86]. There is a special name for these images, introduced by Sugihara. De nition 3.9. A planar composite con guration, (Y ) with vertices Y i, i; : : : ; f , is said to be generically reconstructible if for some  > 0 the deformed planar composite con guration (Y +
Y ) with vertices Y i +
Y i , i; : : : ; f , is strictly correct for every deformation
Y i with P jj
Y ijj < .  Another way to look at this problem is by means of combinatorial geometries or matroids, see the remark after Theorem 4.2 in the next chapter.

4 The Degree of Freedom This section deals with the problem of how to describe the set of preimages of a given planar composite con guration. We begin with a brief review of matroid theory and then we investigate the degree of freedom.

4.1 Brief Review of Matroid Theory

For an introduction to these subjects see [Crapo], [White 86], [Welsh] or [Oxley]. Here we will give a brief introduction. We start with the de nition of a matroid.

De nition 4.1. A matroid M is a nite set, S , together with a collection (family) of subsets, F 2 2S , such that the following three conditions hold (i) ; 2 F (ii) X 2 F ; Y  X ) Y 2 F S (iii) U; V 2 F ; jU j = jV j + 1 ) 9x 2 U n V ; V fxg 2 F where jU j means the cardinality of U . The subsets in F are called independent subsets and the others are called dependent.  Example 4.1. Let V be a nite vector space and F the collection of linearly independent subsets of vectors of V . Then (V; F ) is a matroid. In this matroid independence has the same meaning as linear independence and dependence the same as linear dependence.  The concept of matroids is inspired by the previous example and matroid theory can be seen as an abstract theory of linear independence. From this analogy with vector spaces the following de nitions are made. 12

De nition 4.2. A base of a matroid M is a maximal independent subset of S . The collection of bases is denoted by B or B(M ). A circuit of a matroid is a minimal dependent subset of S .  The term circuit comes from the connection between matroids and planar graphs, where circuits in the matroid corresponds to circuits in the graph. A matroid can also be characterized by the notion of a rank function.

De nition 4.3. The rank function of a matroid is a function  : 2S ! Zde ned by (A) = max(jX j : X  A; X 2 F ); for A  S: The rank of the matroid is de ned as the rank of the set S and is denoted by (M ).



We could equally well have started with the rank function when specifying the matroid as the next theorem shows.

Theorem 4.1. A function  : 2S ! Zis the rank function of a matroid on S if and only if for

any subsets X , Y of S: (i) 0  (X )  jX j

(ii) X  Y ) (X )  (Y ) S T (iii) (X Y ) + (X Y )  (X ) + (Y ) The proof can be found in any of the references given above. Given a function  with the properties (i),(ii) and (iii) above, a matroid is de ned by specifying the independent sets as

X 2 F , (X ) = jX j: One nice property of matroids is the following.

Theorem 4.2. All bases in the matroid M have the same cardinality, that is the same number of elements, and this number is the same as the rank of S . The proof can be found in any of the references given above. A matroid can also be de ned by its circuits, see [Welsh]. It is possible to extend the results above to in nite sets S by requiring that the bases still are nite sets, that is the rank of the matroid is nite. Remark. In this terminology a planar composite con guration is a matroid which has the points in the con guration as underlying objects. Further all subsets of four points on a plane and all subsets of ve points where no four of them are on the same plane are circuits, i.e. minimal dependent sets. The independent sets are the sets consisting of one, two or three points and those consisting of four points not incident with the same face. Formally this can be written S = fpoints in the composite con gurationg = V 8 > < jAj if jAj  3 (A) = > 3 if jAj > 3 and all points in A are on the same face : 4 else This is a rank four geometry on the points of the composite con guration. A coordinatization of a planar composite con guration, (Y ), over a eld K, in our case R, is a mapping

F : (Y ) ! K4; Y 7! xY 13

such that, for all distinct Yi ; Yj ; Yk ; Yl 2 (Y ); det(xYi ; xYj ; xYk ; xYl ) = 0 if and only if Yi ; Yj ; Yk ; Yl is contained in some face of (Y ). A composite con guration is strictly correct if there exists a coordinatization with the rst component of the xY -vectors equal to one, the second equal to the x-coordinates and the third equal to the y -coordinates of the points in the image. Then the fourth components are the depth values of the points in a three-dimensional pre-image. This problem has been treated in the rst chapter in [White 87] and the article [Sturmfels]. 

4.2 A Matroid Describing the Degree of Freedom We begin with the following de nition.

De nition 4.4. The degree of freedom of a planar composite con guration is the maximal

number of vertices in a preimage that can be independently speci ed. This is the same as the maximal number of depths that can be independently prescribed.  We say that two preimages dier if their respective depths are not proportional as vectors in Rn. Further we say that the number of dierent preimages is the same as the dimension of the linear space of possible depths. This can be written in the language of matroids as follows. Let (Y ) be a planar composite con guration. From Theorem 2.3 we know that T S (Y) = 0 gives the possible depths. To specify one point in R3, that is give it a prescribed depth, corresponds to imposing an equation of the form T ei = di , where ei is the i:th unit vector and di is the prescribed depth. We also observe that rank S (Y) = jV j? k means that k depths can be speci ed (at most), where jV j means the number of points in Y according to De nition 3.6. Then we get the following theorem.

Theorem 4.3. The function V : 2V ! R de ned by i h V (X ) = rank( S (Y) j IX ? rank(S (Y)) 



is the rank function of a matroid. Here S (Y) j IX is a block matrix with the second block IX composed by columns ei , unit vectors, with i corresponding to points in X . V (X ) is the maximal number of the depths in X that can be speci ed independently. Proof: The de nition of V is independent of the choice of S -matrix because the linear space spanned by the columns is the same independently of the base chosen for each subcon guration. We begin with the last statement. V ? rank(S (Y)) is the maximum number of depths that can be independently speci ed. If the depths for the points in X are speci ed independently, there  (Y) are jV j ? rank( S j IX ) depths left to specify. The dierence is V (X ), by de nition, and thus the second statement is proved. In order to prove that V (X ) is a matroid the following three conditions have to be checked

(i) 0  V (X )  jX j This is clear from the de nition. (ii) X  Y ) V (X )  V (Y ) This too is clear from the de nition. S T (iii) V (X Y ) + V (X Y )  V (X ) + V (Y ) This can be proved analytically, but we will give a more discussing proof. Consider the equivalent inequality [

\

V (X Y ) ? V (Y )  V (X ) ? V (X Y ): 14

The left hand side is the degree of freedom in X n Y when the depths in Y are speci ed. T The right hand side is the degree of freedom in X n Y when the depths in X Y are speci ed. Thus the inequality holds because less degrees of freedom remain in X n Y if the values of the depths corresponding to vertices outside X n Y are prescribed. This is a matroid on the vertices in the image. We could equally well have de ned a matroid on the faces in the image, because to every face corresponds the vertices incident with the face. We call the rank function of this matroid of faces F . Following Sugihara we introduce

De nition 4.5. Given a subset A of faces, we call F (A) the degree of freedom of A. Further A is said to be independent if F (A) = jAj and dependent otherwise. A maximal independent subset is called a base.  According to the previous theorem we see that we can specify the positions in the preimage of the faces A if and only if A is an independent set of the matroid. The preimage is uniquely de ned if A is a base of the matroid. Then we can talk about the degree of freedom of the vertices or equivalently on the depths and we can talk about the degree of freedom of the faces. Note that these to functions V and F give the same degree of freedom according to the correspondence above between faces and points. The same terminology of dependent subsets and so on applies too. It follows that

V (V ) = jV j ? rank(S (Y)): One property of matroids is that every base has the same cardinality, that is if V (A) = jAj and V (B ) = jB j then jAj = jB j. Further V (V ) = V (B ) for any base B . This means that the degree of freedom is V (V ). We restate this as a theorem

Theorem 4.4. The degree of freedom in a planar composite con guration, (Y ), is V (V ) = jV j ? rank(S (Y)): This theorem can of course be proved by standard linear algebra, but to describe the degree of freedom for every subset of vertices or faces the matroid formulation is needed. A base in the matroid can beSconstructed from one empty set B by going through all vertices, v , adding them to B if B fv g is independent, dropping them otherwise. Eventually V (B) = V (V ) and we get a base. This is called the 'greedy algorithm'. In the sequel the term correct will always mean weakly correct.

5 Objects with Occlusions So far we have not considered occluded objects, that is objects that in the image seem to be partly hidden by another object or by part of itself. In order to determine the correctness of such an image we have to allow that each point where two objects meet in the image corresponds to two points in the preimage, and the point on the occluded part is more distant from the camera than the point on the occluding part. We have to impose some restriction on the class of objects appearing in the scene. One method is to assume that at most three faces meet at a common point. Then all occlusions appear at 'T-crossings', see Figure 3, where the occluding part is the top bar of the 'T' and the occluded part is the vertical line in the 'T'. This was for instance done in [Sugihara 86], and will be done below. 15

T-crossing

Figure 3: A T-crossing Given a planar composite con guration (Y ). As stated before, where occlusion occurs, we have to introduce two points in the preimage, (X ), instead of one. Then an S -matrix of (Y ), S (Y), is formed as usual, with some points coinciding. In this case it is not sucient that (Y ) ful ls the rank condition in order to be correct. We must also impose that the depth value for the occluding point is less than the depth value for the occluded point in the preimage at every occlusion. Thus we need to know the depths for the dierent points in the image. The possible depth values in a preimage are given by Theorem 2.4. There are two dierent ways to cope with this problem. We have to solve a system of linear equations T S (Y) = 0 where = [?1 1 ; : : : ; ?1 n ] are the inverse depths, under some set of inequality constraints

i <  j : This gives a set of inverse inequalities in the inverse depths

i > j : By a physical imaging process, it is assumed that the depths are greater than zero, because otherwise the point in the preimage is behind the camera. For our problem, we have no problem with negative depths. Since e = [1 : : : 1]T is a vector of possible inverse depths, once we have a vector of inverse depths satisfying the inequalities above, by addition of e and linearity we get an inverse depth vector with positive inverse depths satisfying the inequalities too. Observe that the inverse depths form a linear space, the left nullspace of S (Y) , but not the depths. One way to compute is to solve the whole system of equations and inequalities at the same time. This can be done by linear programming. The solution is obtained by nding a basic feasible solution to the LP problem. This method was proposed by Sugihara in [Sugihara 86]. One drawback of the method is that the time complexity is very large. Another way is to solve the system of equations rst by nding a base of the left nullspace to the S -matrix (the number of vectors given depend on the degree of freedom of the image, see Section 4). Then we can discard the vectors lying in the space spanned by e, x and y , because these vectors are not interesting in the lift of the image. Then we have to detect if some linear combination of the remaining vectors ful l the inequality constraints. This is a standard problem of nding a point in a convex set de ned by linear inequalities, and ecient 16

methods exist (see [Schrijver]), based on LP-techniques. The method has a much lower degree of time complexity if the number of vectors obtained in the rst step is small compared to the number of points in the image. Given 1; 2 ; : : : ; n, solutions to T S (Y) = 0, we want to nd some linear combination n X ~ = i i i=1

such that ~j > ~k for some indices j , k. This means n X

( or equivalently

i=1

n X i=1

i i)j

n X

> ( i i)k; i=1

((i i )j ? (i i )k ) > 0:

Introducing ( ^i )k = ( i)j ? ( i )k for i = 1; : : : ; n gives n X i=1

De ne

(i ^i )k > 0:

M = f 2 Rn j 
( ^)k > 0g:

This is a convex set (possibly empty) and if we nd a point in M we also have a solution to the problem above. The set of possible depths is thus given uniquely by the extremal points of M . If we require all components in the inverse depth vector to be positive and to have component sums one the extremal points are unique. This can of course also be done if there is one point in the image corresponding to three dierent points in the preimage. This occurs if there are three objects involved in an occlusion at the same point.

Example 5.1. One interesting example is the following planar composite con guration, see

Figure 4. Here we have four inequalities involving the inverse depths

10 > 12 13 > 17 18 > 22 26 > 24:

The gure on the left has a base for the inverse depths at the points 10, 12, 13, 17, 18, 22, 24 and 26 consisting of e, x, y and 2 3 ?0:1132 6 7 6 0:0494 7 6 7 6 ?0; 071 7 6 7 6 ?1:077 7 6 7 6 0:0815 7 : 6 7 6 7 6 ?0:0695 7 6 7 4 ?0:3170 5 ?0:1661 17

4

4 5

5

2

2 3

1

3

1 25

6 9

25 6

26 24

12

9

21

10

20

11

23

23

22

19

15 22

18

(0,0)

15

16 13

17

14 7

19

18 16

13

21

10 12

20

11

26 24

17

14 7

8

(0,0)

8

Figure 4: Incorrect and correct composite con gurations Note that the minus signs do not cause any trouble according to the comment above. Positive depths can be obtained by addition of e. It can be seen that this vector does not ful l the inequalities above (nor does minus this vector). Thus the rst picture is incorrect. The corresponding inverse depths for the gure on the right are 3 2 0:2402 7 6 6 0:1649 7 7 6 6 0:0068 7 7 6 6 ?0:1119 7 7 6 6 0:0886 7 : 7 6 7 6 6 ?0:0894 7 7 6 4 ?0:2920 5 0:0640 It can be seen that this vector ful ls the inequalities above. Thus the second picture is correct.

 Example 5.2. In this example we show what results can be obtained for a more complicated

composite con guration. The scene in Figure 5 from [Guzman] consists of 131 points and 45 depth discontinuities. The S -matrix is 131x82 and the degree of freedom is 49. Thus there are 46 independent inverse depth vectors (e, x and y has been discarded) spanning the linear space of possible inverse depths. There exists solutions to this linear programming problem and one of them is shown in Figure 6 where curves connecting points with equal depths has been drawn. In order to form the composite con guration from the line-drawing in Figure 5 we have to make the following assumptions. The lines between faces 27 and 28, 28 and 29, 32 and 31, 31 18

300 25

250

23 3

1

26

24

2

200

33

20

27

5

32

150

40

21

22

28

34

19

4

31

29

30 6

100

7

8

38 35

36 37

39

9

18

16

15 13 14 12

17

50 10

11

0

-50 0

50

100

150

200

250

300

350

Figure 5: A scene from Guzman and 30, are lines drawn on planar faces and we have to discard them. The polygonal area 6 is part of the background and either we discard it or assume that there are depth discontinuities at every point on face 6. The common vertical line shared by face 4 and face 37 may be incidental and then we have to impose a depth discontinuity there or it may well be a real alignment. We have assumed a depth discontinuity here. The intersection of face 12, 13 and 18 are not entirely clear from the image. We have assumed that the line between face 13 and 15 meets the line below face 18 before this line below face 18 meets face 12. 

6 Overcoming Superstrictness

One problem that occurs when a planar composite con guration (Y ) is analyzed is that the S -matrix, S (Y), does, in general, not obey the rank condition. This is because when noise is aecting the image, the points will be in generic position (and in S -generic position) and only generically reconstructible images will be considered correct. This problem is called superstrictness. One method to deal with this problem was given by [Sugihara 86]. The idea is to extract a maximal generically reconstructible substructure of the image and then see if the remaining points are close to be in the correct plane. This method is ecient, but a drawback is that it does not treat all points in the image with equal priority. In this section we will outline a 19

300 4

250

4

3 4

5 4

200

17

20

150

22

22

34

9

9 24

12 38

100

35

38

16

36

11 15 16

36

14

34

13

17

15

10

3

50 4 5

9

0

-50 0

50

100

150

200

250

300

350

Figure 6: A scene from Guzman with iso-depthcurves method which is more symmetric, but does not solve the problem fully in some cases. When an S -matrix does not ful l the rank condition but the matrices built up by the subsets of columns that do not obey the rank condition has singular values that are small, it would be desirable to correct the S -matrix. This means nding a new S -matrix which is close to the previous one in some sense but obeys the rank condition. To do this we have to nd all subsets, A, of faces in the planar composite con guration that do not obey the condition rank S (Y) (A)  jV (A)j ? 4 It is sucient to nd all minimal subsets, with respect to inclusion, that obey the condition rank S (Y) (A) = jV (A)j ? 3: This is because a subset, A, of columns can not have rank more than jV (A)j? 3 when there are three independent vectors in the left nullspace, e, x and y . Examples of such minimal subsets are two faces which meets at two distinct lines and so called calottes (see Figure 7). Observe that the number of vertices in the calotte is 12 and the number of columns in an S -matrix is 9. This shows that there must be one linear relation between the columns in the S -matrix if the composite con guration is correct. 20

Figure 7: A calotte If we want to correct the con guration we have to decrease the rank on these minimal subsets, which is the same as making some linear combinations vanish. These linear combinations can be found by a singular value decomposition. This kind of linear dependencies have a special name.

De nition 6.1. A linear dependence between the columns in the S -matrix corresponding to minimal subsets of faces not obeying the rank condition is called a second order syzygy.  Remark. The rst order syzygies are the vectors in S Xi , for each face Xi , with all components

except four equal to zero, because they are minimal linear dependencies of the inverse depths of the points in the preimage.  Remark. Observe that we are only interested in minimal subsets of faces not obeying the rank condition because there must be a linear dependency among the columns in such a subset. This gives us a new matrix of such linear dependencies where each row corresponds to a column in the S -matrix and each column corresponds to a linear dependency of shape vectors corresponding to minimal subsets as above and zeros on the rows corresponding to shape vectors not present in the actual minimal subset. This could be called the second order S -matrix.  This is not the solution to the problem stated above, because we may in this way force more linear combinations than necessary for ful lling the rank condition, because there may be linear dependencies in minimal subsets of these second order syzygies. These linear dependencies of minimal subsets of second order syzygies are called third order syzygies and so on. It is an open problem to nd all higher order syzygies to a composite con guration. We will give some ideas of how an algorithm may look like. We can nd the second order syzygies as mentioned above. Then we can see if the structure of the S -matrix is destroyed when we impose these linear dependencies on the columns of the S -matrix. This means that some entries in the S -matrix are forced to vanish and thus we have, in some sense, too many second order syzygies and we must impose some third order syzygies. Then we can see if third second order syzygies destroy the structure of the second order syzygies and so on. 21

7 A Correction Method An image may be impossible for more or less serious reasons. This has been used by artists, to create special eects, see [Sparr 92b] for examples. Real-world images are in general disturbed by noise, making the image not obey the laws of perspectivity exactly. In this case, methods for correction are needed. One such was given in [Sugihara 86]. A novel one, based on Theorem 3.2, is presented below. Let S (Y) be computed from the image. If the rank is less than jV (A)j ? 3 for every subset, A, of faces the image is correct, otherwise it is incorrect. But, in the rst case, if the S -matrix has singular values close to zero, then the defect in rank may come from measurement noise. We then want to adjust the positions of the vertices in order to get a correct image, ful lling the rank condition in Theorem 3.2. One possible way to do this is to solve the minimization problem n ? X
min (y1i ? y~1i )2 + (y2i ? y~2i )2 ; i=1

under the constraint that (Y~ ) has the same topological structure as (Y ), but rank S (Y)~ (A)  jV (A)j? 4 for every subset, A, of faces. Here Y i are the points of the original image and Y~ i are the points of the corrected image, i = 1; : : : ; n. Unfortunately this optimization problem has nonlinear constraints and can not be easily solved analytically. Another way to obtain a possible image is rst to try to decrease the rank of the subsets of columns of S (Y) , not obeying the rank condition, without changing the structure, which means by maintaining the distributions of zeros. Having found such an approximation of S (Y) , we then try to nd a point con guration which is close to the original one and has the new S -matrix. In order to decrease the rank of S (Y) we start by making a singular value decomposition of the minimal subsets of columns not obeying the rank condition. This is done in order to nd suitable linear combinations that can be used to decrease the rank of S (Y) later. There we can see which linear combination of the columns that is closest to zero (in 2-norm), by solving the problem jjS (Y)(A)wjj2 ; min w jjwjj 2

for the minimal subset A. If the minimum is attained for w = w1 we solve the same problem again with the additional constraint w?w1. This gives w2 . This procedure may be repeated until we have enough linear combinations to decrease the rank to jV (A)j? 4. This singular value decomposition must in general be done on every subset of faces not obeying the rank condition. The next step is to nd a matrix which is close to S (Y) and ful ls the new linear constraints 1 w ; : : : ; wm. One way to do this is showed by the following theorem.

Theorem 7.1. Let I = f (i; j ) j (S (Y))ij 6= 0 g. Then the problem X (S~(Y) ? S (Y) )2ij min jjS~(Y) ? S (Y) jjF = min (i;j )2I

under the constraints w1; : : : ; wm 2 N (S~(Y) ) and eT S~(Y) = 0, where eT = [ 1 1 : : : 1 ], has the solution (S~(Y) )ij = (S (Y) )ij + 1i wj1 + : : : + mi wjm + j ; when (i; j ) 2 I

22

where j and  are obtained from 2

D11 D12 : : : D1m S1 3 2 1 3 6 DT 76 27 6 12 D22 : : : D2m S2 7 6  7 6 7 76 6 6 6 4

with

Dkk = diag(

X

(1;j )2I

.. .

.. .

.. .

.. .

76 76 76 m5 4

D1Tm D2Tm : : : Dmm S S1T S2T : : : SmT E

(wjk )2;

X

(2;j )2I

(Rk)i =

(i;j )2I

7 7 m7 5

 

(S (Y) )ij wjk

(

wjk if (i; j ) 2 I 0

else ; E = diag(

?R1 3 6 ?R 7 6 27 6 7

= 66 ... 6

(wjk )2; : : : ) ; Dkl = diag(

(Sk )ij = X

.. .

2

?R

4

0

X

(1;j )2I

X

1;

7 7 7 m5

wjkwjl ;

X

(1;j )2I (2;j )2I

X

(2;j )2I

wjkwjl ; : : : )

1; : : : ):

The proof of this theorem is performed by a standard minimization technique, using Lagrange multipliers. It may happen that the matrix in the equation system does not have an inverse, due to linear dependencies among the constraints. This can be taken care of by using pseudoinverses instead. The last step is to nd a point con guration which has this new matrix, S~(Y) , as its S -matrix. This can be done using the following theorem Theorem 7.2. The minimization problem min

n ? X

(y1i ? y~1i )2 + (y2i ? y~2i )2




i=1 ~ ( Y) (Y) under the constraint S = S~ has the solution y~k = yk ? S~(Y)((S~(Y))T S~(Y) )?1(S~(Y))T xk;

k = 1; 2

where xk is a vector with components xik ; i = 1; : : : ; n. This theorem too is proved by a standard minimization procedure. Again pseudoinverses may be required. With these three steps one can nd a point con guration close to the original one, having the property that it is a correct image of a piecewise planar object, if there are only second order syzygies.

Example 7.1. We will show some examples of how the previous theorems can be used to

correct slightly incorrect images. The left planar composite con guration in Figure 8 shows a simple image of a truncated pyramid ( lled lines). The planar composite con guration is incorrect because the three nearly vertical lines do not meet in a common point. In the same gure is drawn a corrected planar image (dotted lines) where now the three lines meets in one point. As can be seen, only small displacements of the vertices are needed to obtain a correct image. Here it is assumed that the four points with coordinates (0; 4), (4; 15), (9; 14) and (11; 5) are contained in a common plane. This means that the planar composite con guration is topologically equivalent to the one in Figure 2. The right planar composite con guration in Figure 8 shows another incorrect image ( lled lines). In order to be correct the nearly horizontal lines must be collinear. The corrected image (dotted lines) is again close to the original one. We can also see that the points on the common line of intersection have been moved more than the other points.  23

16

18

14

16 14

12 12 10 10 8

8

6

6

4

4 2

2

0 0

0

2

4

6

8

10

12

0

5

10

15

20

Figure 8: Two incorrect composite con gurations

8 A Comparison with Sugiharas Method In this section we will make same comments of how our method is related to that Sugihara has presented in [Sugihara 86]. There are some similarities as well as some dierences. Sugihara starts with a consistently labelled line-drawing where each line in the image is labelled as convex, concave or occluding. This step is not needed in our method, however what is needed is the composite con guration and knowledge of where occlusion occur in order to build up the S -matrix and the inequality constraints. The composite con guration contains information about which points must by coplanar in every preimage. We use the convention (also used by Sugihara) that every 'T'-crossing of lines represents an occlusion. We can look at the next step in the algorithm as telling us if this initial interpretation of the image is consistent with the laws of projective geometry. The next step in Sugiharas method is to introduce variables for the equation for each plane in the image: aj x + bj y + z + cj = 0 which indicates that the point with coordinates (x; y) in the image has coordinates (x; y ; z) in the preimage and lies on plane number j . This is true for orthogonal projection on the xy-plane and for perspective transformation also, with some modi cation, see [Sugihara 86]. Gathering all these equation yields a system of linear equations: Aw = 0 where A depends on the coordinates of the points in the image and w contains the unknown z-coordinate and the unknown surface variables. Then the occlusions can be gathered in a similar equation: Bw > 0 where w is as before and B is built up by the coordinates in the image. It turns out that the variables for the dierent faces can be eliminated because every face that impose some restriction on the coordinates has at least four points and then there are at least four equations involving the parameters describing the plane. Then equations with only the z -coordinates as unknown are left. These equations are exactly the same as in

T S Y 24

where the inverse depths are identi ed with the z -coordinates. This can be shown by considering the four equations for four points on the same plane:

ax1 + by1 + z1 + c = 0 ax2 + by2 + z2 + c = 0 ax3 + by3 + z3 + c = 0 ax4 + by4 + z4 + c = 0 which can be written

2

z1 z2 z3 z4

3

7 h i i6 1 c a b 664x1 x1 x1 x1 775 = 0 0 0 0 : 1 2 3 4

h

y1 y2 y3 y4

Hence

2

z1 z2 z3 z4

3

6 7 det 664x1 x1 x1 x1 775 = 0: 1 2 3 4

If we expand by the rst row we get

y1 y2 y3 y4

1z1 + 2z2 + 3z3 + 4z4 = 0 where the i are 3x3 subdeterminants. This is exactly the shape of the four point con guration. The computation above is only valid for parallel projection onto the xy -plane and then all depths equal one. If we consider perspective transformations the zi above become inverse depths. The next step is to overcome superstrictness. Sugihara solves this by picking out points in the image until what is called a maximal generically reconstructible substructure remains. In this substructure there are no problems of superstrictness and he can start to solve it and then check if the deleted points are close to lie in the planes they were supposed to lie in. There is an ecient algorithm to pick out this substructure, see [Imai], but one disadvantage is that all point in the image are not used in the same way. We have used a dierent approach and have tried to correct the S -matrix , so that all subsets of the columns obeys the rank condition. In order to do this we have to nd all syzygies of order two and higher. This is a dicult problem and is yet not solved completely. Our solution holds for the common situation of images with only second order syzygies. Then Sugihara solves the set of equations given above by the algorithm used for nding a feasible starting point in a linear programming problem with these equations as restrictions. This is a rather complex algorithm, having both linear equations and inequalities. We have fewer unknown variables because the surface variables are not present and we have used another approach to solve the problem. First we nd all solutions to the system of equations (this is rather easy) and then try to nd if there are some linear combinations obeying the inequality constraints. This is a standard algorithm for nding a point in a convex set after some calculations. This algorithm too is rather complex. The last step is to correct the image if possible. Sugihara just checks if the removed points obey the inequality constraints and if they nearly obeys the equality constraints. In our approach the S -matrix is already modi ed, and we can nd points as close as possible to the original points having a modi ed S -matrix. Above we have described one such algorithm. Below we will give another one based on assumptions on how noise enters the system. 25

9 Taking Noise into Account So far we have not used any model of how noise enters the system. In the following sections this will be done as well as the introduction of a new correction algorithm which uses the information about the noise. For the correspondence problem related investigations of noise have been done in [Grimson 92]. In this section we outline a theory of how the shape of a four point con guration is aected by noise. The shape of a four point con guration, as de ned in Section 2, is one dimensional, which means that if 1 2 s(X ) and 2 2 s(X ) then 1 = 2 for some  2 R. By abuse of notation we write s(X ) =  . In order to get a unique shape-vector assigned to the four points we impose the condition 4 X jjs(X )jj = (i)2 = 1: i=1

This will be done in the rest of this paper. We will assume that the noise is normally distributed with mean zero and variance . This means that xi = xi0 + i; i 2 N (0; ) where the i :s are independent, analogously for y i . This is the case if the image is distorted by a two-dimensional normally distributed noise. The formula can equivalently be written

xi 2 N (xi0; ): Our objective is to determine the distribution of the shape-vector. The following lemma is basic.

Lemma 9.1. Let
xi 2 N (0; ), P4i=1 i = 0 and P4i=1(i)2 = 1 (which is necessary for  to be a shape-vector). Then

4 X i=1

Proof: 4 X i=1

i
xi 2 N (0; ):


xi 2 N (0; ) ) i
xi 2 N (0; jij) ) q

i
xi 2 N (0; (1)2 + (2)2 + (3)2 + (4)2) = N (0; )

If X is a planar, noncollinear four point con guration and s(X ) is its shape, then by de nition

Xes(X ) = 0 where

"

#

Xe = x11 x12 x13 x14 : If Xe is disturbed by an amount
Xe ,  = s(X ) will also be disturbed, say by
 . This can be written as (Xe +
Xe )( +
 ) = 0 ) Xe + Xe
 +
Xe +
Xe
 = 0 ) 26

Xe
 +
Xe +
Xe
 = 0: If the disturbances are small compared to Xe and  ,we get Xe
 +
Xe = 0: The matrix
Xe may be written 3

2

0 0 0 0
Xe = 64
x1
x2
x3
x475
y 1
y 2
y 3
y 4 Introducing eT = [1 1 1 1], xT = [x1 x2 x3 x4], y T = [y 1 y 2 y 3 y 4 ],
xT = [
x1
x2
x3
x4],
y T = [
y 1
y 2
y 3
y 4 ],  T = [1 2 3 4] and
 T = [
1
2
3
4 ], the following equations are obtained (5) (6) (7)

eT
 = 0 xT
 =
xT  = 1 2 N (0; ) yT
 =
yT  = 2 2 N (0; )

according to the previous lemma, where 1 and 2 are independent. This is just three equations but
 has four components so a fourth one is needed. We haven't used the fact that jj jj = 1, which should hold also after the disturbance of the equations. This gives: ( +
 )T ( +
 ) = 1 )

T  + T
 +
T  +
T
 = 1 ) 1 + 2 T
 +
 T
 = 1 ) T
 = 0 where we have used the fact that
 is small compared to  . This gives a fourth equation: (8)

T
 = 0:

In order to solve these equations for
 we make use of the fact that a translational change of the coordinate system dos not change the shape-vector, nor the impact of noise. This means that we can chose the center of the coordinate system so that it is situated at the mass center of the point con guration. This means that it is no restriction to add the additional equations (9)

eT x = 0 ; eT y = 0:

The idea now is rst to solve the system for 1 =  in the case 2 = 0, then for 2 =  in the case 1 = 0 and then, nally, merge these two solutions together. This can be done because the equations are linear in
 and 1 and 2 are normally distributed. More speci cly this can be written as follows.

Lemma 9.2. If 1 2 N (0; ) and the solution to (5)-(9) with 1 =  = constant is
 = v1, then the solution to (5)-(9) is
 2 v1N (0; 1)

27

Proof: This is a fundamental property of the normal distribution, which says that a normally distributed variable is completely determined by its mean and variance. In this lemma the mean value is zero and the variance can be computed directly because of the linear character of the equations.

The linearity of the equations also means that the solutions are independent. We rst have to solve eT x = 0; eT y = 0; eT
 = 0; xT
 = ; yT
 = 0; T
 = 0: It is advantageous to see this geometrically as follows. We can interpret the shape of a four point P con guration as a point in R4. Since 4i=1 i = 0 the set of shapes of four point con gurations can be interpreted as a three dimensional hyperplane in R4 de ned by eT  = 0. Also
 , x and y belong to this hyperplane. We can also see that
 is orthogonal to y and  and has scalar product  with x. This observation,
 being orthogonal to y and  , leads us to search for solutions of the form T
s = (x ? xy T yy y ) because it is a constant times the x-vector minus its projection on the y -vector and thus perpendicular to y . It is also perpendicular to e because both x and y have this property. What remains is to determine the constant , which can be done with the equation xT
 = : 2 2 T 2 1 xT
 =  jxj jyj jy?j2(x y) =  )  = jxj2 1 ? ( jxxTjjyyj )2

If  is the angle between the vectors x and y , cos  = jxxjjyyj , this can be written T

1 1 y x  = jxj2 sin 2  )
 =  jxj sin2  ( jy j ? cos  jxj ): It can easily be showed that this vector is perpendicular to e and y and because xT
 =  this is the unique solution. The length of this vector is j
j =  jx1j j sin1 j From this formula can be seen that the magnitude of the disturbance in  depends on three factors. First the standard deviation of the noise. Second it is inversely proportional to jxj, which means that large con gurations (in the x-direction) are less sensitive to noise. The third factor has something to do with the relative location of the points. It can be noted that if the four points are collinear, then  = 0 and sin  = 0, which makes the con guration extremely sensitive to noise. The conclusion is that the con guration is more sensitive if the points are almost collinear. The same computation can be done with 1 = 0 and 2 = , in which case analogous formulas are obtained with x replaced by y . Let
x and
y denote the solutions obtained above. Then
xT
y = ? cos  = cos( ? ): j
xjj
yj Hence the angle between
x and
y is  ? , which is the same as  because the orientation is irrelevant. 28

We have now seen the separate impacts of noise in the x-direction and noise in the y direction. The problem now is that
x and
y are independent stochastic variables, but they are not perpendicular. Hence the distribution of
s(X ) can be determined if we compute the covariance matrix of
s(X ) = v x 1 + v y 2 where 1 and 2 are independent normally distributed variables with variance ,
x = v x and
y = v y . This gives us the nal theorem.

Theorem 9.3. Let i 2 N (0; 1); i = 1; 2 (independent), , x, and y be vectors as de ned above and e = [1111]T . Then

eT x = 0; eT y = 0; eT
 = 0; xT
 = 1; yT
 = 2; T
 = 0 has the solution


 =



y

x

x



y

jxj sin2  ( jyj ? cos  jxj )1 + jyj sin2  ( jxj ? cos  jyj )2 =
x1 +
y 2

and the covariance matrix is

C (
;
) = E (

T ) =
x
xT +
y
yT . Proof:

The covariance matrix can be computed straight ahead

C (
;
) = E (

T ) = E ((
x1 +
y 2)(
x1 +
y 2)T ) = = E (
x
xT 21 ) + E (
x
yT 1 2 ) + E (
y
xT 2 1 ) + E (
y
yT 22 ) = =
x
xT E (21) +
x
yT E (12 ) +
y
xT E (21 ) +
y
yT E (22) = =
x
xT +
y
yT since 1 and 2 are independent.

10 A New Correction Algorithm For a planar composite con guration, bases of every face can be described by means of fourpoint con gurations. Hence the results of Section 9 can be applied to compute the distribution of the S -matrix.

Theorem 10.1. Let S (Y) be the S -matrix of a planar composite con guration, (Y ) and  be the

variance of the noise aecting it. Further let each column in the S -matrix have the distribution
si = vixxi + viy yi with xi ; yi 2 N (0; ). Then the covariance matrices for the x -vector and the y -vector are 2(S (Y))T S (Y). Proof: Given xi and xj as above, their covariance can be computed using, (6),
xi 2 N (0; ) and that they are independent:

C (xi; xj)

= C( =

n X

ski
xk;

k=1 n n XX k=1 l=1

n X l=1

slj
xl ) =

kl2skislj =

n X k=1

29

n X n X k=1 l=1

C (ski
xk; slj
xl) =

2 skiskj = 2((S (Y))T S (Y))ij ;

where sij denotes the element in position i; j in S (Y) The same calculations can be done for y using (7) which gives the same expression. This suggest a new correction method which minimizes (x )T (S (Y) )T S (Y) x + (y )T (S (Y) )T S (Y) y under the linear constraints (S (Y) + V x diag(x ) + V y diag(y ))wi = 0 where V x = [v1xv2x : : :vmx ] and V y = [v1y v2y : : :vmy ]. A problem here is that it is not certain that a number of given linear combinations can be zero by just modifying each column in the S -matrix in two directions. Therefore we have to add a new degree of freedom. The only degree left is in the original direction of the column (because the sum of the components must be zero). This extra degree of freedom can be interpreted as a way to modify the initial linear combinations. This gives us a new function to minimize: (x )T (S (Y))T S (Y) x + (y )T (S (Y) )T S (Y) y + (s )T diag(i )s under the linear constraints (S (Y) + V x diag(x ) + V y diag(y ) + S (Y) diag(s ))wi = 0: The new variables  must be included in the loss function because otherwise the solution is x = y = 0 and s = ?e = ?(1; 1; : : : ; 1). The new parameters  must be chosen appropriately. We have found that i = 1 works well. The following theorem tells us the solution to this optimization problem.

Theorem 10.2. The solution to the optimization problem

min((x )T (S (Y) )T S (Y) x + (y )T (S (Y) )T S (Y) y + (s )T diag(i )s ) under the constraints (S (Y) + V x diag(x ) + V y diag(y ) + S (Y) diag(s ))wi = 0 is obtained from the linear system of equations Ax = b with 2 (S (Y))T S (Y) 0 0 (V x diag(w1))T : : : (V x diag(wm ))T 3 6 0 (S (Y) )T S (Y) 0 (V y diag(w1))T : : : (V y diag(wm ))T 77 6 6 0 0 diag() (S (Y) diag(w1))T : : : (S (Y) diag(wm ))T 777 6 A = 66 V xdiag(w1) V y diag(w1) S X diag(w1) 7 0 ::: 0 7 6 . . . . . 7 6 .. .. .. .. .. 5 4 V xdiag(wm) V y diag(wm) S (Y) diag(wm) 0 ::: 0 2 2 x3 0 3  6 6 y 7 0 77 6 7 6 6 6 s7 0 777 6 6 7 6 7 6 x = 6 1 7 b = 6 S (Y)w1 7 : 6 7 6 .. 775 6 6 .. 7 4 4 . 5 .

S (Y)wm

m

30

The proof of this theorem is performed by a standard minimization technique using Lagrange multipliers (i ). The new S -matrix is given by

S (Y) + V x diag(x) + V y diag(y ) + S (Y) diag(s)

Example 10.1. Figure 9a shows an image of a truncated cube. An edge detection algorithm

has been used to get the coordinates of the points, see Figure 9b. The points have the following coordinates, " # 125 72 112 165 194 70 142 142 191 37 57 65 64 53 145 162 116 137 where the x-coordinates are in the rst row and the y -coordinates are in the second. The S matrix from these points does not obey the rank condition but has a singular value close to zero. The algorithm applied to this S -matrix gives the following new coordinates, "

#

126:37 71:72 112:81 164:04 192:67 68:65 142:59 142:85 191:29 37:48 57:53 65:47 63:16 51:98 143:84 161:91 116:75 137:88

which represent a correct planar composite con guration.

Figure 9: A box with a corner removed and the corresponding line-drawing



11 Conclusions In this paper we have presented criteria for the correctness of an image of a piecewise planar object. The criterion weak correctness, involving the S -matrix, seems to be easier to work with than previous approaches. We have also presented a combinatorial condition for the weak correctness of an image. Further the degree of freedom in an image have been examined by matroid theory. Two algorithms for correcting slightly incorrect images of piecewise planar 31

objects are given. One uses a model of how noise enters the system and disturbs the S -matrix. The results of experiments are intuitively reasonable, as is illustrated in examples. The methods are able to generate interpretations of line-drawings in many, but not all, cases. For a complete solution, further work on nding all higher order syzygies of the S -matrix has to be done.

32

References [Crapo] Crapo H.H. and Rota G.-C., Combinatorial Geometries, M.I.T. Press, 1970. [Grimson 92] Grimson, W. E. L., Huttenlocher, D. P. and Jacobs, D. W., A Study of Ane Matching With Bounded Sensor Error, ECCV'92, Lecture notes in Computer Science, Vol 588. Ed. G. Sandini, Springer-Verlag 1992, pp. 291-306. [Guzman] Guzman, A, Decomposition of a Visual Scene into Three-dimensional Bodies, FJCC, Vol 33, part 1, pp 291-304. [Heyden] Heyden, A., Methods for Correction of Images of Origami/Polyhedral Objects, To appear in Proceedings of Scandinavian Conference on Image Analysis, Troms, Norway, 1993. [Imai] Imai, Hiroshi, On Combinatorial Structures of Line Drawings of Polyhedra, Discrete Applied Mathematics 10 (1985) 79-92, North Holland, 1985. [Oxley] Oxley, James G., Matroid Theory, Oxford University Press, Oxford, New York, Tokyo, 1992. [Penrose] Penrose R. On the Cohomology of Impossible Figures, Structural Topology, nr 17, 1991, pp. 11-16. [Persson 91] Persson, A., A Method for Correction of Images of Origami/Polyhedral Objects, Proc. Symposium on Image Analysis, SSAB, Uppsala, Sweden, 1992, pp. 93-96. [Schrijver] Schrijver, A., Theory for Linear and Integer Programming, John Wiley & sons, Chichester, New York, Brisbane, Toronto, Singapore, 1986. [Sparr 91] Sparr, G., Projective Invariants for Ane Shapes of Point Con gurations, ESPRIT/DARPA Invariants Workshop, Reykjavik, Iceland, 1991. [Sparr 92a] Sparr, G., Reconstruction of Impossible Objects, Proc. Symposium on Image Analysis, SSAB, Uppsala, Sweden, 1992, pp. 109-112. [Sparr 92b] Sparr, G., Depth-Computations from Polyhedral Images, ECCV'92, Lecture notes in Computer Science, Vol 588. Ed. G. Sandini, Springer-Verlag 1992, pp. 378-386. Also in Image and Vision Computing, Vol 10. 1992, pp. 683-688. [Sturmfels] Sturmfels, B., Computational Algebraic Geometry of Projective Con gurations, Institute for Mathematics and its Applications, University of Minnesota. [Sugihara 86] Sugihara, K., Machine Interpretation of Line Drawings, MIT Press, Cambridge, Massachusets, London, England, 1986. [Welsh] Welsh D.J.A., Matroid Theory, Academic Press, London, New York, San Fransisco, 1976. [White 86] White, N. Theory of Matroids, Encyclopedia of mathematics and its application, Cambridge University Press, Cambridge, London, New York, New Rochelle, Melbourne, Sydney, 1986. [White 87] White, N. Combinatorial Geometries, Encyclopedia of mathematics and its application, Cambridge University Press, Cambridge, New York, New Rochelle, Melbourne, Sydney, 1987. 33

[Whiteley] Whiteley, W. Some Matroids on Hypergraphs, with Applications in Scene Analysis and Geometry, Discrete and Computational Geometry, 4:75-95, 1989.

34