Shocks From Images: Propagation of Orientation Elements
Shocks From Images: Propagation of Orientation Elements Huseyin Tek, Perry A. Stoll, and Benjamin B. Kimia LEMS, Division of Engineering, Brown University Providence RI 02912
Abstract
The extraction of gure symmetry from image contours faces a number of fundamental diculties: object symmetries are distorted due to (i) gaps in the bounding contour of a shape due to gure-ground blending, weak contrast edges, highlights, noise, etc.; (ii) an introduction of parts and occluders, and (iii) spurious edge elements due to surface markings, texture, etc. A framework for extracting such symmetries from real images is proposed based on the propagation of contour orientation information and the detection of four types of singularities (shocks) arising from the collision of propagating elements. In this paper, we show that an additional labeling of shocks based on whether the colliding wavefronts carry true orientation information (regular vs. rarefaction waves) allows a division of shocks into three sets: regular shocks are the partial shocks of partial contours as they remain invariant to the completion of the contour; semi-degenerate and degenerate shocks depict potential parts and gaps. Finally, shocks altered due to spurious edges, occlusion, and gaps are recovered via a simulation of inter-penetrating waves generated at select shock groups which with the aid of the above shock labels leads to second and further generations of shocks.
1 Introduction
The task of gure/ground segmentation is dicult primarily because low-level processes do not yield perfect information about the gure's bounding contour or interior, due to gaps or weak contrast edges arising from gure-ground blending, partial occlusion, specularity highlights, noise, etc., Figure 1a. For example, the extraction of symmetries (shocks) is ill-conditioned in that the removal of a small portion of the boundary introduces large changes in object symmetry, Figure 2. In addition to missing contours, a gure's bounding contour is often altered when one of its parts becomes visible with a viewpoint change, or due to occlusion, Figure 1b, leading to substantial changes in the resulting symmetries Figure 3. Yet another example of
G
M G
S G
H
S G
(a)
S T
(b)
Figure 1: Diculties in extracting shocks from images:
(a) Edge maps do not always yield a perfect bounding contour for objects due to gaps (G), surface markings (M), highlights (H), shadows (S), and noise. (b) A sketch of some cases of occlusion where the gure (elephant), background, and occluders stand out (top), where the occluder and the gure blend in (middle), and where the occluder and background blend in (bottom).
imperfect low-level data is the set of spurious edges arising from surface markings, texture edges, noise, etc., which are spurious in the sense that they make object boundaries ambiguous, Figure 1a, and result in distorted symmetries, Figure 4. The key idea in this paper in dealing with these diculties is the observation that in the propagationbased extraction of symmetries, each orientation element, or contour segment, emanates only two types of waves: (i) regular waves whose front orientation is re ective of the initial source orientation, and (ii) rarefaction waves whose front orientation is \bogus" in the sense that it does not re ect propagation of an initial orientation. Three types of shocks can therefore be distinguished based on this distinction between regular and rarefaction waves: regular, semidegenerate, and degenerate shocks arise from the interaction of regular-regular, regular-rarefaction, and rarefaction/rarefaction waves, respectively. This notion of shock labeling has the following implications. First, it is shown that regular shocks are the shocks that remain intact when the partial boundary is completed.
Figure 2: The shocks of partial contours do not bear much resemblance to the shocks of complete contour, leading to a fundamental instability in computing shocks from edge maps.
(a)
b)
Figure 3: The e ect of occlusion boundaries and parts: (a)
a blend-in occlusion or a part distort the rectangle in symmetry. (b) a stand out occlusion splitting the rectangle; the complete rectangle's skeleton is no longer recognizable from the skeletal representations of the white areas.
Second, shock labels reveal the anatomy of parts: regular shocks remain invariant following a partitioning of the gure; semi-degenerate shocks are altered to recover each part's symmetries; degenerate shocks do not convey shape information, but indicate the attachment of two shapes. Third, based on the idea that partitioning the background is tantamount to grouping pieces of the foreground [10], Figure 5, a gap may be viewed as a part and thus closed by partitioning (background) symmetries as described above. Fourth, shock labels combined with the idea of interpenetrating waves and multiple generation of shocks [13] aid in reconstructing symmetries which are distorted due to spurious elements, occlusion, and gaps. A number of other approaches have advocated extracting symmetries directly from grey-scale images. Scott et al. [7] propagate waves to recover the full symmetries. They also suggest a convolution approach for implementing the full symmetry set. Pizer et al. [6] use a similar approach where by a voting scheme, edges measured at each scale vote for medialness at a point
(a)
(b)
(c)
Figure 4: (a) Rectangle with spurious edge, and (d) skele-
ton of rectangle with spurious edge element. (c) full symmetry set: relevant symmetries are now embedded in a large set of symmetries.
Figure 5: A partitioning of the background (white) leads to a grouping of the foreground and shape segments [10]
which is a constant proportion of scale away. The ridges of the resulting surface constitute the core, a skeleton in x, y and (scale). Kelly and Levine [2] use annular symmetry operators in a similar fashion to derive the full symmetry set. Tari and Shah [12] de ne symmetries as the curvature maxima of level sets which have been constructed consistently with an edge strength functional. August et al. [1] use curve evolution on dilated contour fragments obtained from a depth-segregated edge map and use the notion of a gap skeleton to group certain nearby endpoints.
2 Shock detection and classi cation
Shape can be completely described as the collection of four types of shocks which form in the course of deformations of shape in the reaction-di usion space [4, 5, 9] @C @t
~ = ( 0 ? 1 )N:
(1)
The four types of shocks correspond to intuitive elements of shape, namely, parts, protrusions, and bends [3]. The deformations are implemented via the curve evolution paradigm by embedding the curves C (s; t) as the@ level set of a surface f (x; y; t) = 0g evolving by @t = ( 0 ? 1 )jr j. Table 1 shows a classi cation of shocks from which has been implemented to sub-pixel accuracy [9]. While the derived shock structure when 1 = 0 is related to skeletons, several properties of shocks relating to the notions of type, velocity, grouping, salience, and hierarchy are signi cant. (i) certain deformations, e.g., bending, affect shock types selectively, e.g., third-order shocks of a rectangle; (ii) the skeletal representation lacks sucient explicit dimensions for qualitative approximation, e.g., the addition of shock type substantially narrows down the range of shapes it can generate; (iii) topological and di erential properties of shocks, e.g., velocity, directly re ect boundary properties; (iv) the notion of time of formation induces a hierarchy on shocks; and (v) a notion of shock grouping and salience based on the di usion process ( 1 6= 0) leading to a stability of representation with small changes. It was
1π/4
θ
1 2 3 4
First−Order Shock
2
Fourth−Order Shock
P P
Figure 6: Each of the four types of shocks each is correlated with a perceptual/semantic category, i.e., protrusion, part, bend, and seed. Shock Type
Orientation
First−Order
non− vanishing
1 2 3 4
1 −π/4
−π/4
3
Third−Order Shocks
π/4 s
2 Second−Order Shock
(a)
4 θ s
(b)
3 4
Figure 7: The propagation of waves leads to a hyperbolic PDE for propagation orientation and formation of shocks. (a) formation of a shock (b) formation of a rarefaction wave. The left side of each shows the orientation along the contour; the right side shows the corresponding contour.
Curvature
∆φ
high level set curvature
∆φ
Second−Order
isolated vanishing
Third−Order
non−isolated vanishing
Fourth−Order
isolated vanishing
∆φ
∆φ
κ1 κ2 < 0 κ1 κ2 = 0 κ1 κ2 > 0
Table 1: This table depicts the classification of shock types based on the the gradient level set curvature and the principal curvatures of the surface.
proposed in [13] that the contour-based distance transform (CEDT) is an alternative, ecient paradigm for implementing wave propagation.
3 Shock labeling
A generalization of the shock-based representation of a binary shape to contours extracted from real images requires a notion of shock labels. The key insight underlying shock labeling is an understanding of the evolution of local orientation information during curve deformation and wave propagation. Two signi cant events were observed in [5]. First, the constant deformation of shape is equivalent to the propagation of orientation by a conservative hyperbolic PDE where \orientation-elements" ow as a function of their orientation. The formation of a shock is illustrated in Figure 7a when the = 4 portion ows toward and collides with the = ? 4 portion and in the process annihilating all other orientation elements. Second, a rarefaction wave forms when two orientation elements
ow away from each other as in Figure 7b and create a gap of orientation. The conservation of orientation implies that the gap generated thus should be lled in linearly leading to a circular rarefaction wave emanating from the singular point [5]. The generation of rarefaction waves as applied to propagation of orientation from an edge map implies wavefronts whose orientation is \bogus" in that it does not re ect a propagation of some initial orientation, Figure 8. Thus, while shocks arising from such points are necessary for a complete representation of shape, they do not re ect symmetry between two contour segments. Therefore, an additional labeling of rarefaction fronts allows for proper propagation of contour segments, Figures 8c and 8a. Formally, De nition 1 Regular wavefronts are those points of
the wavefronts which arise from contour points with
regular tangents. Rarefaction or degenerate wavefronts arise from points having no or multiply de ned tangents.
The interaction of these two types of waves lead to three types of shocks. De nition 2 A shock point arising from two regular
waves is regular. A shock point arising from one regular and one degenerate wavefront is semi-degenerate. A shock point formed from the interaction of two degenerate wavefronts is degenerate.
Note that a labeling of shocks into these three classes is independent of a classi cation of shocks into the four types described earlier. We employ two computational paradigms to detect, classify, group and label shocks, namely curve evolution and CEDT. The algorithm for labeling shocks is essentially the same in both paradigms: label each wavefront by tracing back to the source of propagation and measure the length of the wavefront arising from the source. For a rarefaction wave, a small length initial contour gives rise to a large length on the front; the ratio when normalized by the absolute time of evolution is a fanout ratio, Figure 12. Note that no explicit a priori detection of corners or endpoints is required. We have implemented this algorithm and applied it to the labeling of shocks of partial contours, Figures 9 and 10. Observe how the regular shocks properly depict the symmetry of the object. The details of the algorithms and a comparison of the two computational paradigms can be found in [11].
4 Shock Labels Depict Part Anatomy
Partitioning prior to gure-ground segregation can aid in the segmentation process. It is now argued that shock labels provide evidence for parts in the result of low-level processes. A theory for partitioning visual form was proposed in [8] leading to two types of parts: one at necks, or certain locally shortest lines through the shape, and limbs, or curves through two negative curvature minima that smoothly interpolate
3
2
2 1 0
(a)
Regular
4 3
4
1 0
Initial Orientation Element
1 2 3
4
(b)
(c)
Rarefaction
Figure 8: (a) Ideal propagation of orientation element for
shocks representing contour symmetry. (b) Wave-based propagation of orientation includes additional fronts, rarefaction waves are depicted as dashed lines in (c), which give rise to additional (seemingly spurious) shocks when fronts collide.
(a)
(c)
Figure 10: Labeling of shocks into three classes, regular
(green), semi-degenerate shocks (yellow), and degenerate shocks (red) for an incomplete rectangle, (a) and for another incomplete shape (b).
Figure 11: Two types of parts: limbs (left) and necks (middle). The limbs and necks of the sh shape (right) are shown [8].
(a)
(b)
(c)
Figure 9: (a) The interaction of regular waves from two
lines leads to regular shocks. (b) semi-degenerate shocks arise from a rarefaction wave meeting a regular wave. (c) degenerate shocks arise from two degenerate (rarefaction) waves.
the tangent at one point to the tangent at another, Figure 11. The computation of each candidate partline involves computing a measure of salience which is then used to resolve con icts when they arise [8]. While this framework is developed for partitioning segmented shape, the computations are local with respect to the part-line size and can thus be computed prior to gure-ground segmentation. We now suggest that a reformulation of this approach in the shock language using shock labels allows parts to be computed directly from images. While neck-based parts correspond to second-order shocks, the connection between limb-based parts and shocks is more subtle and requires a close examination of the anatomy of limb-based parts with respect to shock labels. Figure 13 illustrates that the two sharp negative curvature minima generate rarefaction waves leading to three sets of \irregular" shocks: i) the stem: the semi-degenerate shocks arising from the interaction of rarefaction waves from A and B with
the regular waves from the body of the shape, ii) the attachment: the degenerate shocks arising from the interaction of rarefaction waves from A and B, and iii) the join: the semi-degenerate shocks arising from the interaction of rarefaction waves from B with the regular waves from the part. Thus, a labeling of shocks in general divides the shock-based representation of a shape into ve distinct groups: the above three groups and iv) the base: the regular shocks pertaining to the main shape itself, and v) the limb: the regular shocks pertaining to the limb part. Figure 13d illustrates how the body and the stem shock groups identi ed by labels remain invariant with movement and change of the part shape. Figure 14a shows actual computation of shocks and shock labels. These labels provide a strategy for partitioning shape and extracting each component's shock-based representation: since the attachment, join, and stem do not re ect each component's symmetry but rather depict the anatomy of the connection itself, propagate waves from a hypothesized limb part-line (a) which then interact with the degenerate and semi-degenerate shocks and produce each component's shock-based representation correctly, Figure 14b. Observe that degenerate shocks are not always present, Figure 14c, but can be detected via multiple generation shocks; see Section 5. The shock labeling of part anatomy also has implications for gaps. It was proposed in [10] that gaps may be closed by a partitioning of the background which produces a grouping of the foreground, Figure 5, the case of an occluder which blends with the back-
(a) (b) Figure 12: The generalization of the notion of a rarefac-
tion wave (a) based on a measure of fanout which indicates the reliability of orientation estimates at the wavefront (b).
(a)
(b)
(c)
(d)
Limb
Attachment
Base
Join A
(a)
B
Stem
(b) Figure 13: Sharp negative curvature extrema A and B (a)
generate rarefaction waves leading to ve distinct groups of shocks identi ed by a labeling of these shocks into regular (green: base and limb), semi-degenerate (yellow: stem and joint) and degenerate (red: attachment). Observe that variations in the part shape leave the base and stem invariant. Observe how a gap is viewed as a \null part," a construction originally proposed in [10].
ground as in Figure 1b. In the case of a null part, or a part with no limb, Figure 13d, partitioning the degenerate shocks leads to successful closure of the shape. Figure 12 illustrates the need for generalizing the notion of a rarefaction wave using a fanout ratio to include scale. Note that the shape of Figure 13c when the sharp points A and B are smoothed out slightly would still be perceived as a composite shape, yet no rarefaction waves form in the strict sense. A similar situation hold in the case of a noisy contour with a corner, Figure 12b. However, observe that a similar phenomenon occurs since the orientation of the evolving front emanating from these \coarse-scale" corners is unreliable, and thus, must be marked as such. The \fan-out" ratio described earlier accomplishes this task. This quantity for a wave emanating from a corner or end point is in nite, small for a line, and high for a sharply curved contour. This generalization of the binary rarefaction measure leads to stability against perturbations, Figure 12. Observe also that in the limit as the part and shape smoothly blend together, the perception of a part gives way to the perception of a protrusion. The generalized rarefaction wave thus captures the part-protrusion continuum [3].
Figure 14: (a) Degenerate and semi-degenerate shocks
hypothesize a limb-based part. (b) Secondary waves from this part-line correct the irregular shocks to form the symmetry set for each component. (a). While degenerate shocks are not always evident, second generation waves launched at irregular shocks will bring them out (d). See also Figure 17. Arrays indicate transitions.
5 Inter-penetrating Waves and Multiple Generation Shocks
We have shown that shock labels identify true contour symmetries via regular shocks, as well as parts and gaps via a combination of semi-degenerate and degenerate shocks. In a companion paper [13] it is proposed that a new generation of waves (interpenerating waves) be launched from select shock groups identi ed by shock labels and the resulting shocks (multiple generation shocks) recorded. The main idea can be motivated by observing how the spurious edge element in Figure 4 alter the underlying symmetries. Waves from this element quench waves from the gure boundary and thus interfere with the proper formation of gure symmetry. The appropriate symmetries exist in the full symmetry set, but are embedded in a large set of unintuitive ones. Instead, these can be recovered by launching waves at select groups of shocks [13]. Secondary waves launched at salient limbs correct the distorted symmetries, Figures 14, 16b, and 17, leading to a partitioning of the shape. Salience of a limb is partially related to the optimal shape of the completion contour, or limb part-line [8]. Figure 15 examines the relationship between salience and shock labels: observe how the length of the degenerate shock group
(a)
(b) Figure 18: Degenerate shock sets signal contour grouping,
some indicate the completion of triangle, while others indicate the completion of the pacman gures. First, observe that separation in depth which is assumed in [1] is not required here but rather follows from shock labeling and subsequent contour grouping. Second, this example illustrates that rarefaction waves generated by corner points are also signi cant. Arrays indicate transitions.
(c)
(d) Figure 15: The length of degenerate shocks signals the saliency of the part-line/completion contour.
varies with the \extent' of the part, and the \ misalignment" of two tangents, Figure 15. Observe how salience allows for distant, second generation grouping over rst generation ones, Figure 17(e-g). The latter example highlights the combined use of shock labels, interpenetrating waves, and multiple generation shocks. Figure 16 illustrates how partitioning and grouping are identical operations in the shock domain. Figure 16a is perceived as two overlapping triangles. As parts are removed one by one, Figures 16b and 16c, the rst triangle is recovered, resulting in two disjoint pieces of the second triangle. Subsequent removal of parts (background) leads to the grouping of two disjoint pieces. Waves from the pair of limb partlines, previously referred to as \hidden limbs" [8], now form the shocks corresponding to the second triangle. Observe that the notion of salience is crucial when the alignment and smoothness of the connecting limb part-line is challenged, Figure 16f.
6 On the connection to the gap skeleton
August et al. presented an interesting connection between certain portions of a skeleton, namely the gap skeleton and the skeleton of a virtual occluder which signals the grouping of contour fragment endpoints. The main idea is that occlusion leads to T-junction discontinuities. They propose that it is not the shape
of the optimal continuation curve at T-junctions that is signi cant, but rather the consistency of the grouping as the outcome of some virtual occluder. Thus, based on the assumption that extracted contours have been partially depth-ordered in separate maps based on T-junctions, they group end points with no contour present in the circular neighborhood whose diameter is the line segment joining the two endpoints. Expressed in the language of shocks, this implies a second-order shock and two rst-order shock groups (gap skeleton) which arise from the end points. Our approach bears some resemblance to this proposal where the main similarity is the expression of contour grouping in the language of shocks, speci cally that certain degenerate rst-order shock groups, arising from end points (not corners) and containing second-order shocks constitute the gap skeleton. However, there are a number of signi cant distinctions. First, the motivation and domain of application is rather di erent: in contrast to grouping contour fragments partially ordered in depth via T-junctions, our goal is to extract partial shocks from partial contours of the full set of extracted contours by separating extracted symmetries into classes: regular shocks represent true contour symmetry, while degenerate shocks represent connectivity. Thus, this approach achieves such a grouping without depth segmentation via limb hypothesis and inter-penetrating waves which in turn lead to depth segregation, Figure 18. August et al. require a separation of the triangular contours from the circular ones to achieve grouping, while we propose that a separation follows from the grouping. This gure also illustrates a second important dis-
(a)
(b)
(c)
(d)
(e)
(f)
Figure 16: (a-e) partitioning and grouping as identical operations lead to the recovery of two triangles. (f) the role of salience of part-lines in grouping when alignment is challenged.Arrays indicate transitions.
(a)
(b)
(c)
(d)
(e)
(f)
(g
Figure 17: (a) A shape with two parts and its shocks. (b) The recovery of individual part's shocks by shock labeling followed by a second generation shocks of waves initiated at limbs. (c) a shape with a gap (null part) and its shocks. (d) Second generation waves initiated at salient limbs remove the degenerate shocks and correct for the distortion at semi-degenerate shocks. (e) a spurious edge element's interference with contour grouping can be removed by considering multiple generation of shocks as shown in (f) where second generation shocks arising from the shocks loops (completed by the image boundary) generate a new grouping hypothesis thus completing the rectangle symmetries (g). tinction, namely, the concept of classi cation of a [4] B. B. Kimia, A. R. Tannenbaum, and S. W. Zucker. Toward a computational theory of shape: An wave based on whether the orientation of its front overview. In ECCV, Antibes, France, 1990. can be reliably related to the original front (regular) or not (rarefaction wave). Thus, corner points which [5] B. B. Kimia, A. R. Tannenbaum, and S. W. Zucker. Shapes, shocks, and deformations, I: The components generate rare-faction waves lead to degenerate shock of shape and the reaction-di usion space. Internagroups which are not gap skeletons. Third, we hold tional Journal of Computer Vision, 15:189{224, 1995. that the shape of the optimal contour connecting the [6] S. M. Pizer and C. A. Burbeck. Object representaend points is signi cant, indicating the salience of the tion by cores: Identifying and representing primitive grouping, e.g., in detecting alignments, Figure 15 and spatial regions. Vision Research, 1994. Figure 16. In fact, Figure 15d illustrates that the de[7] G. L. Scott, S. Turner, and A. Zisserman. Using a termination of whether a gap skeleton exists can be mixed wave/di usion process to elicit the symmetry viewed as a binary representation of the salience of the set. IVC, 7(1):63{70, February 1989. optimal contour completion. This leads to instabili[8] K. Siddiqi and B. B. Kimia. Parts of visual form: ties in that minute changes in contour orientation or Computational aspects. PAMI, 17(3):239{251, March relative position can remove the gap skeleton. In addi1995. tion, no dilation of the initial curve is needed. Finally, [9] K. Siddiqi and B. B. Kimia. A shock grammar for contour grouping via degenerate shock sets requires recognition. In CVPR, June 1996. a notion of inter-penetrating waves and saliency, Figure 17e-g. [10] K. Siddiqi, K. J. Tresness, and B. B. Kimia. On the anatomy of visual form. In IWVF, pages 507{521, Capri, Italy, May 1994. References [11] P. Stoll, H. Tek, and B. B. Kimia. Shocks from im[1] J. August, K. Siddiqi, and S. W. Zucker. Fragment ages: Propagation of orientation elements. Technical grouping via the principle of perceptual occlusion. In Report 153, LEMS, Brown University, October 1996. ICPR, 1996. [12] S. Tari, J. Shah, and H. Pie. A computationally ecient shape analysis via level sets. In IEEE Workshop [2] M. F. Kelly and M. D. Levine. Annular symmetry on Mathematical Methods in Biomedical Image Analoperators: A method for locating and describing obysis, 1996. jects. In ICCV, pages 1016{1021, 1995. [13] H. Tek, F. Leymarie, and B. B. Kimia. Multiple gen[3] B. B. Kimia, K. Siddiqi, A. R. Tannenbaum, and eration shock detection and labeling using CEDT. In S. W. Zucker. The shape triangle: Parts, protrusions, IWVF, Capri, Italy, May 1997. and bends. Vision Research, Submitted.
Abstract
The extraction of gure symmetry from image contours faces a number of fundamental diculties: object symmetries are distorted due to (i) gaps in the bounding contour of a shape due to gure-ground blending, weak contrast edges, highlights, noise, etc.; (ii) an introduction of parts and occluders, and (iii) spurious edge elements due to surface markings, texture, etc. A framework for extracting such symmetries from real images is proposed based on the propagation of contour orientation information and the detection of four types of singularities (shocks) arising from the collision of propagating elements. In this paper, we show that an additional labeling of shocks based on whether the colliding wavefronts carry true orientation information (regular vs. rarefaction waves) allows a division of shocks into three sets: regular shocks are the partial shocks of partial contours as they remain invariant to the completion of the contour; semi-degenerate and degenerate shocks depict potential parts and gaps. Finally, shocks altered due to spurious edges, occlusion, and gaps are recovered via a simulation of inter-penetrating waves generated at select shock groups which with the aid of the above shock labels leads to second and further generations of shocks.
1 Introduction
The task of gure/ground segmentation is dicult primarily because low-level processes do not yield perfect information about the gure's bounding contour or interior, due to gaps or weak contrast edges arising from gure-ground blending, partial occlusion, specularity highlights, noise, etc., Figure 1a. For example, the extraction of symmetries (shocks) is ill-conditioned in that the removal of a small portion of the boundary introduces large changes in object symmetry, Figure 2. In addition to missing contours, a gure's bounding contour is often altered when one of its parts becomes visible with a viewpoint change, or due to occlusion, Figure 1b, leading to substantial changes in the resulting symmetries Figure 3. Yet another example of
G
M G
S G
H
S G
(a)
S T
(b)
Figure 1: Diculties in extracting shocks from images:
(a) Edge maps do not always yield a perfect bounding contour for objects due to gaps (G), surface markings (M), highlights (H), shadows (S), and noise. (b) A sketch of some cases of occlusion where the gure (elephant), background, and occluders stand out (top), where the occluder and the gure blend in (middle), and where the occluder and background blend in (bottom).
imperfect low-level data is the set of spurious edges arising from surface markings, texture edges, noise, etc., which are spurious in the sense that they make object boundaries ambiguous, Figure 1a, and result in distorted symmetries, Figure 4. The key idea in this paper in dealing with these diculties is the observation that in the propagationbased extraction of symmetries, each orientation element, or contour segment, emanates only two types of waves: (i) regular waves whose front orientation is re ective of the initial source orientation, and (ii) rarefaction waves whose front orientation is \bogus" in the sense that it does not re ect propagation of an initial orientation. Three types of shocks can therefore be distinguished based on this distinction between regular and rarefaction waves: regular, semidegenerate, and degenerate shocks arise from the interaction of regular-regular, regular-rarefaction, and rarefaction/rarefaction waves, respectively. This notion of shock labeling has the following implications. First, it is shown that regular shocks are the shocks that remain intact when the partial boundary is completed.
Figure 2: The shocks of partial contours do not bear much resemblance to the shocks of complete contour, leading to a fundamental instability in computing shocks from edge maps.
(a)
b)
Figure 3: The e ect of occlusion boundaries and parts: (a)
a blend-in occlusion or a part distort the rectangle in symmetry. (b) a stand out occlusion splitting the rectangle; the complete rectangle's skeleton is no longer recognizable from the skeletal representations of the white areas.
Second, shock labels reveal the anatomy of parts: regular shocks remain invariant following a partitioning of the gure; semi-degenerate shocks are altered to recover each part's symmetries; degenerate shocks do not convey shape information, but indicate the attachment of two shapes. Third, based on the idea that partitioning the background is tantamount to grouping pieces of the foreground [10], Figure 5, a gap may be viewed as a part and thus closed by partitioning (background) symmetries as described above. Fourth, shock labels combined with the idea of interpenetrating waves and multiple generation of shocks [13] aid in reconstructing symmetries which are distorted due to spurious elements, occlusion, and gaps. A number of other approaches have advocated extracting symmetries directly from grey-scale images. Scott et al. [7] propagate waves to recover the full symmetries. They also suggest a convolution approach for implementing the full symmetry set. Pizer et al. [6] use a similar approach where by a voting scheme, edges measured at each scale vote for medialness at a point
(a)
(b)
(c)
Figure 4: (a) Rectangle with spurious edge, and (d) skele-
ton of rectangle with spurious edge element. (c) full symmetry set: relevant symmetries are now embedded in a large set of symmetries.
Figure 5: A partitioning of the background (white) leads to a grouping of the foreground and shape segments [10]
which is a constant proportion of scale away. The ridges of the resulting surface constitute the core, a skeleton in x, y and (scale). Kelly and Levine [2] use annular symmetry operators in a similar fashion to derive the full symmetry set. Tari and Shah [12] de ne symmetries as the curvature maxima of level sets which have been constructed consistently with an edge strength functional. August et al. [1] use curve evolution on dilated contour fragments obtained from a depth-segregated edge map and use the notion of a gap skeleton to group certain nearby endpoints.
2 Shock detection and classi cation
Shape can be completely described as the collection of four types of shocks which form in the course of deformations of shape in the reaction-di usion space [4, 5, 9] @C @t
~ = ( 0 ? 1 )N:
(1)
The four types of shocks correspond to intuitive elements of shape, namely, parts, protrusions, and bends [3]. The deformations are implemented via the curve evolution paradigm by embedding the curves C (s; t) as the@ level set of a surface f (x; y; t) = 0g evolving by @t = ( 0 ? 1 )jr j. Table 1 shows a classi cation of shocks from which has been implemented to sub-pixel accuracy [9]. While the derived shock structure when 1 = 0 is related to skeletons, several properties of shocks relating to the notions of type, velocity, grouping, salience, and hierarchy are signi cant. (i) certain deformations, e.g., bending, affect shock types selectively, e.g., third-order shocks of a rectangle; (ii) the skeletal representation lacks sucient explicit dimensions for qualitative approximation, e.g., the addition of shock type substantially narrows down the range of shapes it can generate; (iii) topological and di erential properties of shocks, e.g., velocity, directly re ect boundary properties; (iv) the notion of time of formation induces a hierarchy on shocks; and (v) a notion of shock grouping and salience based on the di usion process ( 1 6= 0) leading to a stability of representation with small changes. It was
1π/4
θ
1 2 3 4
First−Order Shock
2
Fourth−Order Shock
P P
Figure 6: Each of the four types of shocks each is correlated with a perceptual/semantic category, i.e., protrusion, part, bend, and seed. Shock Type
Orientation
First−Order
non− vanishing
1 2 3 4
1 −π/4
−π/4
3
Third−Order Shocks
π/4 s
2 Second−Order Shock
(a)
4 θ s
(b)
3 4
Figure 7: The propagation of waves leads to a hyperbolic PDE for propagation orientation and formation of shocks. (a) formation of a shock (b) formation of a rarefaction wave. The left side of each shows the orientation along the contour; the right side shows the corresponding contour.
Curvature
∆φ
high level set curvature
∆φ
Second−Order
isolated vanishing
Third−Order
non−isolated vanishing
Fourth−Order
isolated vanishing
∆φ
∆φ
κ1 κ2 < 0 κ1 κ2 = 0 κ1 κ2 > 0
Table 1: This table depicts the classification of shock types based on the the gradient level set curvature and the principal curvatures of the surface.
proposed in [13] that the contour-based distance transform (CEDT) is an alternative, ecient paradigm for implementing wave propagation.
3 Shock labeling
A generalization of the shock-based representation of a binary shape to contours extracted from real images requires a notion of shock labels. The key insight underlying shock labeling is an understanding of the evolution of local orientation information during curve deformation and wave propagation. Two signi cant events were observed in [5]. First, the constant deformation of shape is equivalent to the propagation of orientation by a conservative hyperbolic PDE where \orientation-elements" ow as a function of their orientation. The formation of a shock is illustrated in Figure 7a when the = 4 portion ows toward and collides with the = ? 4 portion and in the process annihilating all other orientation elements. Second, a rarefaction wave forms when two orientation elements
ow away from each other as in Figure 7b and create a gap of orientation. The conservation of orientation implies that the gap generated thus should be lled in linearly leading to a circular rarefaction wave emanating from the singular point [5]. The generation of rarefaction waves as applied to propagation of orientation from an edge map implies wavefronts whose orientation is \bogus" in that it does not re ect a propagation of some initial orientation, Figure 8. Thus, while shocks arising from such points are necessary for a complete representation of shape, they do not re ect symmetry between two contour segments. Therefore, an additional labeling of rarefaction fronts allows for proper propagation of contour segments, Figures 8c and 8a. Formally, De nition 1 Regular wavefronts are those points of
the wavefronts which arise from contour points with
regular tangents. Rarefaction or degenerate wavefronts arise from points having no or multiply de ned tangents.
The interaction of these two types of waves lead to three types of shocks. De nition 2 A shock point arising from two regular
waves is regular. A shock point arising from one regular and one degenerate wavefront is semi-degenerate. A shock point formed from the interaction of two degenerate wavefronts is degenerate.
Note that a labeling of shocks into these three classes is independent of a classi cation of shocks into the four types described earlier. We employ two computational paradigms to detect, classify, group and label shocks, namely curve evolution and CEDT. The algorithm for labeling shocks is essentially the same in both paradigms: label each wavefront by tracing back to the source of propagation and measure the length of the wavefront arising from the source. For a rarefaction wave, a small length initial contour gives rise to a large length on the front; the ratio when normalized by the absolute time of evolution is a fanout ratio, Figure 12. Note that no explicit a priori detection of corners or endpoints is required. We have implemented this algorithm and applied it to the labeling of shocks of partial contours, Figures 9 and 10. Observe how the regular shocks properly depict the symmetry of the object. The details of the algorithms and a comparison of the two computational paradigms can be found in [11].
4 Shock Labels Depict Part Anatomy
Partitioning prior to gure-ground segregation can aid in the segmentation process. It is now argued that shock labels provide evidence for parts in the result of low-level processes. A theory for partitioning visual form was proposed in [8] leading to two types of parts: one at necks, or certain locally shortest lines through the shape, and limbs, or curves through two negative curvature minima that smoothly interpolate
3
2
2 1 0
(a)
Regular
4 3
4
1 0
Initial Orientation Element
1 2 3
4
(b)
(c)
Rarefaction
Figure 8: (a) Ideal propagation of orientation element for
shocks representing contour symmetry. (b) Wave-based propagation of orientation includes additional fronts, rarefaction waves are depicted as dashed lines in (c), which give rise to additional (seemingly spurious) shocks when fronts collide.
(a)
(c)
Figure 10: Labeling of shocks into three classes, regular
(green), semi-degenerate shocks (yellow), and degenerate shocks (red) for an incomplete rectangle, (a) and for another incomplete shape (b).
Figure 11: Two types of parts: limbs (left) and necks (middle). The limbs and necks of the sh shape (right) are shown [8].
(a)
(b)
(c)
Figure 9: (a) The interaction of regular waves from two
lines leads to regular shocks. (b) semi-degenerate shocks arise from a rarefaction wave meeting a regular wave. (c) degenerate shocks arise from two degenerate (rarefaction) waves.
the tangent at one point to the tangent at another, Figure 11. The computation of each candidate partline involves computing a measure of salience which is then used to resolve con icts when they arise [8]. While this framework is developed for partitioning segmented shape, the computations are local with respect to the part-line size and can thus be computed prior to gure-ground segmentation. We now suggest that a reformulation of this approach in the shock language using shock labels allows parts to be computed directly from images. While neck-based parts correspond to second-order shocks, the connection between limb-based parts and shocks is more subtle and requires a close examination of the anatomy of limb-based parts with respect to shock labels. Figure 13 illustrates that the two sharp negative curvature minima generate rarefaction waves leading to three sets of \irregular" shocks: i) the stem: the semi-degenerate shocks arising from the interaction of rarefaction waves from A and B with
the regular waves from the body of the shape, ii) the attachment: the degenerate shocks arising from the interaction of rarefaction waves from A and B, and iii) the join: the semi-degenerate shocks arising from the interaction of rarefaction waves from B with the regular waves from the part. Thus, a labeling of shocks in general divides the shock-based representation of a shape into ve distinct groups: the above three groups and iv) the base: the regular shocks pertaining to the main shape itself, and v) the limb: the regular shocks pertaining to the limb part. Figure 13d illustrates how the body and the stem shock groups identi ed by labels remain invariant with movement and change of the part shape. Figure 14a shows actual computation of shocks and shock labels. These labels provide a strategy for partitioning shape and extracting each component's shock-based representation: since the attachment, join, and stem do not re ect each component's symmetry but rather depict the anatomy of the connection itself, propagate waves from a hypothesized limb part-line (a) which then interact with the degenerate and semi-degenerate shocks and produce each component's shock-based representation correctly, Figure 14b. Observe that degenerate shocks are not always present, Figure 14c, but can be detected via multiple generation shocks; see Section 5. The shock labeling of part anatomy also has implications for gaps. It was proposed in [10] that gaps may be closed by a partitioning of the background which produces a grouping of the foreground, Figure 5, the case of an occluder which blends with the back-
(a) (b) Figure 12: The generalization of the notion of a rarefac-
tion wave (a) based on a measure of fanout which indicates the reliability of orientation estimates at the wavefront (b).
(a)
(b)
(c)
(d)
Limb
Attachment
Base
Join A
(a)
B
Stem
(b) Figure 13: Sharp negative curvature extrema A and B (a)
generate rarefaction waves leading to ve distinct groups of shocks identi ed by a labeling of these shocks into regular (green: base and limb), semi-degenerate (yellow: stem and joint) and degenerate (red: attachment). Observe that variations in the part shape leave the base and stem invariant. Observe how a gap is viewed as a \null part," a construction originally proposed in [10].
ground as in Figure 1b. In the case of a null part, or a part with no limb, Figure 13d, partitioning the degenerate shocks leads to successful closure of the shape. Figure 12 illustrates the need for generalizing the notion of a rarefaction wave using a fanout ratio to include scale. Note that the shape of Figure 13c when the sharp points A and B are smoothed out slightly would still be perceived as a composite shape, yet no rarefaction waves form in the strict sense. A similar situation hold in the case of a noisy contour with a corner, Figure 12b. However, observe that a similar phenomenon occurs since the orientation of the evolving front emanating from these \coarse-scale" corners is unreliable, and thus, must be marked as such. The \fan-out" ratio described earlier accomplishes this task. This quantity for a wave emanating from a corner or end point is in nite, small for a line, and high for a sharply curved contour. This generalization of the binary rarefaction measure leads to stability against perturbations, Figure 12. Observe also that in the limit as the part and shape smoothly blend together, the perception of a part gives way to the perception of a protrusion. The generalized rarefaction wave thus captures the part-protrusion continuum [3].
Figure 14: (a) Degenerate and semi-degenerate shocks
hypothesize a limb-based part. (b) Secondary waves from this part-line correct the irregular shocks to form the symmetry set for each component. (a). While degenerate shocks are not always evident, second generation waves launched at irregular shocks will bring them out (d). See also Figure 17. Arrays indicate transitions.
5 Inter-penetrating Waves and Multiple Generation Shocks
We have shown that shock labels identify true contour symmetries via regular shocks, as well as parts and gaps via a combination of semi-degenerate and degenerate shocks. In a companion paper [13] it is proposed that a new generation of waves (interpenerating waves) be launched from select shock groups identi ed by shock labels and the resulting shocks (multiple generation shocks) recorded. The main idea can be motivated by observing how the spurious edge element in Figure 4 alter the underlying symmetries. Waves from this element quench waves from the gure boundary and thus interfere with the proper formation of gure symmetry. The appropriate symmetries exist in the full symmetry set, but are embedded in a large set of unintuitive ones. Instead, these can be recovered by launching waves at select groups of shocks [13]. Secondary waves launched at salient limbs correct the distorted symmetries, Figures 14, 16b, and 17, leading to a partitioning of the shape. Salience of a limb is partially related to the optimal shape of the completion contour, or limb part-line [8]. Figure 15 examines the relationship between salience and shock labels: observe how the length of the degenerate shock group
(a)
(b) Figure 18: Degenerate shock sets signal contour grouping,
some indicate the completion of triangle, while others indicate the completion of the pacman gures. First, observe that separation in depth which is assumed in [1] is not required here but rather follows from shock labeling and subsequent contour grouping. Second, this example illustrates that rarefaction waves generated by corner points are also signi cant. Arrays indicate transitions.
(c)
(d) Figure 15: The length of degenerate shocks signals the saliency of the part-line/completion contour.
varies with the \extent' of the part, and the \ misalignment" of two tangents, Figure 15. Observe how salience allows for distant, second generation grouping over rst generation ones, Figure 17(e-g). The latter example highlights the combined use of shock labels, interpenetrating waves, and multiple generation shocks. Figure 16 illustrates how partitioning and grouping are identical operations in the shock domain. Figure 16a is perceived as two overlapping triangles. As parts are removed one by one, Figures 16b and 16c, the rst triangle is recovered, resulting in two disjoint pieces of the second triangle. Subsequent removal of parts (background) leads to the grouping of two disjoint pieces. Waves from the pair of limb partlines, previously referred to as \hidden limbs" [8], now form the shocks corresponding to the second triangle. Observe that the notion of salience is crucial when the alignment and smoothness of the connecting limb part-line is challenged, Figure 16f.
6 On the connection to the gap skeleton
August et al. presented an interesting connection between certain portions of a skeleton, namely the gap skeleton and the skeleton of a virtual occluder which signals the grouping of contour fragment endpoints. The main idea is that occlusion leads to T-junction discontinuities. They propose that it is not the shape
of the optimal continuation curve at T-junctions that is signi cant, but rather the consistency of the grouping as the outcome of some virtual occluder. Thus, based on the assumption that extracted contours have been partially depth-ordered in separate maps based on T-junctions, they group end points with no contour present in the circular neighborhood whose diameter is the line segment joining the two endpoints. Expressed in the language of shocks, this implies a second-order shock and two rst-order shock groups (gap skeleton) which arise from the end points. Our approach bears some resemblance to this proposal where the main similarity is the expression of contour grouping in the language of shocks, speci cally that certain degenerate rst-order shock groups, arising from end points (not corners) and containing second-order shocks constitute the gap skeleton. However, there are a number of signi cant distinctions. First, the motivation and domain of application is rather di erent: in contrast to grouping contour fragments partially ordered in depth via T-junctions, our goal is to extract partial shocks from partial contours of the full set of extracted contours by separating extracted symmetries into classes: regular shocks represent true contour symmetry, while degenerate shocks represent connectivity. Thus, this approach achieves such a grouping without depth segmentation via limb hypothesis and inter-penetrating waves which in turn lead to depth segregation, Figure 18. August et al. require a separation of the triangular contours from the circular ones to achieve grouping, while we propose that a separation follows from the grouping. This gure also illustrates a second important dis-
(a)
(b)
(c)
(d)
(e)
(f)
Figure 16: (a-e) partitioning and grouping as identical operations lead to the recovery of two triangles. (f) the role of salience of part-lines in grouping when alignment is challenged.Arrays indicate transitions.
(a)
(b)
(c)
(d)
(e)
(f)
(g
Figure 17: (a) A shape with two parts and its shocks. (b) The recovery of individual part's shocks by shock labeling followed by a second generation shocks of waves initiated at limbs. (c) a shape with a gap (null part) and its shocks. (d) Second generation waves initiated at salient limbs remove the degenerate shocks and correct for the distortion at semi-degenerate shocks. (e) a spurious edge element's interference with contour grouping can be removed by considering multiple generation of shocks as shown in (f) where second generation shocks arising from the shocks loops (completed by the image boundary) generate a new grouping hypothesis thus completing the rectangle symmetries (g). tinction, namely, the concept of classi cation of a [4] B. B. Kimia, A. R. Tannenbaum, and S. W. Zucker. Toward a computational theory of shape: An wave based on whether the orientation of its front overview. In ECCV, Antibes, France, 1990. can be reliably related to the original front (regular) or not (rarefaction wave). Thus, corner points which [5] B. B. Kimia, A. R. Tannenbaum, and S. W. Zucker. Shapes, shocks, and deformations, I: The components generate rare-faction waves lead to degenerate shock of shape and the reaction-di usion space. Internagroups which are not gap skeletons. Third, we hold tional Journal of Computer Vision, 15:189{224, 1995. that the shape of the optimal contour connecting the [6] S. M. Pizer and C. A. Burbeck. Object representaend points is signi cant, indicating the salience of the tion by cores: Identifying and representing primitive grouping, e.g., in detecting alignments, Figure 15 and spatial regions. Vision Research, 1994. Figure 16. In fact, Figure 15d illustrates that the de[7] G. L. Scott, S. Turner, and A. Zisserman. Using a termination of whether a gap skeleton exists can be mixed wave/di usion process to elicit the symmetry viewed as a binary representation of the salience of the set. IVC, 7(1):63{70, February 1989. optimal contour completion. This leads to instabili[8] K. Siddiqi and B. B. Kimia. Parts of visual form: ties in that minute changes in contour orientation or Computational aspects. PAMI, 17(3):239{251, March relative position can remove the gap skeleton. In addi1995. tion, no dilation of the initial curve is needed. Finally, [9] K. Siddiqi and B. B. Kimia. A shock grammar for contour grouping via degenerate shock sets requires recognition. In CVPR, June 1996. a notion of inter-penetrating waves and saliency, Figure 17e-g. [10] K. Siddiqi, K. J. Tresness, and B. B. Kimia. On the anatomy of visual form. In IWVF, pages 507{521, Capri, Italy, May 1994. References [11] P. Stoll, H. Tek, and B. B. Kimia. Shocks from im[1] J. August, K. Siddiqi, and S. W. Zucker. Fragment ages: Propagation of orientation elements. Technical grouping via the principle of perceptual occlusion. In Report 153, LEMS, Brown University, October 1996. ICPR, 1996. [12] S. Tari, J. Shah, and H. Pie. A computationally ecient shape analysis via level sets. In IEEE Workshop [2] M. F. Kelly and M. D. Levine. Annular symmetry on Mathematical Methods in Biomedical Image Analoperators: A method for locating and describing obysis, 1996. jects. In ICCV, pages 1016{1021, 1995. [13] H. Tek, F. Leymarie, and B. B. Kimia. Multiple gen[3] B. B. Kimia, K. Siddiqi, A. R. Tannenbaum, and eration shock detection and labeling using CEDT. In S. W. Zucker. The shape triangle: Parts, protrusions, IWVF, Capri, Italy, May 1997. and bends. Vision Research, Submitted.
