Local Shape Approximation from Shading
Local Shape Approximation from Shading Daphna Weinshall Department of Computer Science The Hebrew University of Jerusalem 91904 Jerusalem, ISRAEL Internet: [email protected]
Abstract
Shading can be used as an independent cue for exact shape recovery, or it can be used as a supplementary cue for shape interpolation between features, whose depth is known from other cues. Exact shape cannot be inferred from a local analysis of shading. However, for shape interpolation, a crude local approximation may be sucient. This paper explores the limits of such local approximations that are easy to compute. In particular, the shape of shading is used to approximate the surface in areas of monotonic change of intensity. This analysis is accompanied by a method to compute the direction of a single point light source from the shading on occluding contours. A qualitative classi cation of shape near shading singularities is also discussed.
1 Introduction Shading can be used as an independent cue for exact shape recovery, or it can be used as a supplementary cue for shape interpolation between features, whose depth is known from other cues (such as occluding contours or stereo). Psychophysical evidence support the idea that the second role of shading, as a secondary cue for surface interpolation, may be more important to humans than its role as a primary cue, possibly because exact shape from shading is very dicult. One clue follows from the observation that humans perceive shaded images of Lambertian surfaces with mutual illumination (or interre ections) as far more realistic than in the absence of mutual illumination. Mutual illumination can severely degrade the accuracy of shape from shading algorithms (see [21] for an exception). On the other hand, mutual illumination can be used to detect other shape cues, such as occluding contours in shaded regions [11]. Human psychophysics also supports the idea that shading may not necessarily be used to compute an exact depth map, but rather an approximate surface that provides relative depth information more accurately than exact depth. One study shows that humans perceive relative depth from shaded images more reliably than absolute depth [27]. Other studies show that the ability of humans to infer shape from shading is weak1 [6]. 1
This work was done at MIT, at the Center for Biological Information Processing. B. Cumming, personal communication
1
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
2
The problem of inferring shape from shading proved to be one of the most dicult in lowlevel vision. Even with the simplest Lambertian shading model, which is an idealization criticized as irrelevant since it does not necessarily approximate reality [11], and assuming constant albedo and a single point light source, signi cant ambiguity remains. For example, there exist concave, convex and saddle-like surfaces that appear the same from certain viewpoints2 . Therefore most exact shape from shading algorithms involve the propagation of information from special locations on the surface, such as occluding contours or intensity singularities, where the surface is assumed known [32, 14, 12, 22, 26]. Techniques such as the method of characteristic strips were used to solve systems of di erential equations with known initial conditions. Typically these techniques are computationally expensive, requiring thousands of iterations, and rarely is it possible to prove that they converge. Because of these unattractive properties of global shape from shading algorithms, local shading analysis under some special circumstance was studied: spherical surfaces were discussed in [24, 18]; slanted surfaces, where the angle between the light source and the normal is suciently large, were discussed in [23]. Others have studied the behaviors of isophotes (lines of equal brightness) and their relationship to geometrical invariants of surfaces [17, 33, 4]. The present paper follows these directions in exploring approximations to shape using easy to compute local shading analysis (see also [10]). The local shading approximation to shape may be useful for various limited purposes. Some examples are the following: If the task requires a a simple transformation, e.g. the prediction of the image of the surface illuminated from an in nitesimally di erent direction, the approximation may be sucient. In that case, the transformation is very simple: the rotation of the shading surface by the amount of rotation of the light source. If an exact iterative shape from shading algorithm is to be performed, the shading approximation can give a better initial guess of the surface. The shading approximation gives relative depth of points on the surface in an unknown coordinate system, which may be sucient for the computation of bumps and other surface features. The rest of this paper is organized as follows: A qualitative classi cation of surfaces near some intensity singularities is discussed in section 2. An approximation of shape by the shading function itself is explored in section 3. Section 4 deals with the computation of the direction of the light source. Finally, the information that can be obtained on the surface directly from the shading, making use of the shading approximation discussed in section 3 and other cues, is discussed in section 5. 2
e.g., under orthographic projection and when the light source is behind the viewer, the surfaces z = x2 + y2 ,
z = x2 ? y2 , and z = ?x2 ? y2 all appear the same.
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
3
2 Geometrical properties of surfaces near global shading maxima: Many shape from shading algorithms need, or can take advantage of, information on the shape near shading singularities (e.g. [13, 22]). A qualitative analysis of shading extrema was suggested by Koenderink & van Doorn [17], showing that these singularities often cling to parabolic points. In this section, a qualitative classi cation of the shape of surfaces near global shading maxima is obtained from the change in the appearance of isophotes in the image with a change of viewpoint. Consider a Lambertian surface illuminated by a single light source (or a hemispherical sky), so that the irradiance function is proportional to R~ N~ , at a point where the normal to the surface is N~ and the light source direction (or the vertical direction) is R~ . In most scenes, assuming it is not on the boundary of the surface, global shading maxima occur when N~ is parallel to R~ . Choose such a point P . Let us use a coordinate system where the Z axis parallels R~ , P de nes the origin, and the X and Y axes are the directions of principal curvatures on the surface at P . With this selection, for j1j j2j the principal curvatures at P , the surface can be written as follows: 2 2 z(x; y) = 1 x +2 2 y + ; where
lim (x;y)!(0;0) x2 + y 2
= 0:
(1)
If P is parabolic, namely 1 = 0, the isophotes near P are straight lines. If j1 j = j2j, the isophotes are circles on the X ? Y plane ( gure 1(a)), planar curves if P is elliptic (1 = 2 ) and non-planar 3D curves if P is hyperbolic (1 = ?2 ). Active exploration can be used to determine when the isophotes are planar curves (the parabolic and elliptic cases) and when they are not (the hyperbolic case) in the following way: When the isophotes are parallel planar curves, they are projected obliquely to the image plane, and all of them are uniformly foreshortened. This ane transformation is responsible for contraction along one axis of the planar curves, as is illustrated in appendix A. Foreshortening preserves many curve features: straight lines are projected to straight lines, elliptical close contours to elliptical close contours, and in ection points to in ection points. Moreover, this projection is stable in the sense that as the camera (or the light source) moves, the curves uniformly get more or less contracted along a single axis. The case is di erent for isophotes that are not planar curves, and therefore their projection to the image plane is not foreshortening. The further away the isophotes are from parallel planar curves, the less like simple single-axis contraction their projection to the image plane looks. An example of the projection of a particular isophote on an elliptic surface with 1 = 2 , and an example of the projection of the corresponding isophote on a hyperbola with 1 = ?2 , are given in gure 1. As j1j gets further away from j2 j, the distinction between the hyperbolic and elliptic cases becomes less sharp. The projected isophotes still di er in the same way, however, as is shown in gure 2: in b, a hyperbolic surface where 1 = ? 32 2 , the triangular shape of the isophotes ( gure 2(a)) is maintained; in c, an elliptic surface where 1 = 32 2 , the isophotes look more like foreshortened ellipses. This analysis suggests a heuristical method for the classi cation of surfaces near global shading maxima (that arise from the di use, rather than the specular, component of the illumination):
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
a)
b)
4
c)
Figure 1: The projection onto the image of an isophote on the surface z = ?(x2 + y 2 ) (elliptic, above) and z = x2 ? y 2 (hyperbolic, below), illuminated by a single light source; a) viewing angle is parallel to R~ , b) the angle between viewing angle and R~ is 30 , c) the angle between viewing angle and R~ is 45 . o
o
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
a)
b)
5
c)
Figure 2: The isophote map, projected onto the image, near a shading maximum. The viewing coordinate system is obtained by a rotation of 45 around 2the2 Z axis, followed by a rotation of 10 around the X . a) A generic hyperbolic surface z = ?2 ; b) a generic hyperbolic surface 2 2 2 2 2 2 z = ?23 ; c) a generic elliptic surface z = ? +23 . o
x
o
x
y
x
y
y
measure the projection of the isophotes in the image plane and their change with change of
camera location; if the isophote projections are straight lines locally, the surface is parabolic ( gure 3(a)); if the isophote projections can be approximated by foreshortened circles, and their change can be approximated by contraction/expansion along a single axis, the surface is elliptic ( gure 3(b)); if the isophote projections can better be approximated by concentric triangles, the surface is hyperbolic ( gure 3(c)).
This method requires oblique viewing and can be used more e ectively with active exploration. It is assumed that 1 = 0 or 1 = 2 . The classi cation becomes less reliable as 1 gets further away from 2 .
3 A local approximation to shape from shading In this section, the limits of the shape of the shading as an approximation to shape are studied. It is shown that if a linear transformation between the shapes of the shading and the surface is allowed, there exist families of surfaces for which equality holds. One such family is the locally-spherical surfaces, the only surfaces for which local shape from shading can be computed precisely [24]. In the case of such surfaces, and others, the shading is the shape, very little additional computation is required. For additional families of surfaces, the individual isophotes, or lines of equal brightness, are identical to the contour lines on the surface. The shading approximation to shape, proposed in this section, is the approximation of surface patches by members of these families.
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
a)
b)
6
c)
Figure 3: Classi cation of surfaces near global shading maxima: a) parabolic, b) elliptic, c) hyperbolic. A depth map of a surface can be represented by a contour map, a map of the lines of equal depth on the surface. We are interested in partial representations, where only the contour lines of the map are given, and a scalar function F is de ned over the contour lines. When F is a monotonic function of the depth of the contour lines, this representation gives the relative depth between any two points on the surface. When F is a linear function of the depth, this representation gives the depth of the surface at each point up to a single scaling factor. The last two cases will now be discussed. The question addressed in this section is when the isophotes (lines of equal brightness on the surface) are also the contour lines (lines of equal depth on the surface), and what is the function F that relates the intensity of the isophotes to the depth on the contours. Local stability analysis is used to detect cases where this shading approximation should not be used.
3.1 The 3D shading function:
Assume a Lambertian surface z (x; y ) with xed albedo and a single light source or a uniform hemispherical sky (ambient illumination can also exist.) The re ectance function of a Lambertian surface depends only on the angle between the light source direction and the surface normal, in gure 4. The brightness of this surface at a point with normal N~ is:
I = R~ N~ + ; (2) for some constants ; and a xed direction R~ , which is either parallel to the light source direction
or vertical. Since the surface is assumed to be Lambertian, the position of the viewer does not a ect the shading at each point on the surface. Therefore it is possible to consider the shading as a scalar function de ned on a surface in 3D, before computing the projection onto the image plane. Henceforth in this section, isophote map will refer to this function de ned on a surface in 3D. The discussion of the projection to 2D is postponed to the next section.
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
7
Figure 4: The re ectance function of a surface at a point is composed of three basic components: di usive, specular, and ambient. For the analysis of the isophote map, which is independent of the viewer position, a coordinate system is used where the depth Z is parallel to R~ . This coordinate system is denoted
Abstract
Shading can be used as an independent cue for exact shape recovery, or it can be used as a supplementary cue for shape interpolation between features, whose depth is known from other cues. Exact shape cannot be inferred from a local analysis of shading. However, for shape interpolation, a crude local approximation may be sucient. This paper explores the limits of such local approximations that are easy to compute. In particular, the shape of shading is used to approximate the surface in areas of monotonic change of intensity. This analysis is accompanied by a method to compute the direction of a single point light source from the shading on occluding contours. A qualitative classi cation of shape near shading singularities is also discussed.
1 Introduction Shading can be used as an independent cue for exact shape recovery, or it can be used as a supplementary cue for shape interpolation between features, whose depth is known from other cues (such as occluding contours or stereo). Psychophysical evidence support the idea that the second role of shading, as a secondary cue for surface interpolation, may be more important to humans than its role as a primary cue, possibly because exact shape from shading is very dicult. One clue follows from the observation that humans perceive shaded images of Lambertian surfaces with mutual illumination (or interre ections) as far more realistic than in the absence of mutual illumination. Mutual illumination can severely degrade the accuracy of shape from shading algorithms (see [21] for an exception). On the other hand, mutual illumination can be used to detect other shape cues, such as occluding contours in shaded regions [11]. Human psychophysics also supports the idea that shading may not necessarily be used to compute an exact depth map, but rather an approximate surface that provides relative depth information more accurately than exact depth. One study shows that humans perceive relative depth from shaded images more reliably than absolute depth [27]. Other studies show that the ability of humans to infer shape from shading is weak1 [6]. 1
This work was done at MIT, at the Center for Biological Information Processing. B. Cumming, personal communication
1
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
2
The problem of inferring shape from shading proved to be one of the most dicult in lowlevel vision. Even with the simplest Lambertian shading model, which is an idealization criticized as irrelevant since it does not necessarily approximate reality [11], and assuming constant albedo and a single point light source, signi cant ambiguity remains. For example, there exist concave, convex and saddle-like surfaces that appear the same from certain viewpoints2 . Therefore most exact shape from shading algorithms involve the propagation of information from special locations on the surface, such as occluding contours or intensity singularities, where the surface is assumed known [32, 14, 12, 22, 26]. Techniques such as the method of characteristic strips were used to solve systems of di erential equations with known initial conditions. Typically these techniques are computationally expensive, requiring thousands of iterations, and rarely is it possible to prove that they converge. Because of these unattractive properties of global shape from shading algorithms, local shading analysis under some special circumstance was studied: spherical surfaces were discussed in [24, 18]; slanted surfaces, where the angle between the light source and the normal is suciently large, were discussed in [23]. Others have studied the behaviors of isophotes (lines of equal brightness) and their relationship to geometrical invariants of surfaces [17, 33, 4]. The present paper follows these directions in exploring approximations to shape using easy to compute local shading analysis (see also [10]). The local shading approximation to shape may be useful for various limited purposes. Some examples are the following: If the task requires a a simple transformation, e.g. the prediction of the image of the surface illuminated from an in nitesimally di erent direction, the approximation may be sucient. In that case, the transformation is very simple: the rotation of the shading surface by the amount of rotation of the light source. If an exact iterative shape from shading algorithm is to be performed, the shading approximation can give a better initial guess of the surface. The shading approximation gives relative depth of points on the surface in an unknown coordinate system, which may be sucient for the computation of bumps and other surface features. The rest of this paper is organized as follows: A qualitative classi cation of surfaces near some intensity singularities is discussed in section 2. An approximation of shape by the shading function itself is explored in section 3. Section 4 deals with the computation of the direction of the light source. Finally, the information that can be obtained on the surface directly from the shading, making use of the shading approximation discussed in section 3 and other cues, is discussed in section 5. 2
e.g., under orthographic projection and when the light source is behind the viewer, the surfaces z = x2 + y2 ,
z = x2 ? y2 , and z = ?x2 ? y2 all appear the same.
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
3
2 Geometrical properties of surfaces near global shading maxima: Many shape from shading algorithms need, or can take advantage of, information on the shape near shading singularities (e.g. [13, 22]). A qualitative analysis of shading extrema was suggested by Koenderink & van Doorn [17], showing that these singularities often cling to parabolic points. In this section, a qualitative classi cation of the shape of surfaces near global shading maxima is obtained from the change in the appearance of isophotes in the image with a change of viewpoint. Consider a Lambertian surface illuminated by a single light source (or a hemispherical sky), so that the irradiance function is proportional to R~ N~ , at a point where the normal to the surface is N~ and the light source direction (or the vertical direction) is R~ . In most scenes, assuming it is not on the boundary of the surface, global shading maxima occur when N~ is parallel to R~ . Choose such a point P . Let us use a coordinate system where the Z axis parallels R~ , P de nes the origin, and the X and Y axes are the directions of principal curvatures on the surface at P . With this selection, for j1j j2j the principal curvatures at P , the surface can be written as follows: 2 2 z(x; y) = 1 x +2 2 y + ; where
lim (x;y)!(0;0) x2 + y 2
= 0:
(1)
If P is parabolic, namely 1 = 0, the isophotes near P are straight lines. If j1 j = j2j, the isophotes are circles on the X ? Y plane ( gure 1(a)), planar curves if P is elliptic (1 = 2 ) and non-planar 3D curves if P is hyperbolic (1 = ?2 ). Active exploration can be used to determine when the isophotes are planar curves (the parabolic and elliptic cases) and when they are not (the hyperbolic case) in the following way: When the isophotes are parallel planar curves, they are projected obliquely to the image plane, and all of them are uniformly foreshortened. This ane transformation is responsible for contraction along one axis of the planar curves, as is illustrated in appendix A. Foreshortening preserves many curve features: straight lines are projected to straight lines, elliptical close contours to elliptical close contours, and in ection points to in ection points. Moreover, this projection is stable in the sense that as the camera (or the light source) moves, the curves uniformly get more or less contracted along a single axis. The case is di erent for isophotes that are not planar curves, and therefore their projection to the image plane is not foreshortening. The further away the isophotes are from parallel planar curves, the less like simple single-axis contraction their projection to the image plane looks. An example of the projection of a particular isophote on an elliptic surface with 1 = 2 , and an example of the projection of the corresponding isophote on a hyperbola with 1 = ?2 , are given in gure 1. As j1j gets further away from j2 j, the distinction between the hyperbolic and elliptic cases becomes less sharp. The projected isophotes still di er in the same way, however, as is shown in gure 2: in b, a hyperbolic surface where 1 = ? 32 2 , the triangular shape of the isophotes ( gure 2(a)) is maintained; in c, an elliptic surface where 1 = 32 2 , the isophotes look more like foreshortened ellipses. This analysis suggests a heuristical method for the classi cation of surfaces near global shading maxima (that arise from the di use, rather than the specular, component of the illumination):
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
a)
b)
4
c)
Figure 1: The projection onto the image of an isophote on the surface z = ?(x2 + y 2 ) (elliptic, above) and z = x2 ? y 2 (hyperbolic, below), illuminated by a single light source; a) viewing angle is parallel to R~ , b) the angle between viewing angle and R~ is 30 , c) the angle between viewing angle and R~ is 45 . o
o
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
a)
b)
5
c)
Figure 2: The isophote map, projected onto the image, near a shading maximum. The viewing coordinate system is obtained by a rotation of 45 around 2the2 Z axis, followed by a rotation of 10 around the X . a) A generic hyperbolic surface z = ?2 ; b) a generic hyperbolic surface 2 2 2 2 2 2 z = ?23 ; c) a generic elliptic surface z = ? +23 . o
x
o
x
y
x
y
y
measure the projection of the isophotes in the image plane and their change with change of
camera location; if the isophote projections are straight lines locally, the surface is parabolic ( gure 3(a)); if the isophote projections can be approximated by foreshortened circles, and their change can be approximated by contraction/expansion along a single axis, the surface is elliptic ( gure 3(b)); if the isophote projections can better be approximated by concentric triangles, the surface is hyperbolic ( gure 3(c)).
This method requires oblique viewing and can be used more e ectively with active exploration. It is assumed that 1 = 0 or 1 = 2 . The classi cation becomes less reliable as 1 gets further away from 2 .
3 A local approximation to shape from shading In this section, the limits of the shape of the shading as an approximation to shape are studied. It is shown that if a linear transformation between the shapes of the shading and the surface is allowed, there exist families of surfaces for which equality holds. One such family is the locally-spherical surfaces, the only surfaces for which local shape from shading can be computed precisely [24]. In the case of such surfaces, and others, the shading is the shape, very little additional computation is required. For additional families of surfaces, the individual isophotes, or lines of equal brightness, are identical to the contour lines on the surface. The shading approximation to shape, proposed in this section, is the approximation of surface patches by members of these families.
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
a)
b)
6
c)
Figure 3: Classi cation of surfaces near global shading maxima: a) parabolic, b) elliptic, c) hyperbolic. A depth map of a surface can be represented by a contour map, a map of the lines of equal depth on the surface. We are interested in partial representations, where only the contour lines of the map are given, and a scalar function F is de ned over the contour lines. When F is a monotonic function of the depth of the contour lines, this representation gives the relative depth between any two points on the surface. When F is a linear function of the depth, this representation gives the depth of the surface at each point up to a single scaling factor. The last two cases will now be discussed. The question addressed in this section is when the isophotes (lines of equal brightness on the surface) are also the contour lines (lines of equal depth on the surface), and what is the function F that relates the intensity of the isophotes to the depth on the contours. Local stability analysis is used to detect cases where this shading approximation should not be used.
3.1 The 3D shading function:
Assume a Lambertian surface z (x; y ) with xed albedo and a single light source or a uniform hemispherical sky (ambient illumination can also exist.) The re ectance function of a Lambertian surface depends only on the angle between the light source direction and the surface normal, in gure 4. The brightness of this surface at a point with normal N~ is:
I = R~ N~ + ; (2) for some constants ; and a xed direction R~ , which is either parallel to the light source direction
or vertical. Since the surface is assumed to be Lambertian, the position of the viewer does not a ect the shading at each point on the surface. Therefore it is possible to consider the shading as a scalar function de ned on a surface in 3D, before computing the projection onto the image plane. Henceforth in this section, isophote map will refer to this function de ned on a surface in 3D. The discussion of the projection to 2D is postponed to the next section.
Journal of Mathematical Imaging and Vision 4(2):119{138, April 1994
7
Figure 4: The re ectance function of a surface at a point is composed of three basic components: di usive, specular, and ambient. For the analysis of the isophote map, which is independent of the viewer position, a coordinate system is used where the depth Z is parallel to R~ . This coordinate system is denoted
