Multi-view Surface Reconstruction using Polarization - Semantic Scholar
Multi-view Surface Reconstruction using Polarization Gary A. Atkinson and Edwin R. Hancock Department of Computer Science University of York Heslington, York YO10 5DD, UK {atkinson,erh}@cs.york.ac.uk Abstract A new technique for surface reconstruction is developed that uses polarization information from two views. One common problem arising from many multiple view techniques is that of finding correspondences between pixels on each image. In the new method, these correspondences are found by exploiting the spontaneous polarization of light caused by reflection to recover surface normals. These normals are then used to recover surface height. The similarity between reconstructed surface regions determines whether or not a pair of points correspond to each other. The technique is thus able to overcome the convex/concave ambiguity found in many single view techniques. Because the technique relies on smooth surface regions to detect correspondences, rather than feature detection, it is applicable to objects normally inaccessible to stereo vision. Also due to this fact, it is possible to remove noise without causing oversmoothing problems.
1. Introduction Multiple view techniques have proved to be highly effective in recovering models of surface shape. However, most of the existing methods are largely geometric in nature and rely on the availability of salient surface features or interest points to establish correspondence. Unfortunately, these methods are not particularly effective in the recovery of surface shape for smooth featureless surfaces. One alternative is to use photometric methods to characterise the surface, and this is the basis for techniques such as photometric stereo. However, such methods implicitly assume correspondence is known since they rely on the capture of a relatively large number of images under fixed object position and varying light source direction. One source of information that has not been so widely used on the photometric recovery of surface shape is that provided by polarization. The aim in this paper is therefore to explore whether information provided by the polariza-
tion state of diffusely reflected light can be used to establish correspondences on featureless surfaces for the purposes of stereoscopic depth recovery.
1.1
Previous Work
Many methods exist for combining information about an object or scene from two or more views [2]. The basic principle behind most computational stereo is that with two known views of an object it is possible to calculate the three-dimensional location of a point that is visible in both images. One major difficulty with stereo is deciding which points in one image correspond to which points in the second. A different approach, photometric stereo [14], involves the object under study being kept static with respect to the camera, and the direction of the light source being varied. In Helmholtz stereopsis [16], on the other hand, unique surface normal recovery is possible if the light source and camera are interchanged. Occluding contours, i.e. the outer boundaries of the image of the object under study [7], provide another source of information on surface structure. Although this information is limited (but reliable) from a single view, it can reveal much more when several views are used, placing heavy constraints on surface geometry [13]. Analysis of the polarization of light caused by surface reflection has been frequently used as a means to provide constraints on surface geometry. When initially unpolarized light is reflected from a surface, it becomes polarized [12]. This applies to both specular reflection (which we refer to as specular polarization) and diffuse reflection (diffuse polarization) and is due to the directionality of the molecular electron charge density interacting with the electromagnetic field of the incident light [6]. Most research aimed at extracting and interpreting information from polarization data, involves placing a linear polarizer in front of a camera and taking images of an object or a scene with the polarizer oriented at different angles [12, 9]. Other work has involved experimenting with liquid
crystal technology to enable rapid acquisition of polarization images [11]. Ikeuchi and colleagues have used specular polarization in shape recovery, where there is a need for specular reflections across the whole surface. This was done by placing the object under investigation inside a spherical diffuser, with several light sources outside and a hole for the camera [8]. Diffuse polarization [9, 10] can also be used in shape recovery. Although in this case the polarizing effects are weaker, especially far from object limbs, the global specular reflection is not required, meaning less specific lighting conditions are necessary.
1.2
E0i||
E0r||
E0i ni=1 nt
E0r
qr
qi
qt E0t
E0t |
Figure 1. Definitions. Directions of electric fields are indicated. N.B. θi = θr . The electrons in the medium vibrate parallel to the electric field.
Contribution
The aim of this paper is to describe a new method for shape recovery that uses polarization data from images of an object taken from different viewpoints. This firstly obtains a set of correspondences; secondly a field of surface normals; and finally a depth map. To do this, we make use of Fresnel Theory, which relates the reflected and incident wave amplitudes and provides a route to estimating the surface normals from polarization data. The available information is the degree of polarization and phase of the reflected light. The degree of polarization determines the zenith angle. The phase ambiguously defines azimuth angle. Surface normal determination is most reliable when the degree of polarization and, therefore the zenith angle, are large, that is close to the occluding boundary. The ambiguities normally associated with single view shape recovery techniques are eliminated here by a correspondence detection algorithm, which makes use of surface normals recovered from polarization data. Where these normals are similar in the two images, a correspondence is likely. Other constraints are imposed, based on simple geometric arguments, to add confidence to the location of correspondences. For example, if the object is being rotated and we know the direction of rotation, then we can predict the direction of motion of point, relative to the occluding contour. The method assumes that the angle between the two views is known, but a detailed photometric or geometric calibration is not necessary. This is in contrast to many stereo methods where the full epipolar geometry must be known. Indeed the introduction of an optimisation method should allow for an estimate the angle between views to be made. Results clearly demonstrate accurate depth recovery. Since the correspondence detection works for smooth surface regions, it is possible to apply moderately intense smoothing without facing over-smoothing problems. This makes the method robust to camera noise. The combined use of stereo vision and shape from diffuse polarization is an attractive one because the two techniques are complementary. Where stereo is weak – namely
near occluding contours and featureless surface areas – diffuse polarization is at its strongest. On the other hand, stereo allows the convex/concave ambiguity to be solved. The technique reported here should pave the way for future algorithms that use distinctive surface features, as in earlier stereo work, in addition to polarization. Ultimately, the development of a novel sensor will be possible.
2
Fresnel Reflectance
The principle on which this work is based is that when unpolarized light impinges upon a surface, a partially polarized reflection is formed. The degree of polarization (defined more precisely below) and the angle at which the reflected light is polarized reveals much about the surface. We now discuss the physics behind this spontaneous polarization, starting with a description of a classical picture of the origin the phenomenon. The electric field of an electromagnetic wave incident on a surface causes electrons of the reflecting medium near the surface to vibrate, forming dipoles. These vibrating electrons re-radiate, generating the reflected rays. The electrons vibrate parallel to the electric field. If the electric field of the incident wave is perpendicular to the plane of incidence, then so will be the electric field of the reflected wave. This is because the reflected wave is formed by the vibrating electrons – the reverse process to how they were initially set in motion. If, on the other hand, the incident light is polarized parallel to the plane of incidence, the electrons do not vibrate perpendicularly to the reflected ray, as Fig. 1 shows, resulting in a more attenuated wave. As the figure suggests, the degree of polarization depends on the angle of incidence.
2.1
Fresnel Coefficients
The Fresnel equations give the ratios of reflected wave amplitude to incident wave amplitude for incident light that is linearly polarized perpendicular to, or parallel to, the plane of specular incidence [1]. These ratios depend upon
era is rotated, the measured intensity varies sinusoidally according to the transmitted radiance sinusoid (TRS):
1
Fresnel Coefficient
T|| 0.8
T
⊥
0.4 R
⊥
0.2 0
Imax + Imin Imax − Imin + cos (2θpol − 2φ) 2 2 (4) where Imax and Imin are the maximum and minimum observed intensities as the polarizer is rotated, θpol is the angle which the polarizer makes with the arbitrary reference direction (here vertically upwards) and φ is the phase angle, or angle of polarization of reflected light, between 0◦ and 180◦ . From Fig. 2, it is clear that the maximum and minimum intensities detected for a particular surface orientation are I (θpol , φ) =
0.6
R|| 0
30 60 Specular angle of incidence
90
Figure 2. Reflection and transmission coefficients for a typical dielectric (n = 1.5).
Imax = the angle of incidence and the refractive index of the reflecting medium. Since the incident light can be always be resolved into components perpendicular to, and parallel to, the plane of incidence, the Fresnel equations are applicable to all incident polarization states. For the geometry of Fig. 1, and assuming that the material is non-ferrous, the amplitude reflection coefficients for perpendicularly polarized light, r⊥ , and parallel polarized light, rk , at a boundary between two media are given by E0r⊥ ni cos θi − nt cos θt = E0i⊥ ni cos θi + nt cos θt E0rk nt cos θi − ni cos θt = rk ≡ E0ik nt cos θi + ni cos θt
r⊥ ≡
(1) (2)
where ni and nt are the refractive indices of the first and second media and θi and θt are incident and transmitted angles. θt can be obtained from the well-known Snell’s Law: ni sin θi = nt sin θt
(3)
More useful are the intensity coefficients since it is the intensity of reflected light that is measured by cameras. 2 and Rk = rk2 [1]. All this assumes These are R⊥ = r⊥ that the refractive index is wavelength independent [4]. In fact, there is some wavelength dependence, but the equations above provide reasonable results for many situations. Fig. 2 shows the Fresnel intensity coefficients for a typical dielectric. It is the fact that R⊥ (θi ) 6= Rk (θi ) which underpins most of this research. This means that the reflected light is partially linearly polarized, that is, the light is a superposition of an unpolarized component, and a completely polarized component.
3 3.1
Polarization Vision The Polarization Image
We now consider how the above theory can be used in computer vision. As a polarizer placed in front of the cam-
R⊥ Is ; R⊥ + R k
Imin =
Rk Is R⊥ + R k
(5)
where Is is the magnitude of the specular component of reflection (assume for now, that there is no diffuse reflection). The degree of polarization or partial polarization, which is frequently used in computer vision, is defined by ρ=
Imax − Imin Imax + Imin
(6)
The intensity, phase and degree of polarization collectively form the polarization image. We obtain the phase and degree of polarization components by taking a series of images of an object using a digital camera with a linear polarizer mounted on its lens. The polarizer is rotated by a small amount between successive images and the Levenberg-Marquardt non-linear sine fitting algorithm is used to fit the measured intensities to (4) at each pixel.
3.2
Shape from Polarization
As Fig. 2 shows, the reflected light is attenuated to a greater extent if it is polarized parallel to the plane of incidence. Thus, greatest transmission through the polarizer occurs when the polarizer is oriented at an angle 90◦ from the azimuth angle of the surface, α. This provides a direct method for calculating azimuth angle: α = φ ± 90◦ . The ambiguity that this implies is a result of the phase image only having values in the range [0, 180◦ ). The zenith angle, the angle which the normal makes with the viewing direction, can be computed by considering the degree of polarization. Substituting (5) into (6) gives the degree of specular polarization in terms of the Fresnel coefficients. Then using the Fresnel Equations gives ρ in terms of n and the zenith angle, θ: p 2 sin2 θ cos θ n2 − sin2 θ ρs = 2 (7) n − sin2 θ − n2 sin2 θ + 2 sin4 θ Fig. 3a shows the dependence of ρs on θ. Note that there is an angle (known as the Brewster angle) where the light
n = 1.4
0.8
n = 1.6
0.6
qt ’
ni=1 nt
0.4
qi’ qi’
0.2 0
0
10
20
1
30 40 50 60 Specular angle of incidence b
70
80
90
Figure 4. Transmission of internally scattered light back into air.
0.8 0.6 n = 1.6
0.4 0.2 0
n = 1.4 0
10
20
30
40 50 60 Emittance angle
70
80
90
Figure 3. Degree of polarization for (a) specular and (b) diffuse reflection for two different refractive indices.
is totally polarized1 . Also, there is a 2-to-1 mapping of ρs to θ, resulting in another, different ambiguity that must be solved. Miyazaki et al [8] solve this problem using two views of the object. Diffuse polarization is a result of the following process [12]: A portion of the incident light penetrates the surface, is partially polarized in the process, as predicted by the Fresnel equations, and is refracted. Due to the random nature of internal scattering, the light becomes depolarized. Some of the light is then refracted back into the air and is, once again, refracted and partially polarized. The degree of diffuse polarization is highest, and so least noisy, near occluding boundaries where the zenith angle is large. Diffuse polarization has the advantage that less controlled lighting conditions are required than for specular polarization but has the disadvantages that it is not applicable to metals or transparent objects and is more susceptible to noise. When light approaches the surface-air interface from within the medium, as shown in Fig. 4, a similar process to that discussed earlier takes place but with the relative index of refraction being 1/n instead of n (assuming refractive index of air = 1). If the internal angle of incidence is above a critical angle (arcsin 1/n), then total internal reflection occurs. Otherwise, Snells Law (3) can be used to find the angle of emittance for any given angle of internal incidence. The Fresnel transmission coefficient can then be calculated 1 In practice ρ will be a little less than 1 due to a small, but finite, s diffuse component of reflection.
Transmission coefficient
Degree of specular polarisation Degree of diffuse polarisation
a
1
0.6
T
||
T
0.4
⊥
0.2 0
0
30
Emittance angle
60
90
Figure 5. Fresnel coefficients for Fig. 4 (n = 1.5). for a given emittance angle. Fig. 5 shows the result of this calculation for a typical dielectric with an additional factor of 1/n introduced due to a difference in wave impedance. Using (6), the degree of diffuse polarization is ρd =
Tk (1/n, θi0 ) − T⊥ (1/n, θi0 ) Tk (1/n, θi0 ) + T⊥ (1/n, θi0 )
(8)
Snell’s Law (3) can be used to interchange between internal angle of incidence, θi0 , and the more useful angle of emittance, θt0 . When the surface is viewed from this angle of emittance, θt0 is the zenith angle, which from here shall be referred to simply as θ. The relevant Fresnel equations can be substituted into the equation to obtain ρ in terms of the refractive index and surface zenith angle. Fig. 3b shows the degree of diffuse polarization for different angles using the resulting equation: 2
ρd =
(n − 1/n) sin2 θ
p n2 − sin2 θ (9) Using the sine fitting method described above to obtain Imin and Imax (hence ρ) and φ, the zenith angle of the normal can be calculated using (9). The azimuth angle of the normal can be determined using the same method as that used for specular reflection, except that a phase shift of 90◦ is necessary: α = φ or φ + 180◦ . The need for a phase shift is illustrated by Fig. 5, which shows that light polarized parallel to the plane of incidence the highest transmission coefficient a so greater intensity is associated with a polarizer 2
2 − 2n2 − (n + 1/n) sin2 θ + 4 cos θ
at that orientation. This is in contrast to specular reflection (Fig. 2).
4
Algorithm Overview
The algorithm to recover depth can be divided into five main sections
viewing direction and the projection of the surface normal onto the horizontal plane. Note that θD falls in the interval [−90◦ , +90◦ ], where negative values indicate that the surface normal is directed to the left (i.e. α > 180◦ ). At this stage however, the sign of θD is unknown since φ falls within the interval [0, 180◦ ). |θD | is given by
1. Calculation of phase and degree of polarization images. 2. Selection of potential correspondences based of the above calculation. 3. Determination of the most probable correspondences from the selected set of points. 4. Disambiguation of azimuth angles. 5. Needle map integration to form depth map (using the Frankot-Chellappa algorithm [5]).
4.1
Phase and Degree of Polarization
Only a brief overview of this step is given, since we use existing methods to calculate the phase and degree of polarization. Further details can be found in [9, 10]. To obtain the raw images for processing, the object was placed on a rotatable table illuminated by a single light source located near the camera. The walls and table were matte black so that the any pixels with an intensity below a threshold could be treated as background. Images were taken with the polarizer oriented at 5◦ intervals. The second view of the object was found by rotating the object by 20◦ . The objective of the first part of the algorithm is to convert these images into two phase images, one from each view, and two degree of polarization images. This task is performed by applying a Levenberg-Marquardt least squares fitting algorithm to observed pixel brightnesses as the polarizer is rotated. The function that the algorithm fits, of course, is the TRS (4). The phase and degree of polarization can then easily be calculated from the result. (For increments of less than 5◦ , Levenberg-Marquardt fitting is actually unnecessarily complicated and time consuming. It was included so that the algorithm can later be used with greater increments in polarizer angle). Throughout this work we assume that the image is formed by orthographic projection.
4.2
Locating Potential Correspondences
The purpose of this stage is to select pixels from each image that have similar surface normals and to form a list of potential correspondences. Also in this section, some of the possibilities are discarded, following the imposition of a gradient constraint. Before the initial selection of points is made, an angle is calculated from which the algorithm derives all correspondences. This angle, θD , is defined to be that between the
|θD | = arctan (sin (φ) tan (θ))
(10)
Knowledge of θD combines information from the surface azimuth and zenith angles (not without loss2 ) into a single angle, which allows reconstruction of depth of any single slice of the object. After θD has been calculated for each pixel, a search is performed for points with certain values and listed as potential correspondences. For example, if the angle by which the object was rotated was θrot = 20◦ , then it would be reasonable to search for θD = 70◦ (= θ0 ) in the unrotated image, and θD = θ0 − θrot = 50◦ in the rotated image. Because the rotation is about a vertical axis, we know that points on the object move about a horizontal plane. This means that the two points in any correspondence lie in the same horizontal line in the images. More correspondences can be found using other values of θ0 later. Note that the occluding contour (i.e. θD = 90◦ ) in the unrotated image corresponds to 90◦ − θrot in the rotated image. Fig. 6a shows points on a (simulated) sphere that have θD = 70◦ . In Fig. 6b, which shows the sphere rotated by 20◦ (obviously with an identical intensity image) points are highlighted where θD = 50◦ . Without prior knowledge of surface geometry, we know that some of the highlighted points from Fig. 6a are likely to correspond to those in Fig. 6b, but it is not yet clear, for example, whether point A corresponds to A0 or B 0 . Indeed, it may be that A is occluded in the second image so no correspondence exists for that point. All the selected points in Fig. 6 form long curves. Indeed, one can expect to find that the selected points form several long curves for most objects. Clearly, finding correspondences for entire curves [3] is desirable since matching a whole curve is more reliable than a single point. In the future we intend to exploit this fact, although for now we simply discard points that do not lie on a curve of a certain minimum length. The minimum length of curves is not at all a critical parameter. Because two of the main aims of this work were to develop an algorithm that is robust to noise and successful when applied to objects with few or no distinctive features, the following steps have been taken: 1) selected points that 2 It
seems from our experiments that this loss of data has no significant impact on the reliability of results. However, if future work does not support this empirical observation, it should be straightforward to reintroduce the lost phase information, forming an extra constraint.
A
4.3
b
a
B
A’
B’
This stage of the algorithm takes the remaining possibilities and decides which are genuine correspondences, and which to discard. This is done by locally reconstructing the height of parts of surfaces near the selected points and comparing the results from each view. The surface height, z is calculated in the vicinity of each selected point using standard integration methods [15]:
Figure 6. a) Simulated |θD | image for a sphere. Darker areas have higher values. The highlighted regions have |θD | = θ0 , in this case 70◦ . b) Same sphere rotated by 20◦ . Here points with θD = 50◦ are highlighted.
Figure 7. Considerations of the sign in the gradient of |θD | alone is sufficient to prove that the two highlighted points do not correspond.
are part of, or adjacent to, abrupt changes in phase or degree of polarization, are discarded. 2) Moderately intense smoothing is applied to the featureless areas. Since these areas have no features, the problem of over-smoothing, which is frequently encountered in computer vision, is not an issue. Consider an object rotating clockwise if viewed from above. Now imagine the object viewed horizontally. Most, and on many occasions all, of the points on the visible surface of the object will be moving away from the occluding contour that is to left of the point, and toward the contour to its right. This fact is used to reduce the number of potential correspondences for a given point. On the occasions where this is not the case, then no correspondences will be detected for those points. The final stage in selecting potential correspondences involves applying a gradient constraint. Fig. 7 shows crosssectional slices of different parts of a surface. It is clear from this figure that the highlighted points do not correspond since the gradient of θD has a different sign. This observation is used by the algorithm to reduce the number of potential correspondences further. This step is not strictly necessary, since the method described below automatically deals with such situations. It is included since it reduces overall computation time and adds extra reliability to results.
Final Estimates of Correspondences
px = tan (θD ) zn =
n X
px n − c
(11)
where px is the x-component of the surface normal (in 2D) and c is a constant chosen such that the height of the reconstruction at the potentially corresponding point is 0. The area of surface that is reconstructed depends upon its location. The surface is not reconstructed where θD is close to zero since the diffuse polarization theory is less accurate here. Nor is it reconstructed at the abrupt surface features mentioned earlier. Reconstructed surface segments from the rotated image are then rotated by −θrot and aligned with the point’s potential correspondence. After resampling so that all reconstructed points are aligned, the root-mean-square (RMS) difference, ε, between the two surfaces is calculated. The final list of correspondences are then found by the following method: Any potential correspondences with the RMS error below a threshold are discarded. From the remainder, the combination of correspondences that gives the least total RMS error are accepted.
4.4
Disambiguation
After correspondences have been located using a range of θ0 , many points on the surface are unambiguous since the sign of θD for the two points of any detected correspondence is the same. For the remaining points the following procedure was used: Correspondences are sorted according to reliability by introducing the confidence measure given below (a more meaningful measure will be derived for future work): fc = lθ0 /ε (12) where l is the length of the reconstructed slice. The algorithm then propagates away from the correspondences in order of confidence and sets the surface azimuth angle to whichever is closest to the local mean, either φ or φ + 180◦ . This process continues until an area is reached where θD < θmin , (where θmin ≈ 20◦ ), at which point, the next correspondence is used. A few areas, mainly where θ0 < θmin still remain ambiguous. These regions are simply “closed” by aligning with the local mean azimuth angle. Such regions are unimportant, since the surface height changes only slightly across these areas.
Figure 8. Various stages in the processing of porcelain bear images. From left: greyscale images from the original view and after a 20◦ rotation; phase image; degree of polarization (dark areas are highest); |θD |, (the sign of θD is unknown at this stage and, again, dark areas are higher); potential correspondences before stage 3 of algorithm; final estimates of correspondences (for a single value of θ0 ); disambiguated azimuth angles.
5
Results
To demonstrate the various stages of the algorithm, consider the porcelain bear shown in Fig. 8. The phase images give the results that one would expect apart from a few small areas (near arms, legs and ears) where the phase has deviated by 90◦ from the expected value. This is because interreflections are present, where the incident light has been reflected twice or more without absorption, meaning that the theory for specular reflection is obeyed. With a few exceptions, the correspondences have been accurately located. The addition of the constraint that corresponding points move away from the left occluding contour clearly plays an important (and computationally inexpensive) role. The disambiguation routine appears to work well except for areas where θD is near zero, as indicated by the sharp changes in azimuth angle close to 180◦ . This is due to the unsophisticated final stage of processing, which attempts to disambiguate these areas. The algorithm, including the Frankot-Chellappa needle map integration, has been applied to porcelain objects of various shapes. Fig. 9 shows some of the results. The depth maps are rendered as Lambertian with a single point light source from above. Both the bear and the urn in Fig. 9 show details have been recovered. These details would have caused difficulty for most existing techniques since normal stereo would have been unable to locate correspondences and single view polarization analysis and shape from shading would have been unable to disambiguate the azimuth angles. The technique does stumble slightly where interreflections occur. For example, the bears hands and feet are more protruded in the actual object, but inter-reflections have hidden away the steep surface gradients. The cat in Fig. 9 has some different coloured paints added to its face and back but correspondences were still accurately found and the disambiguation routine was largely
successful. Note however, that paints with a significantly different refractive index would have caused inaccurate zenith angle estimates. The bottom-right of Fig. 9 shows that the original colours can be added to the depth map to allow the object to be rendered from different views. It is also possible to simulate different lighting conditions by taking a cross-polaroid image, i.e. an image taken with a horizontal linear polarizer placed between the object and light source and a vertical linear polarizer placed between the object and the camera. This removes all specularities. The cross-polaroid cat image is then warped across the surface and re-illuminated artificially. Mainly due to occlusion, wide baseline stereo (stereo with the two views separated by a large distance/angle) encounters difficulty with many algorithms. This one is not an exception. However, Fig. 10 shows that whilst results using a wide baseline in this method are poorer than when the two views are close together, the overall shape has still been recovered. Fig. 11 shows the accuracy of the recovered heights for a correctly disambiguated region of an object (a circular horizontal cross-section of the urn in Figs. 9 and 10). Although not all surface regions are recovered as well as this, it does show that, away from complications such as interreflections, the height can be determined to a high degree of accuracy. It also shows that where the geometry is simple, the results are reasonable even with a wide baseline.
6
Conclusion
A new method for height reconstruction has been presented that makes use of polarization information from two views. The results are very promising, with the majority of azimuth angles correctly disambiguated and the surface height accurately recovered. There are nevertheless several possible improvements to the technique. Including the de-
°
Error
θrot = 40
Exact θ
rot
= 20°
Cross section
Figure 11. Comparison between recovered height of a circular cross section of the urn recovered using views separated by 20◦ and 40◦ and the exact height.
Figure 9. Left: greyscale images of plain white porcelain urn and porcelain cat with a small amount of paint. Recovered depth maps for the objects and the bear from Fig. 8. In the bottom-right depth map, the original colours have been mapped onto the surface.
Figure 10. Urn recovered using two views separated by 20◦ (left) and 40◦ .
tection of surface features (for objects with one or more textured regions), as in previous stereo methods, would provide a greater density of correspondences. The reliability of correspondences would be increased by curve matching techniques, as can be seen in Fig. 8. Isolating regions of inter-reflections (possibly by detecting sharp changes in azimuth angle) would allow more accurate disambiguation across the object. Accuracy in zenith angle would be improved by the use of normals from both views when applying the Frankot-Chellappa algorithm. Finally, a more careful choice of which values of θ0 to use would increase the algorithms efficiency.
References [1] M. Born and E. Wolf. Principles of Optics. Electromagnetic Theory of Propagation, Interference and Diffraction of Light. Pergamon, London, 1959.
[2] M. Z. Brown and G. D. Hager. Advances in computational stereo. IEEE Trans. Patt. Anal. Mach. Intell., 25:993–1008, 2003. [3] H. Chui and A. Rangarajan. A new point matching algorithm for non-rigid registration. Computer Vision and Image Understanding, 89:114–141, 2003. [4] R. L. Cook and K. E. Torrance. A reflectance model for computer graphics. ACM Transactions on Computer Graphics, 1:7–24, 1982. [5] R. T. Frankot and R. Chellappa. A method for enforcing integrability in shape from shading algorithms. IEEE Trans. Patt. Anal. Mach. Intell., 10:439–451, 1988. [6] E. Hecht. Optics. Addison Wesley Longman, third edition, 1998. [7] J. J. Koenderink. What does the occluding contour tell us about solid shape? Perception, 13:321–330, 1984. [8] D. Miyazaki, M. Kagesawa, and K. Ikeuchi. Transparent surface modelling from a pair of polarization images. IEEE Trans. Patt. Anal. Mach. Intell., 26:73–82, 2004. [9] D. Miyazaki, R. T. Tan, K. Hara, and K. Ikeuchi. Polarization-based inverse rendering from a single view. In Proc. ICCV, volume 2, pages 982–987, 2003. [10] S. Rahmann. Polarization images: A geometric interpretation for shape analysis. In Proc. of ICPR, volume 3, pages 542–546, 2000. [11] L. B. Wolff. Polarization vision: A new sensory approach to image understanding. Image and Vision Computing, 15:81– 93, 1997. [12] L. B. Wolff and T. E. Boult. Constraining object features using a polarisation reflectance model. IEEE Trans. Pattern Anal. Mach. Intell, 13:635–657, 1991. [13] K.-Y. K. Wong and R. Cipolla. Reconstruction of sculpture from its profiles with unknown camera positions. Trans. Image Processing, 13:381–389, 2004. [14] R. J. Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineering, 19:139–144, 1980. [15] Z. Wu and L. Li. A line-integration based method for depth recovery from surface normals. In CVGIP(43), pages 53–66, 1988. [16] T. Zickler, P. N. Belhumeur, and D. J. Kriegman. Helmholtz stereopsis: Exploiting reciprocity for surface reconstruction. International Journal of Computer Vision, 49:215–227, 2002.
1. Introduction Multiple view techniques have proved to be highly effective in recovering models of surface shape. However, most of the existing methods are largely geometric in nature and rely on the availability of salient surface features or interest points to establish correspondence. Unfortunately, these methods are not particularly effective in the recovery of surface shape for smooth featureless surfaces. One alternative is to use photometric methods to characterise the surface, and this is the basis for techniques such as photometric stereo. However, such methods implicitly assume correspondence is known since they rely on the capture of a relatively large number of images under fixed object position and varying light source direction. One source of information that has not been so widely used on the photometric recovery of surface shape is that provided by polarization. The aim in this paper is therefore to explore whether information provided by the polariza-
tion state of diffusely reflected light can be used to establish correspondences on featureless surfaces for the purposes of stereoscopic depth recovery.
1.1
Previous Work
Many methods exist for combining information about an object or scene from two or more views [2]. The basic principle behind most computational stereo is that with two known views of an object it is possible to calculate the three-dimensional location of a point that is visible in both images. One major difficulty with stereo is deciding which points in one image correspond to which points in the second. A different approach, photometric stereo [14], involves the object under study being kept static with respect to the camera, and the direction of the light source being varied. In Helmholtz stereopsis [16], on the other hand, unique surface normal recovery is possible if the light source and camera are interchanged. Occluding contours, i.e. the outer boundaries of the image of the object under study [7], provide another source of information on surface structure. Although this information is limited (but reliable) from a single view, it can reveal much more when several views are used, placing heavy constraints on surface geometry [13]. Analysis of the polarization of light caused by surface reflection has been frequently used as a means to provide constraints on surface geometry. When initially unpolarized light is reflected from a surface, it becomes polarized [12]. This applies to both specular reflection (which we refer to as specular polarization) and diffuse reflection (diffuse polarization) and is due to the directionality of the molecular electron charge density interacting with the electromagnetic field of the incident light [6]. Most research aimed at extracting and interpreting information from polarization data, involves placing a linear polarizer in front of a camera and taking images of an object or a scene with the polarizer oriented at different angles [12, 9]. Other work has involved experimenting with liquid
crystal technology to enable rapid acquisition of polarization images [11]. Ikeuchi and colleagues have used specular polarization in shape recovery, where there is a need for specular reflections across the whole surface. This was done by placing the object under investigation inside a spherical diffuser, with several light sources outside and a hole for the camera [8]. Diffuse polarization [9, 10] can also be used in shape recovery. Although in this case the polarizing effects are weaker, especially far from object limbs, the global specular reflection is not required, meaning less specific lighting conditions are necessary.
1.2
E0i||
E0r||
E0i ni=1 nt
E0r
qr
qi
qt E0t
E0t |
Figure 1. Definitions. Directions of electric fields are indicated. N.B. θi = θr . The electrons in the medium vibrate parallel to the electric field.
Contribution
The aim of this paper is to describe a new method for shape recovery that uses polarization data from images of an object taken from different viewpoints. This firstly obtains a set of correspondences; secondly a field of surface normals; and finally a depth map. To do this, we make use of Fresnel Theory, which relates the reflected and incident wave amplitudes and provides a route to estimating the surface normals from polarization data. The available information is the degree of polarization and phase of the reflected light. The degree of polarization determines the zenith angle. The phase ambiguously defines azimuth angle. Surface normal determination is most reliable when the degree of polarization and, therefore the zenith angle, are large, that is close to the occluding boundary. The ambiguities normally associated with single view shape recovery techniques are eliminated here by a correspondence detection algorithm, which makes use of surface normals recovered from polarization data. Where these normals are similar in the two images, a correspondence is likely. Other constraints are imposed, based on simple geometric arguments, to add confidence to the location of correspondences. For example, if the object is being rotated and we know the direction of rotation, then we can predict the direction of motion of point, relative to the occluding contour. The method assumes that the angle between the two views is known, but a detailed photometric or geometric calibration is not necessary. This is in contrast to many stereo methods where the full epipolar geometry must be known. Indeed the introduction of an optimisation method should allow for an estimate the angle between views to be made. Results clearly demonstrate accurate depth recovery. Since the correspondence detection works for smooth surface regions, it is possible to apply moderately intense smoothing without facing over-smoothing problems. This makes the method robust to camera noise. The combined use of stereo vision and shape from diffuse polarization is an attractive one because the two techniques are complementary. Where stereo is weak – namely
near occluding contours and featureless surface areas – diffuse polarization is at its strongest. On the other hand, stereo allows the convex/concave ambiguity to be solved. The technique reported here should pave the way for future algorithms that use distinctive surface features, as in earlier stereo work, in addition to polarization. Ultimately, the development of a novel sensor will be possible.
2
Fresnel Reflectance
The principle on which this work is based is that when unpolarized light impinges upon a surface, a partially polarized reflection is formed. The degree of polarization (defined more precisely below) and the angle at which the reflected light is polarized reveals much about the surface. We now discuss the physics behind this spontaneous polarization, starting with a description of a classical picture of the origin the phenomenon. The electric field of an electromagnetic wave incident on a surface causes electrons of the reflecting medium near the surface to vibrate, forming dipoles. These vibrating electrons re-radiate, generating the reflected rays. The electrons vibrate parallel to the electric field. If the electric field of the incident wave is perpendicular to the plane of incidence, then so will be the electric field of the reflected wave. This is because the reflected wave is formed by the vibrating electrons – the reverse process to how they were initially set in motion. If, on the other hand, the incident light is polarized parallel to the plane of incidence, the electrons do not vibrate perpendicularly to the reflected ray, as Fig. 1 shows, resulting in a more attenuated wave. As the figure suggests, the degree of polarization depends on the angle of incidence.
2.1
Fresnel Coefficients
The Fresnel equations give the ratios of reflected wave amplitude to incident wave amplitude for incident light that is linearly polarized perpendicular to, or parallel to, the plane of specular incidence [1]. These ratios depend upon
era is rotated, the measured intensity varies sinusoidally according to the transmitted radiance sinusoid (TRS):
1
Fresnel Coefficient
T|| 0.8
T
⊥
0.4 R
⊥
0.2 0
Imax + Imin Imax − Imin + cos (2θpol − 2φ) 2 2 (4) where Imax and Imin are the maximum and minimum observed intensities as the polarizer is rotated, θpol is the angle which the polarizer makes with the arbitrary reference direction (here vertically upwards) and φ is the phase angle, or angle of polarization of reflected light, between 0◦ and 180◦ . From Fig. 2, it is clear that the maximum and minimum intensities detected for a particular surface orientation are I (θpol , φ) =
0.6
R|| 0
30 60 Specular angle of incidence
90
Figure 2. Reflection and transmission coefficients for a typical dielectric (n = 1.5).
Imax = the angle of incidence and the refractive index of the reflecting medium. Since the incident light can be always be resolved into components perpendicular to, and parallel to, the plane of incidence, the Fresnel equations are applicable to all incident polarization states. For the geometry of Fig. 1, and assuming that the material is non-ferrous, the amplitude reflection coefficients for perpendicularly polarized light, r⊥ , and parallel polarized light, rk , at a boundary between two media are given by E0r⊥ ni cos θi − nt cos θt = E0i⊥ ni cos θi + nt cos θt E0rk nt cos θi − ni cos θt = rk ≡ E0ik nt cos θi + ni cos θt
r⊥ ≡
(1) (2)
where ni and nt are the refractive indices of the first and second media and θi and θt are incident and transmitted angles. θt can be obtained from the well-known Snell’s Law: ni sin θi = nt sin θt
(3)
More useful are the intensity coefficients since it is the intensity of reflected light that is measured by cameras. 2 and Rk = rk2 [1]. All this assumes These are R⊥ = r⊥ that the refractive index is wavelength independent [4]. In fact, there is some wavelength dependence, but the equations above provide reasonable results for many situations. Fig. 2 shows the Fresnel intensity coefficients for a typical dielectric. It is the fact that R⊥ (θi ) 6= Rk (θi ) which underpins most of this research. This means that the reflected light is partially linearly polarized, that is, the light is a superposition of an unpolarized component, and a completely polarized component.
3 3.1
Polarization Vision The Polarization Image
We now consider how the above theory can be used in computer vision. As a polarizer placed in front of the cam-
R⊥ Is ; R⊥ + R k
Imin =
Rk Is R⊥ + R k
(5)
where Is is the magnitude of the specular component of reflection (assume for now, that there is no diffuse reflection). The degree of polarization or partial polarization, which is frequently used in computer vision, is defined by ρ=
Imax − Imin Imax + Imin
(6)
The intensity, phase and degree of polarization collectively form the polarization image. We obtain the phase and degree of polarization components by taking a series of images of an object using a digital camera with a linear polarizer mounted on its lens. The polarizer is rotated by a small amount between successive images and the Levenberg-Marquardt non-linear sine fitting algorithm is used to fit the measured intensities to (4) at each pixel.
3.2
Shape from Polarization
As Fig. 2 shows, the reflected light is attenuated to a greater extent if it is polarized parallel to the plane of incidence. Thus, greatest transmission through the polarizer occurs when the polarizer is oriented at an angle 90◦ from the azimuth angle of the surface, α. This provides a direct method for calculating azimuth angle: α = φ ± 90◦ . The ambiguity that this implies is a result of the phase image only having values in the range [0, 180◦ ). The zenith angle, the angle which the normal makes with the viewing direction, can be computed by considering the degree of polarization. Substituting (5) into (6) gives the degree of specular polarization in terms of the Fresnel coefficients. Then using the Fresnel Equations gives ρ in terms of n and the zenith angle, θ: p 2 sin2 θ cos θ n2 − sin2 θ ρs = 2 (7) n − sin2 θ − n2 sin2 θ + 2 sin4 θ Fig. 3a shows the dependence of ρs on θ. Note that there is an angle (known as the Brewster angle) where the light
n = 1.4
0.8
n = 1.6
0.6
qt ’
ni=1 nt
0.4
qi’ qi’
0.2 0
0
10
20
1
30 40 50 60 Specular angle of incidence b
70
80
90
Figure 4. Transmission of internally scattered light back into air.
0.8 0.6 n = 1.6
0.4 0.2 0
n = 1.4 0
10
20
30
40 50 60 Emittance angle
70
80
90
Figure 3. Degree of polarization for (a) specular and (b) diffuse reflection for two different refractive indices.
is totally polarized1 . Also, there is a 2-to-1 mapping of ρs to θ, resulting in another, different ambiguity that must be solved. Miyazaki et al [8] solve this problem using two views of the object. Diffuse polarization is a result of the following process [12]: A portion of the incident light penetrates the surface, is partially polarized in the process, as predicted by the Fresnel equations, and is refracted. Due to the random nature of internal scattering, the light becomes depolarized. Some of the light is then refracted back into the air and is, once again, refracted and partially polarized. The degree of diffuse polarization is highest, and so least noisy, near occluding boundaries where the zenith angle is large. Diffuse polarization has the advantage that less controlled lighting conditions are required than for specular polarization but has the disadvantages that it is not applicable to metals or transparent objects and is more susceptible to noise. When light approaches the surface-air interface from within the medium, as shown in Fig. 4, a similar process to that discussed earlier takes place but with the relative index of refraction being 1/n instead of n (assuming refractive index of air = 1). If the internal angle of incidence is above a critical angle (arcsin 1/n), then total internal reflection occurs. Otherwise, Snells Law (3) can be used to find the angle of emittance for any given angle of internal incidence. The Fresnel transmission coefficient can then be calculated 1 In practice ρ will be a little less than 1 due to a small, but finite, s diffuse component of reflection.
Transmission coefficient
Degree of specular polarisation Degree of diffuse polarisation
a
1
0.6
T
||
T
0.4
⊥
0.2 0
0
30
Emittance angle
60
90
Figure 5. Fresnel coefficients for Fig. 4 (n = 1.5). for a given emittance angle. Fig. 5 shows the result of this calculation for a typical dielectric with an additional factor of 1/n introduced due to a difference in wave impedance. Using (6), the degree of diffuse polarization is ρd =
Tk (1/n, θi0 ) − T⊥ (1/n, θi0 ) Tk (1/n, θi0 ) + T⊥ (1/n, θi0 )
(8)
Snell’s Law (3) can be used to interchange between internal angle of incidence, θi0 , and the more useful angle of emittance, θt0 . When the surface is viewed from this angle of emittance, θt0 is the zenith angle, which from here shall be referred to simply as θ. The relevant Fresnel equations can be substituted into the equation to obtain ρ in terms of the refractive index and surface zenith angle. Fig. 3b shows the degree of diffuse polarization for different angles using the resulting equation: 2
ρd =
(n − 1/n) sin2 θ
p n2 − sin2 θ (9) Using the sine fitting method described above to obtain Imin and Imax (hence ρ) and φ, the zenith angle of the normal can be calculated using (9). The azimuth angle of the normal can be determined using the same method as that used for specular reflection, except that a phase shift of 90◦ is necessary: α = φ or φ + 180◦ . The need for a phase shift is illustrated by Fig. 5, which shows that light polarized parallel to the plane of incidence the highest transmission coefficient a so greater intensity is associated with a polarizer 2
2 − 2n2 − (n + 1/n) sin2 θ + 4 cos θ
at that orientation. This is in contrast to specular reflection (Fig. 2).
4
Algorithm Overview
The algorithm to recover depth can be divided into five main sections
viewing direction and the projection of the surface normal onto the horizontal plane. Note that θD falls in the interval [−90◦ , +90◦ ], where negative values indicate that the surface normal is directed to the left (i.e. α > 180◦ ). At this stage however, the sign of θD is unknown since φ falls within the interval [0, 180◦ ). |θD | is given by
1. Calculation of phase and degree of polarization images. 2. Selection of potential correspondences based of the above calculation. 3. Determination of the most probable correspondences from the selected set of points. 4. Disambiguation of azimuth angles. 5. Needle map integration to form depth map (using the Frankot-Chellappa algorithm [5]).
4.1
Phase and Degree of Polarization
Only a brief overview of this step is given, since we use existing methods to calculate the phase and degree of polarization. Further details can be found in [9, 10]. To obtain the raw images for processing, the object was placed on a rotatable table illuminated by a single light source located near the camera. The walls and table were matte black so that the any pixels with an intensity below a threshold could be treated as background. Images were taken with the polarizer oriented at 5◦ intervals. The second view of the object was found by rotating the object by 20◦ . The objective of the first part of the algorithm is to convert these images into two phase images, one from each view, and two degree of polarization images. This task is performed by applying a Levenberg-Marquardt least squares fitting algorithm to observed pixel brightnesses as the polarizer is rotated. The function that the algorithm fits, of course, is the TRS (4). The phase and degree of polarization can then easily be calculated from the result. (For increments of less than 5◦ , Levenberg-Marquardt fitting is actually unnecessarily complicated and time consuming. It was included so that the algorithm can later be used with greater increments in polarizer angle). Throughout this work we assume that the image is formed by orthographic projection.
4.2
Locating Potential Correspondences
The purpose of this stage is to select pixels from each image that have similar surface normals and to form a list of potential correspondences. Also in this section, some of the possibilities are discarded, following the imposition of a gradient constraint. Before the initial selection of points is made, an angle is calculated from which the algorithm derives all correspondences. This angle, θD , is defined to be that between the
|θD | = arctan (sin (φ) tan (θ))
(10)
Knowledge of θD combines information from the surface azimuth and zenith angles (not without loss2 ) into a single angle, which allows reconstruction of depth of any single slice of the object. After θD has been calculated for each pixel, a search is performed for points with certain values and listed as potential correspondences. For example, if the angle by which the object was rotated was θrot = 20◦ , then it would be reasonable to search for θD = 70◦ (= θ0 ) in the unrotated image, and θD = θ0 − θrot = 50◦ in the rotated image. Because the rotation is about a vertical axis, we know that points on the object move about a horizontal plane. This means that the two points in any correspondence lie in the same horizontal line in the images. More correspondences can be found using other values of θ0 later. Note that the occluding contour (i.e. θD = 90◦ ) in the unrotated image corresponds to 90◦ − θrot in the rotated image. Fig. 6a shows points on a (simulated) sphere that have θD = 70◦ . In Fig. 6b, which shows the sphere rotated by 20◦ (obviously with an identical intensity image) points are highlighted where θD = 50◦ . Without prior knowledge of surface geometry, we know that some of the highlighted points from Fig. 6a are likely to correspond to those in Fig. 6b, but it is not yet clear, for example, whether point A corresponds to A0 or B 0 . Indeed, it may be that A is occluded in the second image so no correspondence exists for that point. All the selected points in Fig. 6 form long curves. Indeed, one can expect to find that the selected points form several long curves for most objects. Clearly, finding correspondences for entire curves [3] is desirable since matching a whole curve is more reliable than a single point. In the future we intend to exploit this fact, although for now we simply discard points that do not lie on a curve of a certain minimum length. The minimum length of curves is not at all a critical parameter. Because two of the main aims of this work were to develop an algorithm that is robust to noise and successful when applied to objects with few or no distinctive features, the following steps have been taken: 1) selected points that 2 It
seems from our experiments that this loss of data has no significant impact on the reliability of results. However, if future work does not support this empirical observation, it should be straightforward to reintroduce the lost phase information, forming an extra constraint.
A
4.3
b
a
B
A’
B’
This stage of the algorithm takes the remaining possibilities and decides which are genuine correspondences, and which to discard. This is done by locally reconstructing the height of parts of surfaces near the selected points and comparing the results from each view. The surface height, z is calculated in the vicinity of each selected point using standard integration methods [15]:
Figure 6. a) Simulated |θD | image for a sphere. Darker areas have higher values. The highlighted regions have |θD | = θ0 , in this case 70◦ . b) Same sphere rotated by 20◦ . Here points with θD = 50◦ are highlighted.
Figure 7. Considerations of the sign in the gradient of |θD | alone is sufficient to prove that the two highlighted points do not correspond.
are part of, or adjacent to, abrupt changes in phase or degree of polarization, are discarded. 2) Moderately intense smoothing is applied to the featureless areas. Since these areas have no features, the problem of over-smoothing, which is frequently encountered in computer vision, is not an issue. Consider an object rotating clockwise if viewed from above. Now imagine the object viewed horizontally. Most, and on many occasions all, of the points on the visible surface of the object will be moving away from the occluding contour that is to left of the point, and toward the contour to its right. This fact is used to reduce the number of potential correspondences for a given point. On the occasions where this is not the case, then no correspondences will be detected for those points. The final stage in selecting potential correspondences involves applying a gradient constraint. Fig. 7 shows crosssectional slices of different parts of a surface. It is clear from this figure that the highlighted points do not correspond since the gradient of θD has a different sign. This observation is used by the algorithm to reduce the number of potential correspondences further. This step is not strictly necessary, since the method described below automatically deals with such situations. It is included since it reduces overall computation time and adds extra reliability to results.
Final Estimates of Correspondences
px = tan (θD ) zn =
n X
px n − c
(11)
where px is the x-component of the surface normal (in 2D) and c is a constant chosen such that the height of the reconstruction at the potentially corresponding point is 0. The area of surface that is reconstructed depends upon its location. The surface is not reconstructed where θD is close to zero since the diffuse polarization theory is less accurate here. Nor is it reconstructed at the abrupt surface features mentioned earlier. Reconstructed surface segments from the rotated image are then rotated by −θrot and aligned with the point’s potential correspondence. After resampling so that all reconstructed points are aligned, the root-mean-square (RMS) difference, ε, between the two surfaces is calculated. The final list of correspondences are then found by the following method: Any potential correspondences with the RMS error below a threshold are discarded. From the remainder, the combination of correspondences that gives the least total RMS error are accepted.
4.4
Disambiguation
After correspondences have been located using a range of θ0 , many points on the surface are unambiguous since the sign of θD for the two points of any detected correspondence is the same. For the remaining points the following procedure was used: Correspondences are sorted according to reliability by introducing the confidence measure given below (a more meaningful measure will be derived for future work): fc = lθ0 /ε (12) where l is the length of the reconstructed slice. The algorithm then propagates away from the correspondences in order of confidence and sets the surface azimuth angle to whichever is closest to the local mean, either φ or φ + 180◦ . This process continues until an area is reached where θD < θmin , (where θmin ≈ 20◦ ), at which point, the next correspondence is used. A few areas, mainly where θ0 < θmin still remain ambiguous. These regions are simply “closed” by aligning with the local mean azimuth angle. Such regions are unimportant, since the surface height changes only slightly across these areas.
Figure 8. Various stages in the processing of porcelain bear images. From left: greyscale images from the original view and after a 20◦ rotation; phase image; degree of polarization (dark areas are highest); |θD |, (the sign of θD is unknown at this stage and, again, dark areas are higher); potential correspondences before stage 3 of algorithm; final estimates of correspondences (for a single value of θ0 ); disambiguated azimuth angles.
5
Results
To demonstrate the various stages of the algorithm, consider the porcelain bear shown in Fig. 8. The phase images give the results that one would expect apart from a few small areas (near arms, legs and ears) where the phase has deviated by 90◦ from the expected value. This is because interreflections are present, where the incident light has been reflected twice or more without absorption, meaning that the theory for specular reflection is obeyed. With a few exceptions, the correspondences have been accurately located. The addition of the constraint that corresponding points move away from the left occluding contour clearly plays an important (and computationally inexpensive) role. The disambiguation routine appears to work well except for areas where θD is near zero, as indicated by the sharp changes in azimuth angle close to 180◦ . This is due to the unsophisticated final stage of processing, which attempts to disambiguate these areas. The algorithm, including the Frankot-Chellappa needle map integration, has been applied to porcelain objects of various shapes. Fig. 9 shows some of the results. The depth maps are rendered as Lambertian with a single point light source from above. Both the bear and the urn in Fig. 9 show details have been recovered. These details would have caused difficulty for most existing techniques since normal stereo would have been unable to locate correspondences and single view polarization analysis and shape from shading would have been unable to disambiguate the azimuth angles. The technique does stumble slightly where interreflections occur. For example, the bears hands and feet are more protruded in the actual object, but inter-reflections have hidden away the steep surface gradients. The cat in Fig. 9 has some different coloured paints added to its face and back but correspondences were still accurately found and the disambiguation routine was largely
successful. Note however, that paints with a significantly different refractive index would have caused inaccurate zenith angle estimates. The bottom-right of Fig. 9 shows that the original colours can be added to the depth map to allow the object to be rendered from different views. It is also possible to simulate different lighting conditions by taking a cross-polaroid image, i.e. an image taken with a horizontal linear polarizer placed between the object and light source and a vertical linear polarizer placed between the object and the camera. This removes all specularities. The cross-polaroid cat image is then warped across the surface and re-illuminated artificially. Mainly due to occlusion, wide baseline stereo (stereo with the two views separated by a large distance/angle) encounters difficulty with many algorithms. This one is not an exception. However, Fig. 10 shows that whilst results using a wide baseline in this method are poorer than when the two views are close together, the overall shape has still been recovered. Fig. 11 shows the accuracy of the recovered heights for a correctly disambiguated region of an object (a circular horizontal cross-section of the urn in Figs. 9 and 10). Although not all surface regions are recovered as well as this, it does show that, away from complications such as interreflections, the height can be determined to a high degree of accuracy. It also shows that where the geometry is simple, the results are reasonable even with a wide baseline.
6
Conclusion
A new method for height reconstruction has been presented that makes use of polarization information from two views. The results are very promising, with the majority of azimuth angles correctly disambiguated and the surface height accurately recovered. There are nevertheless several possible improvements to the technique. Including the de-
°
Error
θrot = 40
Exact θ
rot
= 20°
Cross section
Figure 11. Comparison between recovered height of a circular cross section of the urn recovered using views separated by 20◦ and 40◦ and the exact height.
Figure 9. Left: greyscale images of plain white porcelain urn and porcelain cat with a small amount of paint. Recovered depth maps for the objects and the bear from Fig. 8. In the bottom-right depth map, the original colours have been mapped onto the surface.
Figure 10. Urn recovered using two views separated by 20◦ (left) and 40◦ .
tection of surface features (for objects with one or more textured regions), as in previous stereo methods, would provide a greater density of correspondences. The reliability of correspondences would be increased by curve matching techniques, as can be seen in Fig. 8. Isolating regions of inter-reflections (possibly by detecting sharp changes in azimuth angle) would allow more accurate disambiguation across the object. Accuracy in zenith angle would be improved by the use of normals from both views when applying the Frankot-Chellappa algorithm. Finally, a more careful choice of which values of θ0 to use would increase the algorithms efficiency.
References [1] M. Born and E. Wolf. Principles of Optics. Electromagnetic Theory of Propagation, Interference and Diffraction of Light. Pergamon, London, 1959.
[2] M. Z. Brown and G. D. Hager. Advances in computational stereo. IEEE Trans. Patt. Anal. Mach. Intell., 25:993–1008, 2003. [3] H. Chui and A. Rangarajan. A new point matching algorithm for non-rigid registration. Computer Vision and Image Understanding, 89:114–141, 2003. [4] R. L. Cook and K. E. Torrance. A reflectance model for computer graphics. ACM Transactions on Computer Graphics, 1:7–24, 1982. [5] R. T. Frankot and R. Chellappa. A method for enforcing integrability in shape from shading algorithms. IEEE Trans. Patt. Anal. Mach. Intell., 10:439–451, 1988. [6] E. Hecht. Optics. Addison Wesley Longman, third edition, 1998. [7] J. J. Koenderink. What does the occluding contour tell us about solid shape? Perception, 13:321–330, 1984. [8] D. Miyazaki, M. Kagesawa, and K. Ikeuchi. Transparent surface modelling from a pair of polarization images. IEEE Trans. Patt. Anal. Mach. Intell., 26:73–82, 2004. [9] D. Miyazaki, R. T. Tan, K. Hara, and K. Ikeuchi. Polarization-based inverse rendering from a single view. In Proc. ICCV, volume 2, pages 982–987, 2003. [10] S. Rahmann. Polarization images: A geometric interpretation for shape analysis. In Proc. of ICPR, volume 3, pages 542–546, 2000. [11] L. B. Wolff. Polarization vision: A new sensory approach to image understanding. Image and Vision Computing, 15:81– 93, 1997. [12] L. B. Wolff and T. E. Boult. Constraining object features using a polarisation reflectance model. IEEE Trans. Pattern Anal. Mach. Intell, 13:635–657, 1991. [13] K.-Y. K. Wong and R. Cipolla. Reconstruction of sculpture from its profiles with unknown camera positions. Trans. Image Processing, 13:381–389, 2004. [14] R. J. Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineering, 19:139–144, 1980. [15] Z. Wu and L. Li. A line-integration based method for depth recovery from surface normals. In CVGIP(43), pages 53–66, 1988. [16] T. Zickler, P. N. Belhumeur, and D. J. Kriegman. Helmholtz stereopsis: Exploiting reciprocity for surface reconstruction. International Journal of Computer Vision, 49:215–227, 2002.
