An Automatic Alignment Procedure for a 4-Source ... - Semantic Scholar
IMTC 2006 – Instrumentation and Measurement Technology Conference Sorrento, Italy 24-27 April 2006
An Automatic Alignment Procedure for a 4-Source Photometric Stereo Technique applied to Scanning Electron Microscopy Ruggero Pintus, Simona Podda and Massimo Vanzi Department of Electrical and Electronic Engineering, University of Cagliari Piazza d’Armi, 09123 Cagliari (CA), Italy Phone: +39-070-675-5859, Fax: +39-070-675-5900, Email: [email protected] Abstract – This paper presents an automatic alignment procedure for a 4-Source Photometric Stereo technique to reconstruct the depth map in the Scanning Electron Microscopy. PS, based on the so-called reflectance map, used several images of a surface to estimate the surface depth at each image point. Lambertian reflectivity function is the simplest one. In the SEM one of the most important signals, the Backscattered Electron emission, is nearly Lambertian, and, to simplify matters, SEM images are intrinsically greyscale maps. The possibility for electron-PS is assumed, taking advantage of one of the most exciting features of the technique, which doesn't return some depth illusion from ordinary pictures, but true numerical 3D models. Keywords – Photometric Stereo, Scanning Electron Microscopy, image alignment.
I. INTRODUCTION In a previous paper [1] a 4-Source Photometric Stereo technique has been proposed to reconstruct the third dimension in the Scanning Electron Microscopy (SEM). The original Photometric Stereo (PS) was conceived in the optical domain by Woodham [3-6] some 30 years ago and used several images of a surface, taken from the same viewpoint but under different illumination directions, to estimate the relative surface depth at each image point. So far many papers improved that technique using different approaches [7-15], all based on a suitable choice of the so-called reflectance map, the function that describes how incident light is diffused by a surface. Among those maps, the Lambertian reflectivity function is the simplest one because it allows to formulate the problem in a linear and then a matrix form [16]. In the optical field, the Lambertian case is often an over-simplified view of reality, because specular reflections and chromatic differences call for more complex models. On the contrary, in the SEM one of the most important signals, the Back Scattered Electron emission (BSE), is nearly ideally Lambertian [8]. To further simplify matters, only the intensity of the electron emission is measured, and not its energy, so SEM images are indeed intrinsically greyscale maps. The possibility for electron-PS at the SEM is then assumed, opening a way for some unexplored access to the microscopic world, taking advantage of one of the most exciting features of the technique, which doesn’t return some depth illusion from ordinary pictures, but true numerical 3D models. Giving a closer look to the PS-based 3D recovery method, it consists of the acquisition of 4 Back-Scattered (BS) images,
0-7803-9360-0/06/$20.00 ©2006 IEEE
from which the gradient of the surface and its “intrinsic brilliance” (in electron microscopy, the emission coefficient, that makes gold brighter than carbon also at the SEM) are first separately extracted by means of some image algebra, and then the depth map is recovered by integration of the gradient vector field. Some attempts have been made in the past [18-19] in that direction, but all limited by the choice of having as many BSE detectors simultaneously active as the number of required different light sources for proper PS in the optical domain. Now, this is by far not a standard case and makes the microscope a dedicated instrument, the exact contrary of the most appreciated feature of a SEM, i.e. its extremely wide application range without other devices but the standard equipment. As a result, the most popular current method for 3D electron microscopy still remains the stereoscopy [20], although it is only able to give the binocular perception of the depth. However, most of the benefits of PS-based 3D-SEM may be reached introducing a sequential acquisition for the required 4 BS images. The fee to be paid is to renounce dynamic 3D (which would obviously require simultaneous images to reconstruct a time evolving 3D frame), and to introduce some mechanical movement inside the SEM. It is necessary, indeed, either to move the detector (not at all a recommended choice) or to rotate the specimen under the fixed detector: a referential, solid with the specimen, would display the relative motion as a motion of the light source. In details, the solution consists in sequentially acquiring 4 images of the same specimen upon the imposition onto the specimen of four 90° rotation steps, and then rotating back the images: a sequence of 4 pictures of the same area, each illuminated from a different cardinal point will be obtained. Exact alignment would then be required, in order to have each point on the specimen imaged on the same pixel in each and all the four images. This is a crucial point, because pixel precision is not achieved without image comparison, and this task is nearly impossible when details are differently shaded in each picture. Another recent paper [2] built the foundations of this work, based on the capability of the SEM to use several different signals to produce images, under the same scanning conditions. At least two signals in the SEM, different from the BSE, collected by a standard off-axis detector, display direction-independent contrast, i.e. isotropic shading: the axially-detected BSE, and the Specimen Current (SC). In both cases, the respective detectors are normally
1278
available for any electron microscope, and often come as standard devices in the basic instrument configuration. The advantage of using isotropic shaded images is that their alignment is not complicated by different shading, which may reverse the contrast of a same detail under opposite lighting. The alignment transformations found for the uniformly shaded set are then used to properly align the four BSE images used for 3D reconstruction. In any case, a twin set of images is acquired: one for alignment purposes and the other for 3D reconstruction. The proposed method bases itself on the complete computer control currently available in any SEM, and develops a completely automated tool that enables a standard instrument to access the 3D domain without any dedicated optional tool. The aim of this paper is to demonstrate the pixel precision achieved by the proposed mixed mechanicalnumerical procedure for image alignment. Nevertheless, some comments will also be given on the expected performances of the method, in terms of noise and accuracy. II. THE AUTOMATED SEQUENCE In the electron microscope some specific actions, among the many available, are put into sequence. A. The initial setup
The directional shading is evident in the first image where “lighting” seems to come from the upper side (and simply corresponds to the direction where the off-axis detector is placed), while the second image has contrast variations (the shell is brighter than the surrounding mould) but there is no directionality in the light. Where available, all suitable mechanical calibrations are also performed. For some machines, combined mechanical rotation and xy shift may be set in order to keep the centre of the image at the central point of the screen, to emulate the rigid connection of the optical axis with the mechanical rotation axis. Anyway, as sub-micrometric resolution is reached (magnification larger than 1000x), the mechanical precision becomes considerably insufficient. B. The automated sequence A mixed use of the electronic and mechanical controls is then programmed, as summarized in the following automatic sequence: 1) Acquisition of a pair BS and IS images, as in fig.1. 2) mechanical rotation of the specimen by 90° 3) Acquisition of a reduced IS image in the central area, that is first counter-rotated by 90°, and then
First of all the subject is framed, magnified and focused as for normal optimum SEM imaging. Also brightness and contrast are separately set for the two detectors to be used, i.e. the off-axis BackScattered (BS) electron detector and the available one able to produce isotropic shading, that may be the Specimen Current detector or the complete circular axial BS detector. The latter will be indicated as the detector for Isotropic Shade (IS). Fig.1 reports the BS and IS images that simultaneously form, since both detectors are enabled during the same frame scan.
4) cross-correlated with the previous IS image. This gives the ∆x,∆y shift to be compensated.
Fig.1 BS (left) and IS (right) images of the same frame scan.
frame
rotation
shift
5) The new image centre of the rotated specimen is mechanically brought back by -∆x,-∆y as close as possible to the previous centre. 6) Steps 1-4 are repeated 3 times. At the end of the process (which takes few minutes, including the image acquisition), a twin set of 4+4 images is stored.
1279
BS
IS
Fig.2 A couple of views of the 3D surface from fig.1.
The image processing then automatically proceeds as: 1) Rotation and fine alignment of the IS set: computation of the residual ∆x,∆y with respect to the first IS picture. 2) Rotation and alignment of the BS set by imposing the corresponding -∆x,-∆y numerical shifts. 3) Selection of the maximum common area in the BS image set and cutting of the exceeding stripes. 4) Storage of the resulting four IE, IN, IW, IS BS images for PS computation, separately representing the specimen under the same point of view and orientation, but lighted from the cardinal points.
Fig.3. A set of BS images (upper line), the corresponding IS twin set (central line) and some 3D views (lower line) of the reconstructed surface of a damaged solid-state electron device.
The overall automated sequence lasts some few minutes, depending on the selected acquisition time for the images to be stored, and pixel accuracy is currently achieved.
Fig.4 One of the BS SEM images of the detail of a coin and its 3D reconstruction.
C. 3D processing Since this point, the procedure is the same as for optical Photometric Stereo, it may be demonstrated [16] that the simple image algebra
p = q =
I E − IW I E + IW IN − IS IN + IS
empowers p,q with the role of x,y derivatives of the surface z(x,y), and then the gradient of that surface is available at each pixel of the original image domain. It is a matter of numerical integration to retrieve the depth from the gradient. The operator is now enabled to see the reconstructed surface (fig.2, 3,4 and 5), to rotate it in the virtual space, to render it by false texturing or applying the original SEM contrast, and finally to save the file, including the original pictures and the reconstructed surface. A possibility is included to generate the suitable files for direct 3D printing of the virtual solid by fast-prototyping devices.
Fig.5.Conventional optical image (upper left) and SEM detail (right) of the percussion surface of an exploded bullet-case. The lower image reveals the 3D shape.
1280
III.
PERFORMANCES AND LIMITATIONS
Only one kind of limitation is peculiar of the alignment technique, and it is related to some possible distortion of the scanning path inside the SEM. It happened, indeed, in the past that on analogic instruments some deviation from a straight line caused the images to stretch or shrink at the edges of the frame [2]. It is clear that a distortion, i.e. at the right edge, in each original image will result upon rotation as a general image mismatch at all edges. Some correction procedures may then be invoked, based again on image comparison tools, with the possibility of obtaining perfectly matched images but all carrying a common distortion with respect to the real specimen. The digital SEMs display superior control of the raster geometry, and in general pixel accuracy is routinely available. In this case, the procedure will produce a set of images IE, IN, IW, IS that is undistinguishable from what would be acquired by four detectors during a single scan. At this point, all properties and limitations belong to the intrinsic peculiarities of the very general Photometric Stereo technique. In general, the most relevant perturbations to the ideal treatment of Photometric Stereo come from non-lambertian shading and from geometrical shadows. The quoted literature deals with both those subjects, but for the specific case it may be said that: the absence of the electron analogous of specular reflection, the absence of colors and the cited lambertian behaviour of the BS emission make this signal possibly a better candidate than light for PS applications. Anyway, anomalous emissions take place also for the BS signal, mainly on light materials and thin edges, when the enhanced penetration of the electron beam and the concurrent emission of electron from outside the beam impact point create the analogous of a self-luminescent emission. In that case, automated recognition and correction of the anomalous cases may be faced by the same techniques that in optical PS allow to successfully handle the specular reflections [16] the presence of geometrical shadows is a problem exactly as for optical PS. Techniques for recovering the gradient exist for regions that are illuminated by any 3 of the 4 “lights” [16]. In any case, the best objects for PS reconstruction remain the bassrelieves.
when non-ideal conditions apply, as for any real case and as reported in fig. 6, and where a simple surface was numerically modelled in order to introduce controlled nonidealities, that solution introduces unacceptable slicing effects. It is common practice to refer to [17] numerical Poisson methods, or [17] matrix methods (that may be demonstrated to be equivalent) that in any case operate on the xy domain with intrinsically bi-dimensional kernels that assign a dominant role to the first neighbouring pixels around each calculation point. The last image in fig.6 shows as the same level of additive noise that resulted in the poor result of fig.6b, affects the Poisson-integrated surface as a uniformly added roughness.
Fig.6. An ideal noiseless numerical surface, made of a low pyramid on a flat base is perfectly reconstructed (a) by Riemann integration
As additive noise is introduced, the Riemann method (b) results in unacceptable slicing
Poisson methods (c) on the same noisy data recover the original shape, and noise gives some uniform roughness.
A. Influence of the integration method. Three main elements may affect the 3D reconstruction in Photometric Stereo: resolution of the xy sampling, number of gray-levels and noise. Anyway, their influence on the quality of the reconstructed slope strongly depends on the integration method. In the ideal case, even the brutal 1D integration along each pixel row or column would reconstruct the original surface. Nevertheless,
Fig.7 a) A simple surface made of flattened dome was designed and rendered under virtual lighting from the 4 cardinal points, as for experimental Photometric Stereo. Its reconstruction at standard xy resolution (b) returns the expected geometry. (c) The reconstruction from sub-sampled original images results in evident surface faceting.
1281
In other words, the noise in electron intensity transforms into noise on the measurement of z, which calls for suitable time integration during the acquisition process. B. Pixelation Once the distortion of the raster scan is avoided, as said before, the only deformation in the x-y domain may come from sub-sampling. Sub-sampling is sometimes required to shorten the image processing time, mainly during the setup of the instrument, when many shots are taken and displayed to progressively reach the optimum result. In that case, the well known effect of pixelation, the same that appears when highly magnified details of digital images are displayed, will not only affect the x-y domain, but also the reconstructed vertical slope. In PS, indeed, a single outward normal “belongs” to each pixel of the xy domain. After integration, the “large pixel” will transform into an inclined plane facet of a roughly outlined solid (fig.7). It is obvious that the problem is solved by increasing the original image resolution. Anyway, the faceting effect recalls that the z is reconstructed by integrating a discrete differential form, so that the vertical resolution ∆z depends on both the shading and the lateral resolution ∆x. In order to give an estimation, let us consider the ideal noiseless case, where also line integration works. The x component of the gradient, as indicated before, may be expressed as
∆z = β
erroneously increased to β, resulting in a much higher pyramid. Fig. 8 represents the artificial case of the same noiseless surface of fig.7, sampled at high resolution on the xy plane but at different byte resolution. As the number of allowed grey levels reduces, only few inclinations are available for the final surface, which results in some complex faceting, where the projection of each facet onto the xy plane is by no means a square as for the pixelation case, as clearly shown by the differently reconstructed circular edge in fig.7c and in fig. 8b. Increasing the byte resolution leads to progressive reduction of the artefacts, and the standard 8-bit depth for the byte, with its 256 available grey levels, seems more than sufficient to avoid byte-related distortion. This is an important point, because it is possible to avoid to use the full range of the grey levels, and in particular to keep the image contrast far from both the absolute darkness and the total brightness. This prevents from the most part of the non-linear effect that would arise from saturated images, and also takes advantage of the better linearity of the total detection chain in the mid-gain range. In the real cases, the mixed effects of noise, xy sampling and intensity AD conversion mix in a complex way, whose exact evaluation should also include the analysis of the integration algorithm.
I E − IW ∆x I E + IW
where β is an inclination constant of the system corresponding to the tangent of angle between the light direction and the vertical z-axis. β may be considered an exact value, or an adjustable common constant, that does not differ too much from unity. Being the ratio of the intensities obviously limited to the -1,1 range, it results the thumb-rule that the vertical resolution in PS reconstructed 3D surfaces is roughly not worse than the lateral resolution.
Fig.8 Effects of different bytedepths a)
8 bit
b)
4 bit
c)
2 bit
d)
1 bit
C. Byte depth A similar, but not at all identical, effect comes from the byte-depth, that is the number of grey levels used for coding the AD-converted intensity of the signal at each pixel of each BS electron image. The same formula used for evaluating the pixelation effect shows as, at the extreme case of a binary coding, in a bass-relief whose soft shading makes the intensity ratio
I E − IW to never exceed, i.e., I E + IW
the 0.5
absolute value (as for the surfaces used in fig.6 and 7), the reconstructed surface would be an horizontal plane, while for a value just a little higher than 0.5 the slope ∆z/∆x would be
1282
Anyway, the capability of a SEM, different from a photographic camera, to correct possible scan distortions allows aberration-free imaging, at least with respect to the proper distortion aberrations (as barrel and pincushion). Chromatic aberration is intrinsically absent, and the scanning analogous of spherical aberration is made absolutely negligible by the very high focus depth of the SEM, that makes any detail in a bass-relief surface equally sharp. On the other hand, astigmatism may be of major importance if not properly corrected. As a result, the limitation to the 3D reconstruction may be associated to the intrinsic resolution of the SEM. IV.
CONCLUSIONS
[6]
[7] [8]
[9]
[10]
[11]
[12]
The proposed automated alignment procedure has proven accurate enough to produce images suitable for standard Photometric Stereo. The limitations of the method (the bass-relief specimen, the shadow perturbation, the non linearity of the detectors) are the same of the optical version of the same technique, while not all the limitations of optical PS are present in electron PS. Some peculiar features of SEM imaging have been indeed considered, in comparison with the optical counterpart, which indicated the BS SEM imaging as more fit to PS than light itself. The key point of the method is the use of standard commercial SEMs of the latest generation, in absolutely conventional configuration, which makes 3D SEM available at no additional cost. The perspective of merging Photometric Stereo and classical Stereo Imaging at the SEM seems highly stimulating, because of its potential capability to overcome the bass-relief limitation and to produce 3D description also of full-relief specimens, including difficult details as cavities and suspended elements. In any case, all evolution in that direction will have a direct replica in the optical domain.
[13]
[14] [15] [16]
[17] [18]
[19]
[20]
REFERENCES [1]
[2]
[3]
[4]
[5]
R.Pintus, S. Podda, F. Mighela and M. Vanzi “Quantitative 3D Reconstruction From BS Imaging” Microelectronics Reliability 44, 2004 1547-1552. R.Pintus, S. Podda and M. Vanzi “Image alignment for 3D reconstruction in a SEM” Microelectronics Reliability 45, 2005 15811584. R.J. Woodham, “Reflectance map techniques for analyzing surface defects in metal casting”, Massachusetts Institute of Technology, June 1978 R.J. Woodham, Y. Iwahori, and R.A. Barman, “Photometric Stereo: Lambertian Reflectance and Light Sources with Unknown Direction and Strength,” technical report, Dept. of Computing Science, Univ. of British Columbia, 1991. R.J. Woodham, “Photometric Method for Determining Surface Orientation from Multiple Images,” Optical Eng., vol. 19, no. 1, pp. 139-144, 1980.
1283
R.J. Woodham, “Gradient and Curvature from the Photometric-Stereo Method, Including Local Confidence Estimation,” J. Optical Soc. Am., vol. 11, no. 11, pp. 3050-3068, Nov. 1994. R.J. Woodham. Analysing images of curved surfaces. “Artificial Intelligence”, 17:117.140, 1981. Athinodoros S. Georghiades, “Recovering 3-D Shape and Refectance From a Small Number of Photographs”, Eurographics Symposium on Rendering 2003 G. Kay and T. Caelly, “Estimating the Parameters of an Illumination Model Using Photometric Stereo,” Graphical Models and Image Processing, vol. 57, no. 5, pp. 365-388, Sept. 1995. S.K. Nayar, K. Ikeuchi, and T. Kanade, “Determining Shape and Reflectance of Hybrid Surfaces by Photometric Sampling,” IEEE Trans. Robotics and Automation, vol. 6, no. 4, pp. 418-431, Aug. 1990. H.D. Tagare and R.J.P. deFigueiredo, “A Theory of Photometric Stereo for a Class of Diffuse Non-Lambertian Surfaces,” IEEE Trans. Pattern Analysis Machine Intelligence, vol. 13, no. 2, pp. 133- 152, Feb. 1991. E.N. Coleman and R. Jain, “Obtaining 3-Dimensional Shape of Textured and Specular Surfaces Using Four-Source Photometry,” Computer Graphics and Image Processing, vol. 18, pp. 309-328, 1982. M. Drew and M. Brill, “Color from Shape from Color: A Simple Formalism with Known Light Sources,” J. Optical Am. Soc. A, vol. 17, no. 8, pp. 1371-1381, Aug. 2000. M.S. Drew, “Shape from Color,” Technical Report CSS/LCCR TR 9207, Simon Fraser School of Computer Science, 1992. G.J. Klinker, A Physical Approach to Color Image Understanding. A.K. Peters, 1993. Barsky and Petrou, “The 4-Source Photometric Stereo Technique for Three-Dimensional Surfaces in the Presence of Highlights and Shadows”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, NO. 10 October 2003, pp. 1239-1252 William H. Press, Numerical Recipes in C : The Art of Scientific Computing, Cambridge University Press Yuji Iwahori, Haruki Kawanaka, Shinji Fukui, and Kenji Funahashi, “Obtaining Shape from Scanning Electron Microscope Using Hopfield Neural Network”, KES 2004, LNAI 3214, pp.632-639 L. Reimer, R. Bongeler, V. Desai, “Shape from shading using multiple detector signals in Sanning Electron Microscopy”, Scanning Microscopy, Vol. 1, No. 3, 1987, pp 963-973 W. Beil and I.C. Carlsen, “Surface Reconstruction from Stereoscopy and Shape from shading”, Machine Vision and Applications, (1991) 4: 271-285
An Automatic Alignment Procedure for a 4-Source Photometric Stereo Technique applied to Scanning Electron Microscopy Ruggero Pintus, Simona Podda and Massimo Vanzi Department of Electrical and Electronic Engineering, University of Cagliari Piazza d’Armi, 09123 Cagliari (CA), Italy Phone: +39-070-675-5859, Fax: +39-070-675-5900, Email: [email protected] Abstract – This paper presents an automatic alignment procedure for a 4-Source Photometric Stereo technique to reconstruct the depth map in the Scanning Electron Microscopy. PS, based on the so-called reflectance map, used several images of a surface to estimate the surface depth at each image point. Lambertian reflectivity function is the simplest one. In the SEM one of the most important signals, the Backscattered Electron emission, is nearly Lambertian, and, to simplify matters, SEM images are intrinsically greyscale maps. The possibility for electron-PS is assumed, taking advantage of one of the most exciting features of the technique, which doesn't return some depth illusion from ordinary pictures, but true numerical 3D models. Keywords – Photometric Stereo, Scanning Electron Microscopy, image alignment.
I. INTRODUCTION In a previous paper [1] a 4-Source Photometric Stereo technique has been proposed to reconstruct the third dimension in the Scanning Electron Microscopy (SEM). The original Photometric Stereo (PS) was conceived in the optical domain by Woodham [3-6] some 30 years ago and used several images of a surface, taken from the same viewpoint but under different illumination directions, to estimate the relative surface depth at each image point. So far many papers improved that technique using different approaches [7-15], all based on a suitable choice of the so-called reflectance map, the function that describes how incident light is diffused by a surface. Among those maps, the Lambertian reflectivity function is the simplest one because it allows to formulate the problem in a linear and then a matrix form [16]. In the optical field, the Lambertian case is often an over-simplified view of reality, because specular reflections and chromatic differences call for more complex models. On the contrary, in the SEM one of the most important signals, the Back Scattered Electron emission (BSE), is nearly ideally Lambertian [8]. To further simplify matters, only the intensity of the electron emission is measured, and not its energy, so SEM images are indeed intrinsically greyscale maps. The possibility for electron-PS at the SEM is then assumed, opening a way for some unexplored access to the microscopic world, taking advantage of one of the most exciting features of the technique, which doesn’t return some depth illusion from ordinary pictures, but true numerical 3D models. Giving a closer look to the PS-based 3D recovery method, it consists of the acquisition of 4 Back-Scattered (BS) images,
0-7803-9360-0/06/$20.00 ©2006 IEEE
from which the gradient of the surface and its “intrinsic brilliance” (in electron microscopy, the emission coefficient, that makes gold brighter than carbon also at the SEM) are first separately extracted by means of some image algebra, and then the depth map is recovered by integration of the gradient vector field. Some attempts have been made in the past [18-19] in that direction, but all limited by the choice of having as many BSE detectors simultaneously active as the number of required different light sources for proper PS in the optical domain. Now, this is by far not a standard case and makes the microscope a dedicated instrument, the exact contrary of the most appreciated feature of a SEM, i.e. its extremely wide application range without other devices but the standard equipment. As a result, the most popular current method for 3D electron microscopy still remains the stereoscopy [20], although it is only able to give the binocular perception of the depth. However, most of the benefits of PS-based 3D-SEM may be reached introducing a sequential acquisition for the required 4 BS images. The fee to be paid is to renounce dynamic 3D (which would obviously require simultaneous images to reconstruct a time evolving 3D frame), and to introduce some mechanical movement inside the SEM. It is necessary, indeed, either to move the detector (not at all a recommended choice) or to rotate the specimen under the fixed detector: a referential, solid with the specimen, would display the relative motion as a motion of the light source. In details, the solution consists in sequentially acquiring 4 images of the same specimen upon the imposition onto the specimen of four 90° rotation steps, and then rotating back the images: a sequence of 4 pictures of the same area, each illuminated from a different cardinal point will be obtained. Exact alignment would then be required, in order to have each point on the specimen imaged on the same pixel in each and all the four images. This is a crucial point, because pixel precision is not achieved without image comparison, and this task is nearly impossible when details are differently shaded in each picture. Another recent paper [2] built the foundations of this work, based on the capability of the SEM to use several different signals to produce images, under the same scanning conditions. At least two signals in the SEM, different from the BSE, collected by a standard off-axis detector, display direction-independent contrast, i.e. isotropic shading: the axially-detected BSE, and the Specimen Current (SC). In both cases, the respective detectors are normally
1278
available for any electron microscope, and often come as standard devices in the basic instrument configuration. The advantage of using isotropic shaded images is that their alignment is not complicated by different shading, which may reverse the contrast of a same detail under opposite lighting. The alignment transformations found for the uniformly shaded set are then used to properly align the four BSE images used for 3D reconstruction. In any case, a twin set of images is acquired: one for alignment purposes and the other for 3D reconstruction. The proposed method bases itself on the complete computer control currently available in any SEM, and develops a completely automated tool that enables a standard instrument to access the 3D domain without any dedicated optional tool. The aim of this paper is to demonstrate the pixel precision achieved by the proposed mixed mechanicalnumerical procedure for image alignment. Nevertheless, some comments will also be given on the expected performances of the method, in terms of noise and accuracy. II. THE AUTOMATED SEQUENCE In the electron microscope some specific actions, among the many available, are put into sequence. A. The initial setup
The directional shading is evident in the first image where “lighting” seems to come from the upper side (and simply corresponds to the direction where the off-axis detector is placed), while the second image has contrast variations (the shell is brighter than the surrounding mould) but there is no directionality in the light. Where available, all suitable mechanical calibrations are also performed. For some machines, combined mechanical rotation and xy shift may be set in order to keep the centre of the image at the central point of the screen, to emulate the rigid connection of the optical axis with the mechanical rotation axis. Anyway, as sub-micrometric resolution is reached (magnification larger than 1000x), the mechanical precision becomes considerably insufficient. B. The automated sequence A mixed use of the electronic and mechanical controls is then programmed, as summarized in the following automatic sequence: 1) Acquisition of a pair BS and IS images, as in fig.1. 2) mechanical rotation of the specimen by 90° 3) Acquisition of a reduced IS image in the central area, that is first counter-rotated by 90°, and then
First of all the subject is framed, magnified and focused as for normal optimum SEM imaging. Also brightness and contrast are separately set for the two detectors to be used, i.e. the off-axis BackScattered (BS) electron detector and the available one able to produce isotropic shading, that may be the Specimen Current detector or the complete circular axial BS detector. The latter will be indicated as the detector for Isotropic Shade (IS). Fig.1 reports the BS and IS images that simultaneously form, since both detectors are enabled during the same frame scan.
4) cross-correlated with the previous IS image. This gives the ∆x,∆y shift to be compensated.
Fig.1 BS (left) and IS (right) images of the same frame scan.
frame
rotation
shift
5) The new image centre of the rotated specimen is mechanically brought back by -∆x,-∆y as close as possible to the previous centre. 6) Steps 1-4 are repeated 3 times. At the end of the process (which takes few minutes, including the image acquisition), a twin set of 4+4 images is stored.
1279
BS
IS
Fig.2 A couple of views of the 3D surface from fig.1.
The image processing then automatically proceeds as: 1) Rotation and fine alignment of the IS set: computation of the residual ∆x,∆y with respect to the first IS picture. 2) Rotation and alignment of the BS set by imposing the corresponding -∆x,-∆y numerical shifts. 3) Selection of the maximum common area in the BS image set and cutting of the exceeding stripes. 4) Storage of the resulting four IE, IN, IW, IS BS images for PS computation, separately representing the specimen under the same point of view and orientation, but lighted from the cardinal points.
Fig.3. A set of BS images (upper line), the corresponding IS twin set (central line) and some 3D views (lower line) of the reconstructed surface of a damaged solid-state electron device.
The overall automated sequence lasts some few minutes, depending on the selected acquisition time for the images to be stored, and pixel accuracy is currently achieved.
Fig.4 One of the BS SEM images of the detail of a coin and its 3D reconstruction.
C. 3D processing Since this point, the procedure is the same as for optical Photometric Stereo, it may be demonstrated [16] that the simple image algebra
p = q =
I E − IW I E + IW IN − IS IN + IS
empowers p,q with the role of x,y derivatives of the surface z(x,y), and then the gradient of that surface is available at each pixel of the original image domain. It is a matter of numerical integration to retrieve the depth from the gradient. The operator is now enabled to see the reconstructed surface (fig.2, 3,4 and 5), to rotate it in the virtual space, to render it by false texturing or applying the original SEM contrast, and finally to save the file, including the original pictures and the reconstructed surface. A possibility is included to generate the suitable files for direct 3D printing of the virtual solid by fast-prototyping devices.
Fig.5.Conventional optical image (upper left) and SEM detail (right) of the percussion surface of an exploded bullet-case. The lower image reveals the 3D shape.
1280
III.
PERFORMANCES AND LIMITATIONS
Only one kind of limitation is peculiar of the alignment technique, and it is related to some possible distortion of the scanning path inside the SEM. It happened, indeed, in the past that on analogic instruments some deviation from a straight line caused the images to stretch or shrink at the edges of the frame [2]. It is clear that a distortion, i.e. at the right edge, in each original image will result upon rotation as a general image mismatch at all edges. Some correction procedures may then be invoked, based again on image comparison tools, with the possibility of obtaining perfectly matched images but all carrying a common distortion with respect to the real specimen. The digital SEMs display superior control of the raster geometry, and in general pixel accuracy is routinely available. In this case, the procedure will produce a set of images IE, IN, IW, IS that is undistinguishable from what would be acquired by four detectors during a single scan. At this point, all properties and limitations belong to the intrinsic peculiarities of the very general Photometric Stereo technique. In general, the most relevant perturbations to the ideal treatment of Photometric Stereo come from non-lambertian shading and from geometrical shadows. The quoted literature deals with both those subjects, but for the specific case it may be said that: the absence of the electron analogous of specular reflection, the absence of colors and the cited lambertian behaviour of the BS emission make this signal possibly a better candidate than light for PS applications. Anyway, anomalous emissions take place also for the BS signal, mainly on light materials and thin edges, when the enhanced penetration of the electron beam and the concurrent emission of electron from outside the beam impact point create the analogous of a self-luminescent emission. In that case, automated recognition and correction of the anomalous cases may be faced by the same techniques that in optical PS allow to successfully handle the specular reflections [16] the presence of geometrical shadows is a problem exactly as for optical PS. Techniques for recovering the gradient exist for regions that are illuminated by any 3 of the 4 “lights” [16]. In any case, the best objects for PS reconstruction remain the bassrelieves.
when non-ideal conditions apply, as for any real case and as reported in fig. 6, and where a simple surface was numerically modelled in order to introduce controlled nonidealities, that solution introduces unacceptable slicing effects. It is common practice to refer to [17] numerical Poisson methods, or [17] matrix methods (that may be demonstrated to be equivalent) that in any case operate on the xy domain with intrinsically bi-dimensional kernels that assign a dominant role to the first neighbouring pixels around each calculation point. The last image in fig.6 shows as the same level of additive noise that resulted in the poor result of fig.6b, affects the Poisson-integrated surface as a uniformly added roughness.
Fig.6. An ideal noiseless numerical surface, made of a low pyramid on a flat base is perfectly reconstructed (a) by Riemann integration
As additive noise is introduced, the Riemann method (b) results in unacceptable slicing
Poisson methods (c) on the same noisy data recover the original shape, and noise gives some uniform roughness.
A. Influence of the integration method. Three main elements may affect the 3D reconstruction in Photometric Stereo: resolution of the xy sampling, number of gray-levels and noise. Anyway, their influence on the quality of the reconstructed slope strongly depends on the integration method. In the ideal case, even the brutal 1D integration along each pixel row or column would reconstruct the original surface. Nevertheless,
Fig.7 a) A simple surface made of flattened dome was designed and rendered under virtual lighting from the 4 cardinal points, as for experimental Photometric Stereo. Its reconstruction at standard xy resolution (b) returns the expected geometry. (c) The reconstruction from sub-sampled original images results in evident surface faceting.
1281
In other words, the noise in electron intensity transforms into noise on the measurement of z, which calls for suitable time integration during the acquisition process. B. Pixelation Once the distortion of the raster scan is avoided, as said before, the only deformation in the x-y domain may come from sub-sampling. Sub-sampling is sometimes required to shorten the image processing time, mainly during the setup of the instrument, when many shots are taken and displayed to progressively reach the optimum result. In that case, the well known effect of pixelation, the same that appears when highly magnified details of digital images are displayed, will not only affect the x-y domain, but also the reconstructed vertical slope. In PS, indeed, a single outward normal “belongs” to each pixel of the xy domain. After integration, the “large pixel” will transform into an inclined plane facet of a roughly outlined solid (fig.7). It is obvious that the problem is solved by increasing the original image resolution. Anyway, the faceting effect recalls that the z is reconstructed by integrating a discrete differential form, so that the vertical resolution ∆z depends on both the shading and the lateral resolution ∆x. In order to give an estimation, let us consider the ideal noiseless case, where also line integration works. The x component of the gradient, as indicated before, may be expressed as
∆z = β
erroneously increased to β, resulting in a much higher pyramid. Fig. 8 represents the artificial case of the same noiseless surface of fig.7, sampled at high resolution on the xy plane but at different byte resolution. As the number of allowed grey levels reduces, only few inclinations are available for the final surface, which results in some complex faceting, where the projection of each facet onto the xy plane is by no means a square as for the pixelation case, as clearly shown by the differently reconstructed circular edge in fig.7c and in fig. 8b. Increasing the byte resolution leads to progressive reduction of the artefacts, and the standard 8-bit depth for the byte, with its 256 available grey levels, seems more than sufficient to avoid byte-related distortion. This is an important point, because it is possible to avoid to use the full range of the grey levels, and in particular to keep the image contrast far from both the absolute darkness and the total brightness. This prevents from the most part of the non-linear effect that would arise from saturated images, and also takes advantage of the better linearity of the total detection chain in the mid-gain range. In the real cases, the mixed effects of noise, xy sampling and intensity AD conversion mix in a complex way, whose exact evaluation should also include the analysis of the integration algorithm.
I E − IW ∆x I E + IW
where β is an inclination constant of the system corresponding to the tangent of angle between the light direction and the vertical z-axis. β may be considered an exact value, or an adjustable common constant, that does not differ too much from unity. Being the ratio of the intensities obviously limited to the -1,1 range, it results the thumb-rule that the vertical resolution in PS reconstructed 3D surfaces is roughly not worse than the lateral resolution.
Fig.8 Effects of different bytedepths a)
8 bit
b)
4 bit
c)
2 bit
d)
1 bit
C. Byte depth A similar, but not at all identical, effect comes from the byte-depth, that is the number of grey levels used for coding the AD-converted intensity of the signal at each pixel of each BS electron image. The same formula used for evaluating the pixelation effect shows as, at the extreme case of a binary coding, in a bass-relief whose soft shading makes the intensity ratio
I E − IW to never exceed, i.e., I E + IW
the 0.5
absolute value (as for the surfaces used in fig.6 and 7), the reconstructed surface would be an horizontal plane, while for a value just a little higher than 0.5 the slope ∆z/∆x would be
1282
Anyway, the capability of a SEM, different from a photographic camera, to correct possible scan distortions allows aberration-free imaging, at least with respect to the proper distortion aberrations (as barrel and pincushion). Chromatic aberration is intrinsically absent, and the scanning analogous of spherical aberration is made absolutely negligible by the very high focus depth of the SEM, that makes any detail in a bass-relief surface equally sharp. On the other hand, astigmatism may be of major importance if not properly corrected. As a result, the limitation to the 3D reconstruction may be associated to the intrinsic resolution of the SEM. IV.
CONCLUSIONS
[6]
[7] [8]
[9]
[10]
[11]
[12]
The proposed automated alignment procedure has proven accurate enough to produce images suitable for standard Photometric Stereo. The limitations of the method (the bass-relief specimen, the shadow perturbation, the non linearity of the detectors) are the same of the optical version of the same technique, while not all the limitations of optical PS are present in electron PS. Some peculiar features of SEM imaging have been indeed considered, in comparison with the optical counterpart, which indicated the BS SEM imaging as more fit to PS than light itself. The key point of the method is the use of standard commercial SEMs of the latest generation, in absolutely conventional configuration, which makes 3D SEM available at no additional cost. The perspective of merging Photometric Stereo and classical Stereo Imaging at the SEM seems highly stimulating, because of its potential capability to overcome the bass-relief limitation and to produce 3D description also of full-relief specimens, including difficult details as cavities and suspended elements. In any case, all evolution in that direction will have a direct replica in the optical domain.
[13]
[14] [15] [16]
[17] [18]
[19]
[20]
REFERENCES [1]
[2]
[3]
[4]
[5]
R.Pintus, S. Podda, F. Mighela and M. Vanzi “Quantitative 3D Reconstruction From BS Imaging” Microelectronics Reliability 44, 2004 1547-1552. R.Pintus, S. Podda and M. Vanzi “Image alignment for 3D reconstruction in a SEM” Microelectronics Reliability 45, 2005 15811584. R.J. Woodham, “Reflectance map techniques for analyzing surface defects in metal casting”, Massachusetts Institute of Technology, June 1978 R.J. Woodham, Y. Iwahori, and R.A. Barman, “Photometric Stereo: Lambertian Reflectance and Light Sources with Unknown Direction and Strength,” technical report, Dept. of Computing Science, Univ. of British Columbia, 1991. R.J. Woodham, “Photometric Method for Determining Surface Orientation from Multiple Images,” Optical Eng., vol. 19, no. 1, pp. 139-144, 1980.
1283
R.J. Woodham, “Gradient and Curvature from the Photometric-Stereo Method, Including Local Confidence Estimation,” J. Optical Soc. Am., vol. 11, no. 11, pp. 3050-3068, Nov. 1994. R.J. Woodham. Analysing images of curved surfaces. “Artificial Intelligence”, 17:117.140, 1981. Athinodoros S. Georghiades, “Recovering 3-D Shape and Refectance From a Small Number of Photographs”, Eurographics Symposium on Rendering 2003 G. Kay and T. Caelly, “Estimating the Parameters of an Illumination Model Using Photometric Stereo,” Graphical Models and Image Processing, vol. 57, no. 5, pp. 365-388, Sept. 1995. S.K. Nayar, K. Ikeuchi, and T. Kanade, “Determining Shape and Reflectance of Hybrid Surfaces by Photometric Sampling,” IEEE Trans. Robotics and Automation, vol. 6, no. 4, pp. 418-431, Aug. 1990. H.D. Tagare and R.J.P. deFigueiredo, “A Theory of Photometric Stereo for a Class of Diffuse Non-Lambertian Surfaces,” IEEE Trans. Pattern Analysis Machine Intelligence, vol. 13, no. 2, pp. 133- 152, Feb. 1991. E.N. Coleman and R. Jain, “Obtaining 3-Dimensional Shape of Textured and Specular Surfaces Using Four-Source Photometry,” Computer Graphics and Image Processing, vol. 18, pp. 309-328, 1982. M. Drew and M. Brill, “Color from Shape from Color: A Simple Formalism with Known Light Sources,” J. Optical Am. Soc. A, vol. 17, no. 8, pp. 1371-1381, Aug. 2000. M.S. Drew, “Shape from Color,” Technical Report CSS/LCCR TR 9207, Simon Fraser School of Computer Science, 1992. G.J. Klinker, A Physical Approach to Color Image Understanding. A.K. Peters, 1993. Barsky and Petrou, “The 4-Source Photometric Stereo Technique for Three-Dimensional Surfaces in the Presence of Highlights and Shadows”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, NO. 10 October 2003, pp. 1239-1252 William H. Press, Numerical Recipes in C : The Art of Scientific Computing, Cambridge University Press Yuji Iwahori, Haruki Kawanaka, Shinji Fukui, and Kenji Funahashi, “Obtaining Shape from Scanning Electron Microscope Using Hopfield Neural Network”, KES 2004, LNAI 3214, pp.632-639 L. Reimer, R. Bongeler, V. Desai, “Shape from shading using multiple detector signals in Sanning Electron Microscopy”, Scanning Microscopy, Vol. 1, No. 3, 1987, pp 963-973 W. Beil and I.C. Carlsen, “Surface Reconstruction from Stereoscopy and Shape from shading”, Machine Vision and Applications, (1991) 4: 271-285
