Optimal Halftoning for Tactile Imaging - Semantic Scholar
1
Optimal Halftoning for Tactile Imaging Amit Nayak, Student Member, IEEE and Kenneth E. Barner, Senior Member, IEEE Department of Electrical and Computer Engineering University of Delaware, Newark, Delaware 19716, USA [email protected], [email protected] Abstract Reading of text and understanding images by touch is an important alternative and additional source of information when sight is absent or lost. Tactile graphics and models such as edge maps, binary output etc. are the solution for simple access to images for blind persons. This paper introduces an approach to model the human tactile system based on the responses produced by stimuli on micro-capsule paper. This system is utilized for the purpose of generating optimum halftone patterns on micro-capsule paper that can be utilized for the effective generation of tactile graphics.
I. I NTRODUCTION Vision plays a pivotal role in information gathering for human beings. For instance, much of the sciences involve the visualization of materials and concepts. Valuable information in text and images is thus gathered by humans through the visual sense. This reliance on vision is hampered, or lost, in individuals with visual impairments. In such cases, it is necessary for the information display to include alternative modalities. The main system of written communication utilized by blind people throughout the world is the Braille system. Invented by Louise Braille in the early nineteenth century, this system proved to be better than the earlier, difficult and cumbersome, method of embossed letter reading. The use of Braille method for reading and writing, has been researched at depth for its utility by various researchers [1]. This method, however, allows tactile access to only text based information. Tactile imaging is the process of converting a visual image into a touchable binary raised version. Applications of tactile graphics have been researched on in several places. Tactile pattern generation received a boost with the invention of Optacon and the subsequent modification [2],
2
[3]. Loomis [4] and Craig [5] initiated a discussion on the tactile pattern perception and the factors which could affect this perception. One of the earliest groups to look in to the sensory aspects of skin in the human finger, Johnson et al. [6], [7], [8] worked on the the discrimination and detection of points, gaps, gratings and letters by the human finger. Martin et al. [9] have worked in the presentation of business graphics for blind people. Pie charts, bar charts and other form of representation of information in image format have been successfully converted into relevant form. The data on which the graphical information is based is extracted from the image and then the tabulated version of this data is conveyed to the user either by speech, sound and/or tactile form. Speech-synthesizer, using acoustically different frequency tones are some ways and in the case of bar charts the most favored way is using a 2D-Braille-Display, wherein raised dots are used to depict the length of the actual bar in the y-axis and the x-axis is coupled as a Braille output. Klatzky et al. [10], in 1985, discussed the identification of objects by touch and have subsequently compared the amount of information garnered by the visual and the haptic interface [11] in 1987. Similarly the same time investigators had attempted to quantify the information channel capacity of the human sensory organs and Kokjer [12] suggested an order-of-magnitude upper limit on this property for the vibrotactile stimulation of human fingertip as 100 bit/s. This, when compared with corresponding limits in the human ear and eye, a progression of
10 :10 :10 2
4
6
re-
spectively, was found. Thus the capacity of visual sense to receive and perceive information is the highest, as compared to hearing and touch. Of the various studies and research done in the field of generation of tactile graphics, of particular interest are that of [13], [14], which successfully developed software algorithms for automatic generation of tactile graphics. Meaningful information is identified and extracted from original image data incorporating a multi-step procedure. These include edge and boundary information extraction from the original image and representing them in tactile format. Edge and boundary information refers to identification of regions in an image where a rapid change of intensity takes place. Also important in the procedures are the successful segmentation of the original image. These image processing methods were further developed by Hernandez et al. [15], [16], [17] who
3
used the edge information to generate graphics that rendered image boundary in tactile form. Edge information on its own is, however, not enough to present the complete effect. A tactile image should contain different textures [18] to help distinguish and discriminate between various shapes and colors present in an image. We, therefore, propose optimal tactile imaging textures. The underlying idea is that the various regions in segmented images can now be easily identified by touch if they are given a separate textures. To accomplish this, we use digital halftoning methods to create patterns for the images which can then be suitably enhanced to create the feel of various textures. Digital Halftoning is the process of generating a binary pixel patterns that create the illusion of a continuous-tone image. This technique is important for display of gray-scale images in printing process, where the direct rendition of the gray tones is not possible. In recent years a number of comprehensive methods have been designed to carry out the halftoning process. The primary methods are reviewed in [19], [20]. Optimum algorithms [21], [22] were found to be those that exploit the dot-printer models and take into account the low pass nature of the human vision. We extend this idea of model-based halftoning to the domain of the human tactile system. As used extensively in the human visual system (HVS) modeled halftoning system, a transfer function for the sensory system is required along with an understanding of the printing process. In an analogous fashion, we develop a model for the human tactile system and the tactile printing process. In the case considered, we utilize the Tactile Image Enhancer, Fig. 1, operating on microcapsule paper [13], [14], [23] as the tactile output. The output characteristics of the TIE were examined. We develop a model for the surface generated from the TIE. We also devise a model of the human tactile perception based on the spatial frequency response of the human tactile sense. These models are integrated into the process of generating halftones best suited for a given gray scale. The remainder of this paper is organized as follows. The working of the TIE and the measurements of the output characteristics and resulting enhancement model are presented in Section 2. In Section 3, we discuss the work carried out earlier on the characterization and specification of the human finger pad. New experiments are carried out to find how sensitive the human finger is to
4
Fig. 1. Tactile Image Enhancer (TIE is manufactured by Repro-tronics Inc.).
the surface of the tactile paper, the results of which are reported in Section 4. Optimal halftoning methods are discussed in Section 5. Finally, the conclusions and further research are presented in Section 6. II. I MAGE E NHANCEMENT P ROCESS Tactile images can be formed by enhancing, or raising, the surface of a special paper called tactile or micro-capsule paper through use of the TIE, Fig. 1. The TIE is manufactured by the Repro-tronics Inc. The first prototype was introduced in the market in 1993 and have been in use since then. As of now there is a smaller, more portable version of TIE available in the market. The TIE works on a very simple principle. It has a motor-driven roller that passes the tactile paper, face up, underneath a tubular light bulb, Fig. 2. The light bulb acts as a heat source and the rollers help in guiding the paper through the direct contact of the bulb. The type of tactile paper which we intend to focus on is manufactured by the Matsumoto Kosan Company Ltd. in Japan. This is also known as the micro-capsule paper. As the name suggests it is a white paper coated with microscopic polystyrene capsules. The Micro-capsule coating on the paper is heat reactive. Black lines or images printed on the paper, when exposed to a heat source, absorb more heat than the surrounding areas. This causes the underlying capsules to grow. The result is the generation of raised lines, areas, and symbols, i.e., tactile graphics. A temperature range of 120-125 C (248-257 F) is appropriate for the heating of the papers. In the present study
5
Polystyrene Microcapsules
Polyethylene medium Image (printed toner particles)
(a) Heating element
Paper path
Transport rollers
(b) Expanded Capsules
(c) Fig. 2. (a) Micro-capsule paper with image toner particles photocopied on it.(b) Inside the TIE.(c) Raised microcapsules after the paper is passed through the TIE
we have preferred to use 124 C(255.2 F). In order to understand and model the changes occurring in the enhancement process, a simple experiment was carried out. An array of test samples of dots were used. The dot were square pixels printed on a white paper at a resolution of 100 dots per inch (dpi). The size of the dots was uniformly increased from a 1 1 to a 20 20 dot cluster. The resulting outputs were examined for their change in dimension. Images were prepared, comprising of black square dots on a white background and black dots with square gaps simulating white square dots on black background, Fig. 3. These images were photocopied onto the tactile paper. The tactile paper was then passed through the TIE.
6
(a)
(b)
Fig. 3. (a) Black Dots on white background. (b) White dots on black background.
The structures such generated through this test have profiles that range between -0.5 and 0.5 mm in height. A Scanning Electron Microscope(SEM) was used to view the vertical growth of the samples. The profiles of the samples were captured and recorded. The profile of one of the black dot on white background, with a dot size of 8 8, is shown in Fig 4. Simple image processing techniques were used to extract the edge information and these were superimposed to see the growth of the dot clusters as a function of cluster size. Fig. 5 shows one-dimensional profiles for squares of sizes 1 1, 2 2,... etc which are printed at a resolution of 100 dpi. The inverse case, which we can observe as white dots in black background, follow a complementary pattern. The growth in both directions were recorded. The average height or depth of the dot profiles were found to follow the curves shown in Fig. 6. The clusters of dots are characterized by the way the height/depth and width of the dots behave after passing through the TIE. The curves of the profiles are modeled as follows:
Fu = Au + Buln(i)
(1)
Fd = Ad (e, i,Bd 2 =Cd , 1)
(2)
(
)
Fu and Fd denote the height and depth parameters respectively, and the constants were determined experimentally to be Au =0.026, Bu = 0:16 and Ad = 0:5, Bd = 0:75, Cd = 1:596 and i denotes the size of the cluster of dots. Thus, the height and depth parameter of the dot clusters where
7
Fig. 4. The Test pattern corresponding to the 8 x 8 pattern as seen through the SEM. 0.025
Height (inches)
0.02
0.015
0.01
0.005
0 −0.08 −0.06 −0.04 −0.02 0 0.02 0.04 Width of clusters of dots (inches)
0.06
(a) 0.005
Depth (inches)
0 −0.005 −0.01 −0.015 −0.02 −0.025
−0.04
−0.02 0 0.02 Width of clusters of dots (inches)
0.04
(b) Fig. 5. Edge variation for the (a) black dots on a white background, (b) white dots on a black background.
have been modeled either as exponential or logarithmic curves.
8
0.5
Height of dots (mm)
0.4
0.3
0.2
0.1
Experimental data Logarithmic curve fit 0 0
1
2 3 Width of dots (mm)
4
5
(a) Curve fit Experimental data
0
Depth of dots (mm)
−0.1 −0.2 −0.3 −0.4 −0.5
1
2
3 4 Width of dots (mm)
5
(b) Fig. 6. (a) Height variation of black dots on white background. (b) Depth variation of white dots on black background.
Consider next the profiles of the dot clusters. On a simple suitable observation, we find that the profile of a single pixel is approximated by a Gaussian surface as shown in the Fig. 7:
G ; = Ae,x2 =B
(3)
11
where the constants were determined experimentally to be
A = 0:00134, B = 0:02752 and x
represents the distance along the width. Larger clusters are modeled as linear combinations of dots. Thus a n n cluster is modeled through an amplitude adjusted convolution model:
Fn;n = n;n(G ; In;n) 11
(4)
9 −4
Height (inches)
15
x 10
10
5
0 −0.015
−0.01
−0.005 0 0.005 Width of dot (inches)
0.01
0.015
Fig. 7. Profile and the model for a unit impulse comprising of dot of size 1x1. −3
Height (inches)
15
x 10
Experimental data Model fit
10
5
0 −0.06
−0.04
−0.02 0 0.02 Width of dot−cluster (inches)
0.04
0.06
Fig. 8. Model fitted curves for the black dots on white background
where Fn;n is the model suitable for an enhanced profile of an n n cluster of dots and n;n is the average attainable height/depth of the profiles for an n n cluster dot, as shown in Fig. 5(a) and
Fig. 5(b). In;n is the collection of impulses representing a cluster of dot of size n n. These have been plotted in Fig. 6. The resulting modeled curves are found to follow the profiles as predicted. The modeled curves superimposed with the measured curves are seen in Fig. 8. This kind of modeling can also be achieved for the white dots on black background.
10
III. TACTILE
RESOLUTION
Images that are enhanced using the micro-capsule paper processed by the TIE are ready to be used as tactile graphics. It is necessary, however, to understand how human subjects respond to the structures on the micro-capsule paper. A group of experiments were carried out to arrive at a mathematical model for the human tactile system and its response to the structures on the micro-capsule paper. The spatial resolution of the dots and the ability to produce a dot profile of substantial distinction as well as the frequency response of the tactile system is investigated in the experiments. A. Review of previous work The discussion on understanding the tactile pattern perception and the factors affecting this perception was started by Loomis [4] and Craig [5]. Johnson et al.
[6], [7], [8] began the
important exploration of the sensory aspects of the human skin. More recently, investigation was carried out by Pawluk et al. [24], [25] to develop an engineering model describing the response of the human peripheral tactile system when an object(a flat indentor) dynamically contacts the finger-pad. Here the system was measured at both mechanically, at the surface of the skin, and neurophysiologically, as the resulting nerve signal heads towards the brain. The investigation made into the spatial neural mechanisms underlying the tactile sensation by Johnson et al. [6], [7], [8] is divided into three parts. In the first part, specifications of the discrimination behavior that depends strictly on spatial neural mechanism is discussed. Experiments are carried out in this regard to define which aspects of the spatial discrimination are based on spatial information in the neural discharge patterns. It was found that human subjects could reliably discriminate between one and two 0.5 mm (0.019 in.) diameter points even when there was no separation between the two points. A high level of spatial resolution was thus demonstrated. Another set of experiments were carried out in which the subjects were required to discriminate between stimuli with and without gaps. A success criteria of 75% was used and it was found on this basis that gap size at the threshold of edge detection was 0.87 mm(0.034 in.). Subjects were also
11
required to feel square-wave gratings that were oriented along or across the axis of the finger. It was found that grating orientation was difficult to decipher until the gaps exceeded 0.5 mm(0.019 in.). The gap size, which confirmed with the success criteria of
75% accuracy, was found to be
0.84 mm(0.033 in.). In the second part of the experiments a similar study was carried out to find the neural responses in the monkey finger pad skin. In the third series of experiments a mechanistic model of skin is developed to predict the stress and strain at mechanoreceptor terminals within the skin. A parallel approach made by Lamotte and Whitehouse [26], discusses the capabilities of humans to detect the presence of a single raised dot of 550m(0.021 in.) diameter on a smooth plate and to judge the magnitude of evoked sensation. The corresponding stimuli on three different kind of mechanoreceptive peripheral nerve fibers of the monkey finger-pad (namely the slowly adapting (SA), the rapidly adapting(RA)and the pacinian (PC) mechanoreceptive peripheral nerve fibers ) was studied for dots of different heights. The mean detection threshold was found to be about 2.10.3 m (9.410,5 in). Also it has
been found that a height of 1 m (3.937 10,5 in ) could be detected if the diameter of dot was 600 m (0.0236 in) or wider. In similar fashion a height of 6 m (2.36
10,
4
in) was required
for detection if the dot diameter was reduced to 40 m (1.5748 10,3 in). In another experiment, stroke velocities of about 10 mm/s (0.393 in/s) were used and the dot detection thresholds were found out to be between 1(3.937 10,5 in) and 3 m (1.181 10,4 in) for all human observers. This again was different for different mechanoreceptive nerve fibers. It was reported that the magnitude of sensation evoked in humans increased with an increase in dot height above the dot detection threshold as defined in the preceding discussion. It was found that the impulses evoked in monkey RAs increased with dot height as did the widths of RA receptive fields. Also, it was hypothesized that the mechanical event responsible for the activating the RA was the lateral deformation of elevated regions of skin. This was found to be true because the number of impulses evoked in RAs by a dot was greater when the dot was stroked across, as opposed to along, the papillary ridges. It is conclusively proved that the responses of RAs alone
12
account for the sensory capacity to detect dot of minimal height on a smooth surface with the finger pad. In all these experiments described above the object to be felt by the finger-pad is an ideal system with flat surfaces and definite structures. The enhanced micro-capsule paper does not suit an ideal system features and hence the results cannot be directly used in its case. B. Discrimination of dots This experiment simulated the work carried out by and Whitehouse [26] on the micro-capsule paper. Test patterns of individual dots of sizes ranging between a single dot and a cluster of 20 20 dots were created as shown in Fig. 3. The human subjects were required to feel the patterns and respond on the detection of the presence or absence of any structures. Sufficient time was provided to each subject to feel the structures and then define the response. The micro-capsule paper and the enhanced structures on it were shielded from the subject by blindfolds. The tactile paper sample containing these test material were made accessible to the human subjects in random order. The data were derived from experiments conducted on 8 subjects aged between 22 and 30 years of age. The discrimination of dots experiment tested the ability of subjects to distinguish the presence or absence of stimuli on the surface of the micro-capsule paper. Both black on white and white on black dot patterns were used. Psychophysics is extensively used to answer questions regarding the way human beings react to external stimuli, such as variation in specified characteristics of environmental stimulation. Experiments usually reveal typical S-shaped curves [27]. The results for the conducted experiments show that the detection curves follow typical S-shaped curves, Fig. 9. The threshold value is typically taken as the stimulus value corresponding to the detection response of 50%. The threshold for the black dot detection in this study was found to be 0.4 mm. The detection of white dots in black background was found to be around 1.9 mm. Hence there exists a difference in the detection threshold of the dots in different background. These are attributed to characteristics and resolution of the tactile paper. The gaps in black background, which are seen as the white dots, do not produce noticeable depth for small dots. This is because the black dots
13
Probability of feeling dots (%)
100
80
60
40
20
Black dots White dots 0 0
0.5
1 1.5 2 Stimulus dimension (mm)
2.5
3
Fig. 9. Detection curves for dots
cause an enhancement that raises the material inside the square block, thereby negating any effect of depth. However for bigger clusters of white dots, the problem is not as pronounced. C. Frequency Response The frequency response of the human tactile system is another way of understanding the way human subjects react to the raised structures on the tactile paper. There has been substantial research in the designing and conducting of psychophysical experiments [28], [29] to analyze the human visual system. The basis of the study was to understand what happens to the spatial information 1 after they are received by the eye. Since the goal is to understand the processing of data flow within the tactile system, the visual experiments provide us with a good model to carry out the tactile system experiments. Visual perception is studied by recording and analyzing the response to sine wave gratings. Spatial sine wave gratings (whose luminance varies sinusoidally across space) are simple building 1
Spatial information refers to the brightness and constant attributes across the space
14
(a)
(b)
Fig. 10. Psychophysical experiments using gratings. (a)Gratings for human visual system tests; (b)Gratings for the human tactile system tests
blocks for complex stimuli and they are influenced by frequency, contrast, phase and orientation. To determine a psychophysical measurement of the sensitivity of humans to the sine wave gratings, human subjects are presented with a gratings of a given spatial frequency, Fig. 10(a). The frequency range is between 3 and 48 cycles per degree of visual angle [29]. Initially the subject cannot see a grating as the image of the grating is below threshold. The contrast of the grating is increased till the subject reports the sighting of the grating. The sensitivity for the particular spatial frequency is the inverse of the contrast threshold. This procedure is repeated for a large number of frequencies. The resulting curve plotting contrast sensitivity as a function of spatial frequency is known as the contrast sensitivity function (CSF). A number of models have been discussed [30] to model the CSF. The dependence of the contrast sensitivity Shvs on the radial spatial frequency, fhvs was developed by Campbell et al. [31], [32]. It can be written as follows:
Shvs = [e, fhvs , e, fhvs ] 2
2
(5)
where ,
and are constants. The constant is proportional to the average illumination. The constants and are set depending on the type of model chosen during the course of the experiment. A similar approach was carried out in the current study to determine the tactile sensitivity func-
15
100 Experimental data Model fit 90
80
Tactile Sensitivity
70
60
50
40
30
20
10
0
0
10
20
30 40 50 60 Spatial Frequency (cycles/paper)
70
80
90
Fig. 11. Tactile Sensitivity Curve for human beings.
tion (TSF). Gratings such as that shown in Fig. 10(b) were utilized. Non-blind subjects were blindfolded and presented with square-wave gratings, which were sufficiently raised after passing through the TIE. These gratings were at a fixed spatial frequency. The spatial frequency used were in the range between 1 to 30 cycles per unit of space. Noise was introduced in the square gratings to simulate the change in signal to noise ratio or the contrast of the gratings. The distribution of the noise varies exponentially with space. At one edge the noise power is very high, so that the subjects could not feel the gratings. The noise level was reduced until a point is reached where the subject reports feeling the gratings. The corresponding noise level was recorded and treated as the sensitivity value for the particular spatial frequency. This experiment was repeated for a number of different spatial frequencies. The resulting spatial sensitivity values are plotted against the width of the gratings, as shown in Fig. 11. The measured human TSF is a band-pass function, as it shows a peak and decreasing sensitivity on either side of this peak. A model for the tactile system utilizing the TSF data generated from the experiments is given
16
75 70
Tactile Sensitivity
65 60 55 50 45 40 0 5 10
20 15 15
10 5 20
0
Fig. 12. Human Tactile Sensitivity model. The x-axis is the horizontal spatial frequency (cycles/mm) and the y-axis is the vertical spatial frequency (cycles/mm)
by:
q
Shts = ( fhts=C ) (e,fhts=D )
(6)
where the constants C and D are calculated from the empirical data to be 4:65 and 43:245, respectively. Also, fhts is the radial spatial frequency in cycles per unit of space. The resulting model is shown in Fig. 12. IV. TACTILE
HALFTONE STRUCTURES
In printing devices where the direct reproduction of all the gray tones is impossible, patterns of binary pixels or halftones are substituted for the original image, in the process creating a visual illusion of similarity. This is possible due to the low-pass nature of the human eye, which averages black dots and white space to perceive a particular gray level. A similar band-pass response of the human touch sense needs to be taken into account when we are trying to devise a tactile imaging scheme based on halftones. In conventional halftoning using model-based techniques, the CSF of the human visual system is utilized to compare the halftoned image and the original continuous tone image. Our intention is to effectively map this idea to the tactile domain.
17
Human Tactile System
Image
Error
Human Tactile
Halftone
TIE
Process
Model
System
TIE Halftoned
Tactile
Image
Image
Fig. 13. Block Diagram for the optimal tactile imaging process.
Figure 13 shows the implementation of the proposed model based halftoning process for tactile imaging. This model is similar to the model-based halftoning method reported in [21], [22], [20]. We have substituted the human tactile system and the TIE for the human visual system and the printer, respectively. The model details the algorithm to produce the halftone patterns that are optimum for a particular halftone process. The original image is passed through an initial halftoning process. The model based halftoning requires a quantitative comparison between the tactile-perceived original image and the tactile-perceived halftoned image. The tactile-perceived original image is obtained by filtering the original image with a band-pass filter modeled according to the tactile sensitivity function, Equation ( 6) in Section III-C. The halftoned original image is passed through a model simulating the TIE. This transforms the distribution of dots to the simulated output of the TIE by taking into consideration the Gaussian surface model (Eq. 4) of the TIE. At this stage, the minimum criteria for recognition of the black and the white dots, as discussed in Section III-B, is used to screen the halftoned image. The halftoned original image is converted to raised halftone structures with the necessary constraints on dots characteristics. Finally, the tactile-perceived halftoned image is the result of output after filtering the raised halftone structures with the tactile sensitivity function band-pass filter model. The comparison of the two perceived images produces an error that is then superimposed on the halftoned image to arrive at an optimal dot arrangement.
18
The successful implementation of the model-based halftoning and the production of the optimum halftone structures requires a choice on the type of halftoning method. The following section discuss the advantages and shortfalls of the common halftoning techniques available. These methods are then compared with each other through a set of experiments. A. Halftoning methods Over the years a number of effective methods have been proposed in the field of halftoning. Ordered dither, or amplitude modulated (AM) halftoning [19], is one of the simplest methods. This method of creating an illusion of a continuous tone image is produced by varying the size of the dots that are printed along a regular lattice. The number of dots, or frequency of the dots, remains fixed. The original continuous-tone image is divided into blocks of smaller sizes that are then compared to a threshold in a pixel by pixel comparison. The dither array is composed of consecutive thresholds clustered together. Floyd and Steinberg [33] developed a halftoning technique, known as error diffusion, that distributes the error generated at each pixel. This became the foundation of the present day frequency modulated (FM) halftoning. In FM halftoning, the required illusion is achieved by varying the distance between printed dots while the dots are held at a constant size. This process is an adaptive algorithm that quantizes each pixel. The quantization error produced in each of the single pixel operations is distributed amongst the neighborhood of yet to be processed pixels. The pattern generated by this process was referred by Robert Ulichney as the blue-noise halftones [19]. This name results from the fact that the dither patterns is composed exclusively of high frequency spectral components. For a given gray level g , the minority pixels are separated by an average distance of b , which is the principle blue noise wavelength. The relation between the two parameters is given by:
8 q < D=( (g)) b = : q D=( (1 , g))
1=2 for 1=2 < g 1 for 0 < g
(7)
where D is the minimum distance between addressable points on the display. The error diffusion algorithm devised by Floyd and Steinberg, shown in Fig. 14, can be mathematically represented
19
X[n]
+
Y[n] + + −
e Y[n]
e X[n] B
Fig. 14. Error diffusion (Floyd and Steinberg, 1976)
(
as follows:
T e (8) Y [n] = 01 ifelse(X [n] , B Y [n]) 0 P b = 1, the output Y [n] = [y[n , 1]; y[n , 2]; :::; y[n , N ]]T where B = [b ; b ; :::; bN ]T , N i i and Y e [n] = [y e [n , 1]; y e[n , 2]; :::; y e [n , N ]]T , which follows from the equation Y e [n] = Y [n] , (X [n] , (B T Y e[n])). The input pixel under consideration in this expression is X [n]. 1
2
=0
An ideal printer outputs patterns that are composed of perfect square black dots. In high quality printing situations, where this effect is true to certain extent, blue-noise halftoning is considered the “optimum technique for minimizing visibility [34] and maximizing the apparent spatial resolution [35]”, as reported in [20]. In practical considerations involving digital printing, printer distortions are relevant. They are summarily categorized as dot-gain and dot-loss [20]. Dot-gain is the increase in size of the printed dot relative to its intended size, and is the main distortion in ink jet printing. The quality of paper also has a very crucial effect, with denser papers causing ink to spread instead of absorbing it completely. Dot loss appears in higher resolution printers where instead of creating a bigger dot, there is a difficulty in printing an isolated black dot. Model based halftoning and clustering of dots are techniques that have been developed to address these problems. The halftoning process that are of particular interest in the present context of clustering are the AM-FM hybrid stochastic halftoning techniques. These create minority pixel clusters that vary, according to tone, in both their size and spacing. The spectral characteristics of such patterns have
20
h X[n] H
X[n]
+
A
+
Y[n]
+
+ + -
e Y[n]
e X[n] B
Fig. 15. Error diffusion with output-dependent feedback: a weighted sum of the previous output pixels is used to vary the threshold. H-hysteresis term, A-hysteresis filter, B-error filter, X[n]-input image, Y[n]-halftoned image.
mid-frequency components. Hence they are collectively known as green-noise halftone patterns. In green-noise, the minority pixel clusters are distributed homogeneously. The average separation (center-to-center) between clusters is termed as g . Its square is inversely proportional to the average number of minority pixel clusters per unit area. The green-noise principle wavelength [20] is given by the following equation.
8 q < D=( (g)=M ) g = : q D=( (1 , g)=M )
1=2 for 1=2 < g 1 for 0 < g
(9)
where D is the minimum distance between addressable points on the display, g is a particular gray
is the average number of minority pixels per cluster in the binary dither pattern. level and M
The green noise halftone is generated by a error diffusion technique proposed by Levien [36]. He referred to it as the error diffusion with output dependent feedback, Fig. 15. In this algorithm a weighted sum of the previous output pixels is used to vary the threshold. This makes the minority pixels more likely to occur in clusters. The amount of clustering is controlled through the hysteresis constant H. Large values of H can cause large clustering and smaller values lead to smaller clusters. Levien’s algorithm is precisely defined as follows:
(
Y [n] = 01
if (X [n] + B T Y e [n] + HAT Y [n]) 0 else
(10)
21
= [a ; a ; :::; aN ]T , B = [b ; b ; :::; bN ]T , PNi ai = 1, PNi bi = 1, the output Y [n] = [y[n , 1]; y[n , 2]; :::; y[n , N ]]T and Y e[n] = [ye[n , 1]; ye[n , 2]; :::; ye[n , N ]]T , which follows from the equation Y e [n] = Y [n] , (X [n] + (B T Y e [n])). The error filter and hysteresis filter can where A
1
2
1
2
=0
=0
take on a wide range of values, including special cases such as Floyd-Steinberg [33], Jarvis [37] and Stucki [38] filter coefficients. Perturbation, in the form of noise, can also be added to these filter weights to create desirable results. B. Results and Evaluation Psychophysical experiments can be used to determine the optimum tactile halftone technique. The measure of goodness is defined as the level of effectiveness in feeling and understanding information through the manual exploration of the structures. As detailed in Fig. 13, the halftoning systems defined by the model transfer functions for AM, FM, and AM-FM hybrid methods are considered. As a first examination, a gray scale ramp test image is used as an original image and the corresponding halftones are generated by each method. These are shown in Fig. 16. A sample image having four gray levels is also processed by the three halftone techniques, Fig. 17. Upon observation of the generated halftone structures, we find certain differences that are particularly important in the case of tactile halftoning. First note that, as expected, the halftoned gray scale ramp produced by the AM halftoning method has a uniform distribution of dots, but varying dot size. Due to the regular spacing of dots, the (micro-capsule) growth of the dots is inhibited. That is, dots begin to merge once they are of a certain size. Thus no discrimination can be made once the dots begin to merge. Also, due to the asymmetry of the micro-capsule enhancement process, black dots and white dots merge at different thresholds. Accordingly, the tactile printing process places significant limitations on the use of AM halftoning. In the case of the blue noise, or FM, halftoning technique, the dots have similar size but are distributed at varying distance depending on the gray level. Once again, the minimum dot size imposed by the tactile printing process imposes a constraint. Since the minimum dot size for black
22
(a)
(b)
(c)
(d) Fig. 16. Test Image: (a) Original Ramp of gray scale; Test image halftoned using the HTS by (b)Amplitude Modulation (AM) halftoning (c) Frequency Modulation (FM) halftoning and (d) AM-FM hybrid halftoning
23
(a)
(b)
(c)
(d)
Fig. 17. Example using a test image (a) Original image (b) AM halftoned (c)FM halftoned (d)AM-FM halftoned
and white are different, both constraints cannot be met simultaneously under the FM structure. This results in merging of white dots and, as in the AM case, leads to a degradation in the quality of the resulting halftone. The green noise, or AM-FM hybrid, halftoning structure allows the minimum black and white dot size to co-exist, and the resulting halftone thus consists of dots within the experimentally defined limits. Also, since both spacing and dot distribution are stochastic processes, this method of generating tactile halftones is more robust to errors in either dot location or size. This results in halftones where the dots are not merged by the printing (micro-capsule paper enhancement) process. Thus, the produced halftones are tactilely distinct and more easily discriminated than either the AM or FM generated halftones. To test the validity of these visual observations, a set of experiments was conducted. The halftone methods discussed in previous sections were utilized to create halftones for the simple
24
(a)
(b)
(c)
(d)
(e)
Fig. 18. Test images halftoned by the AM-FM hybrid method for the edge and region identification experiments: (a) Hexagon (b) Triangle (c) Rectangle (d) Circle (e) image of rock with two gray levels.
geometric figures of hexagon, triangle, rectangle, circle and an image of rock. The structures, with halftoned patterns generated by the AM, FM, and AM-FM methods, constituted the set of stimuli for the experiments. Figure 18 shows the structures generated by the AM-FM hybrid method.
25
10 AM halftoning FM halftoning AM−FM hybrid
9
8
7
Time(secs)
6
5
4
3
2
1
0
1
2
3 Geometric Figures
4
5
Fig. 19. Timed identification of the geometric figures. The figures numbered 1 to 5 are Hexagon, Triangle, Rectangle, Circle, and an image of rock with two gray levels, respectively
The test patterns were enhanced as detailed earlier with the help of the TIE and were presented to blindfolded human subjects. The data were derived from experiments conducted on 8 subjects aged between 22 and 30 years of age, as before. The micro-capsule paper and the enhanced structures on it were shielded from the subject by blindfolds. In the first experiment, the halftones of the geometric figures produced by the three methods were numbered for our identification and presented to the subjects in random order. The subjects were asked to recognize the shape of the geometric figures by manual exploration of the figures on the tactile paper. The time taken by each subject to identify the figure, through manual exploration, was noted and recorded. This experiment was undertaken to understand how quickly the human subjects were able to recognize shapes based on each of the halftoning methods. As each halftoning method results in a different distribution of dots and subsequently different height/depth for the edge dots, this
26
70 AM halftoning FM halftoning AM−FM hybrid 60
Response to structures(%)
50
40
30
20
10
0
1
2
3 Geometric Figures
4
5
Fig. 20. Texture Discrimination.The figures numbered 1 to 5 are Hexagon, Triangle, Rectangle, Circle, and an image of rock with two gray levels, respectively
recording helps in defining which halftoning method produces better recognition. The average identification times for each method are shown in Fig. 19. The compiled results showed that, in all cases, the AM-FM hybrid halftoning method leads to the shortest identification time. Thus it can be noted that the more distinct structures produced by the AM-FM hybrid methods aid in identification. The second experiment tested how well a human subject is able to recognize the details in the textures of the structures. This was aimed at the texture discrimination rather than the shape identification, which was the goal in the first experiment. The human subjects were presented with the three halftones of the same geometrical figure and were asked to make a comparison of the quality of the structures on their surface. As before, the subjects were asked to manually explore the figures and report which of the three halftones presented to them, again in random order, was most distinct i.e. easily identifiable by touch. The experiment recorded which halftone method would give the most tactile perceivable
27
patterns on a surface for a given gray level. Figure 20 reports the responses recorded from this experiment. It was found that, in most cases, the AM halftoning is too weak to produce the necessary discrimination, most likely due to merging of dots. The FM halftoning, on the other hand, enables greater discrimination than the AM halftoning. The AM-FM hybrid halftoning method however, yields the greater discrimination. The psychophysical experiment results are thus in alignment with the analysis of the algorithms. The hybrid AM-FM halftoning, or green noise halftoning, is optimal for the generation of tactile output among the methods available in the literature. The performance of the hybrid method is likely due to its flexibility, in that it can be jointly optimized according to human tactile perception and the characteristics of the printing (microcapsule enhancement) process. Both the AM and FM methods produce suboptimal results, which can be attributed to identifiable shortcomings such as dot merging. Considerable work must still be undertaken to fully optimized and evaluate halftoning methods for tactile representations. For instance, it is not currently known what an appropriate number of tactile gray level representations is. It is unlikely that a user would be able to discriminate amongst, say, the 256 gray levels in an 8–bit image, but discrimination amongst a smaller number of halftoning patterns should be possible. Further studies are needed to clarify this issue. Additionally, the methods must be evaluated on other output medium and displays, such as the TIGER embossing printer. These issues are amongst those that will be the subject of further investigations. V. C ONCLUSIONS This paper introduced the concept of digital halftoning for tactile imaging. To develop appropriate tactile halftoning methods, the characteristics of the Tactile Image Enhancer (TIE) were studied in detail. This study led to an accurate model of the microcapsule paper enhancement process. Additionally, experiments were conducted to determine human tactile sensitivity and frequency response for the case of manual exploration of microcapsule paper. The TIE model and human tactile response were integrated into a general tactile halftoning procedure. This procedure includes as special cases the widely used AM and FM halftoning, as well as the more flexible
28
AM–FM hybrid green noise technique. Each of the methods were analyzed for their applicability in tactile halftoning and sample outputs were generated. Psychophysical experiments on timed identification and discrimination were conducted. The algorithm analysis and experiments indicate that the AM–FM hybrid method produces the most tactilely appropriate halftoning patterns. Further investigations will be conducted to more fully evaluate the developed methods to determine, for instance, the number of halftoning patterns that can be discriminated by users, and additional research will be conducted to optimize the procedures for other tactile display media. VI. ACKNOWLEDGMENT This work was supported by the National Science Foundation under the grants 9800175 and 9875658. R EFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]
S. Millar, Reading by Touch. Routledge, New York, USA, 1997. J. Bliss, “A relatively high-reslution reading aid for the blind,” IEEE Transactions on Man-Machine Systems, vol. 10, pp. 1–9, 1969. J.G.Linvill and J. Bliss, “A direct translation reading aid for the blind,” in Proceedings of the IEEE, vol. 54, pp. 40–51, 1966. J. Loomis, “Tactile pattern perception,” Perception, vol. 10, 1981. J. Craig and C. Sherrick, “Dynamic tactile displays,” in Tactual Perception: A Sourcebook (W. Schiff and E. Foulke, eds.), Cambridge University Press, 1982. J. Phillips and K. Johnson, “Tactile spatial resolution (i): Two-point discrimination, gap detection, grating resolution, and letter recognition,” Journal of Neurophysiology, vol. 46, pp. 1177–1191, 1981. J. Phillips and K. Johnson, “Tactile spatial resolution (ii): Neural representation of bars, edges and gratings in monkey primary afferents,” Journal of Neurophysiology, vol. 46, pp. 1192–1203, December 1981. J. Phillips and K. Johnson, “Tactile spatial resolution (iii):a continuum mechanics model of skin predicting mechanorecptor responses to bars, edges and gratings,” Journal of Neurophysiology, vol. 46, pp. 1204–1225, December 1981. M. Kurze, L. Reichert, and T. Strothotte, “Access to business graphics for blind people,” Proceedings of the RESNA ’94 Annual Conference, June 17-22 1994. R. Klatzky, S. Lederman, and V. Metzger, “Identifying objects by touch: An “expert system”,” Perception and Psychophysics, vol. 37, no. 4, 1985. R. Klatzky, S. Lederman, and C. Reed, “There’s more to touch than meets the eye: The salience of object attributes for haptics with and without vision,” Journal of Experimental Psychology: General, vol. 116, 1987. K. Kokjer, “The information capacity of the human fingertip,” IEEE Transactions on Systems, Man, and Cybernetics, vol. SMC-17, no. 1, 1987. T. P. Way and K. E. Barner, “Automatic visual to tactile translation, part I: Human factors, access methods and image manipulation,” IEEE Transactions on Rehabilitation Engineering, vol. 5, pp. 81–94, Mar. 1997. T. P. Way and K. E. Barner, “Automatic visual to tactile translation, part II: Evaluation of the tactile image creation system,” IEEE Transactions on Rehabilitation Engineering, vol. 5, pp. 95–105, Mar. 1997. S. Hernandez and K. Barner, “Joint region merging criteria for watershed-based image segm entation,” in 2000 International Conference on Image Processing, (Vancouver, BC, Canada), September 10-13 2000. S. Hernandez and K. E. Barner, “Color image segmentation using watersheds and joint region merging criteria,” IEEE Transactions on Image Processing, Aug. 2000. Submitted for publication. S. Hernandez and K. Barner, “Tactile imaging using watershed-based image segmentation,” in ASSETS 2000, (Arlington, Virginia, USA), November 13-15 2000. W. Schiff and E. Foulke, Tactual perception: a sourcebook. Cambridge University Press, 1982. R. A. Ulichney, Digital Halftoning. Cambridge, MA: MIT Press, 1987. D. L. Lau, Modern Digital Halftoning. PhD thesis, University of Delaware, Newark, Delaware, USA, May 1999. T. N. Pappas and D. L. Neuhoff, “Model-based halftoning,” in Proceedings of SPIE, Human Vision, Vision Processing and Digital Display II (B. E. Rogowitz, M. H. Brill, and J. P. Allebach, eds.), vol. 1453, pp. 244–255, June 1991.
29
[22] T. N. Pappas, “Model-based techniques for digital halftoning,” in Proceedings, International Conference on Image Processing (B. Werner, ed.), vol. II, pp. 26–30, IEEE, 1994. [23] J. Gardner, “Tactile graphics: An overview and resource guide,” Available from http://www.rit.edu/ easi/itd/itdv03n4/article2.html, 1999. [24] D. Pawluk, A Viscoelastic Model of the Human Fingerpad and a Holistic Model of the Human Touch. PhD thesis, Harvard University, Cambridge, Massachusetts, USA, May 1997. [25] D. Pawluk and R. Howe, “A holistic model of human touch,” in 5th Annual C.N.S. Meeting, 1996. [26] R. LaMotte and J. Whitehouse, “Tactile detection of a dot on a smooth surface: Peripheral neural events,” Journal of Neurophysiology, vol. 56, no. 4, 1986. [27] H. Schiffman, Sensation and Perception: An Itegrated Approach. John Wiley and Sons, New York, 1976. [28] T. Cornsweet, Visual Perception. New York: Academic press, 1971. [29] F. W. Campbell, “The transmission of spatial information through the visual system,” The Neurosciences Third Study Program, pp. 95–103, 1974. [30] H. R. Kang, Digital Color Halftoning. SPIE Optical Engineering Press and IEEE Press,USA, 1999. [31] F. W. Campbell, R. H. Carpenter, and J. Levinson, “Visibility of aperiodic patterns compared with that of sinusoidal gratings,” The Journal of Physiology, vol. 190, pp. 283–298, 1969. [32] F. W. Campbell, “The human eye as an optical filter,” Proceedings of the IEEE, vol. 56, no. 6, pp. 1009–1014, 1968. [33] R. W. Floyd and L. Steinberg, “An adaptive algorithm for spatial gray-scale,” Proceedings Society Information Display, vol. 17, pp. 75–78, Feb. 1976. [34] J. Sullivan, L. Ray, and R. Miller, “Design of minimum visual modulation halftone patterns,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 21, pp. 33–38, Jan. 1991. [35] M. Rodriguez, “Graphic arts perspective on digital halftoning,” in Proceedings of SPIE, Human Vision, Visual Processing, and Digital Display V (B. E. Rogowitz and J. P. Allebach, eds.), vol. 2179, pp. 144–149, Feb. 1994. [36] R. Levien, “Output dependant feedback in error diffusion halftoning,” in IS&T’s Eighth International Congress on Advances in Non-Impact Printing Technologies, (Williamsburg, Virginia, USA), pp. 280–282, October 25-30 1992. [37] J. F. Jarvis, C. N. Judice, and W. H. Ninke, “A survey of techniques for the display of continuous-tone pictures on bilevel displays,” Computer Graphics and Image Processing, vol. 5, pp. 13–40, 1976. [38] P. Stucki, “Mecca-a multiple-error correcting computation algorithm for bilevel image hardcopy reproduction,” Tech. Rep. RZ1060, IBM Research Laboratory, Zurich, Switzerland, 1981.
Optimal Halftoning for Tactile Imaging Amit Nayak, Student Member, IEEE and Kenneth E. Barner, Senior Member, IEEE Department of Electrical and Computer Engineering University of Delaware, Newark, Delaware 19716, USA [email protected], [email protected] Abstract Reading of text and understanding images by touch is an important alternative and additional source of information when sight is absent or lost. Tactile graphics and models such as edge maps, binary output etc. are the solution for simple access to images for blind persons. This paper introduces an approach to model the human tactile system based on the responses produced by stimuli on micro-capsule paper. This system is utilized for the purpose of generating optimum halftone patterns on micro-capsule paper that can be utilized for the effective generation of tactile graphics.
I. I NTRODUCTION Vision plays a pivotal role in information gathering for human beings. For instance, much of the sciences involve the visualization of materials and concepts. Valuable information in text and images is thus gathered by humans through the visual sense. This reliance on vision is hampered, or lost, in individuals with visual impairments. In such cases, it is necessary for the information display to include alternative modalities. The main system of written communication utilized by blind people throughout the world is the Braille system. Invented by Louise Braille in the early nineteenth century, this system proved to be better than the earlier, difficult and cumbersome, method of embossed letter reading. The use of Braille method for reading and writing, has been researched at depth for its utility by various researchers [1]. This method, however, allows tactile access to only text based information. Tactile imaging is the process of converting a visual image into a touchable binary raised version. Applications of tactile graphics have been researched on in several places. Tactile pattern generation received a boost with the invention of Optacon and the subsequent modification [2],
2
[3]. Loomis [4] and Craig [5] initiated a discussion on the tactile pattern perception and the factors which could affect this perception. One of the earliest groups to look in to the sensory aspects of skin in the human finger, Johnson et al. [6], [7], [8] worked on the the discrimination and detection of points, gaps, gratings and letters by the human finger. Martin et al. [9] have worked in the presentation of business graphics for blind people. Pie charts, bar charts and other form of representation of information in image format have been successfully converted into relevant form. The data on which the graphical information is based is extracted from the image and then the tabulated version of this data is conveyed to the user either by speech, sound and/or tactile form. Speech-synthesizer, using acoustically different frequency tones are some ways and in the case of bar charts the most favored way is using a 2D-Braille-Display, wherein raised dots are used to depict the length of the actual bar in the y-axis and the x-axis is coupled as a Braille output. Klatzky et al. [10], in 1985, discussed the identification of objects by touch and have subsequently compared the amount of information garnered by the visual and the haptic interface [11] in 1987. Similarly the same time investigators had attempted to quantify the information channel capacity of the human sensory organs and Kokjer [12] suggested an order-of-magnitude upper limit on this property for the vibrotactile stimulation of human fingertip as 100 bit/s. This, when compared with corresponding limits in the human ear and eye, a progression of
10 :10 :10 2
4
6
re-
spectively, was found. Thus the capacity of visual sense to receive and perceive information is the highest, as compared to hearing and touch. Of the various studies and research done in the field of generation of tactile graphics, of particular interest are that of [13], [14], which successfully developed software algorithms for automatic generation of tactile graphics. Meaningful information is identified and extracted from original image data incorporating a multi-step procedure. These include edge and boundary information extraction from the original image and representing them in tactile format. Edge and boundary information refers to identification of regions in an image where a rapid change of intensity takes place. Also important in the procedures are the successful segmentation of the original image. These image processing methods were further developed by Hernandez et al. [15], [16], [17] who
3
used the edge information to generate graphics that rendered image boundary in tactile form. Edge information on its own is, however, not enough to present the complete effect. A tactile image should contain different textures [18] to help distinguish and discriminate between various shapes and colors present in an image. We, therefore, propose optimal tactile imaging textures. The underlying idea is that the various regions in segmented images can now be easily identified by touch if they are given a separate textures. To accomplish this, we use digital halftoning methods to create patterns for the images which can then be suitably enhanced to create the feel of various textures. Digital Halftoning is the process of generating a binary pixel patterns that create the illusion of a continuous-tone image. This technique is important for display of gray-scale images in printing process, where the direct rendition of the gray tones is not possible. In recent years a number of comprehensive methods have been designed to carry out the halftoning process. The primary methods are reviewed in [19], [20]. Optimum algorithms [21], [22] were found to be those that exploit the dot-printer models and take into account the low pass nature of the human vision. We extend this idea of model-based halftoning to the domain of the human tactile system. As used extensively in the human visual system (HVS) modeled halftoning system, a transfer function for the sensory system is required along with an understanding of the printing process. In an analogous fashion, we develop a model for the human tactile system and the tactile printing process. In the case considered, we utilize the Tactile Image Enhancer, Fig. 1, operating on microcapsule paper [13], [14], [23] as the tactile output. The output characteristics of the TIE were examined. We develop a model for the surface generated from the TIE. We also devise a model of the human tactile perception based on the spatial frequency response of the human tactile sense. These models are integrated into the process of generating halftones best suited for a given gray scale. The remainder of this paper is organized as follows. The working of the TIE and the measurements of the output characteristics and resulting enhancement model are presented in Section 2. In Section 3, we discuss the work carried out earlier on the characterization and specification of the human finger pad. New experiments are carried out to find how sensitive the human finger is to
4
Fig. 1. Tactile Image Enhancer (TIE is manufactured by Repro-tronics Inc.).
the surface of the tactile paper, the results of which are reported in Section 4. Optimal halftoning methods are discussed in Section 5. Finally, the conclusions and further research are presented in Section 6. II. I MAGE E NHANCEMENT P ROCESS Tactile images can be formed by enhancing, or raising, the surface of a special paper called tactile or micro-capsule paper through use of the TIE, Fig. 1. The TIE is manufactured by the Repro-tronics Inc. The first prototype was introduced in the market in 1993 and have been in use since then. As of now there is a smaller, more portable version of TIE available in the market. The TIE works on a very simple principle. It has a motor-driven roller that passes the tactile paper, face up, underneath a tubular light bulb, Fig. 2. The light bulb acts as a heat source and the rollers help in guiding the paper through the direct contact of the bulb. The type of tactile paper which we intend to focus on is manufactured by the Matsumoto Kosan Company Ltd. in Japan. This is also known as the micro-capsule paper. As the name suggests it is a white paper coated with microscopic polystyrene capsules. The Micro-capsule coating on the paper is heat reactive. Black lines or images printed on the paper, when exposed to a heat source, absorb more heat than the surrounding areas. This causes the underlying capsules to grow. The result is the generation of raised lines, areas, and symbols, i.e., tactile graphics. A temperature range of 120-125 C (248-257 F) is appropriate for the heating of the papers. In the present study
5
Polystyrene Microcapsules
Polyethylene medium Image (printed toner particles)
(a) Heating element
Paper path
Transport rollers
(b) Expanded Capsules
(c) Fig. 2. (a) Micro-capsule paper with image toner particles photocopied on it.(b) Inside the TIE.(c) Raised microcapsules after the paper is passed through the TIE
we have preferred to use 124 C(255.2 F). In order to understand and model the changes occurring in the enhancement process, a simple experiment was carried out. An array of test samples of dots were used. The dot were square pixels printed on a white paper at a resolution of 100 dots per inch (dpi). The size of the dots was uniformly increased from a 1 1 to a 20 20 dot cluster. The resulting outputs were examined for their change in dimension. Images were prepared, comprising of black square dots on a white background and black dots with square gaps simulating white square dots on black background, Fig. 3. These images were photocopied onto the tactile paper. The tactile paper was then passed through the TIE.
6
(a)
(b)
Fig. 3. (a) Black Dots on white background. (b) White dots on black background.
The structures such generated through this test have profiles that range between -0.5 and 0.5 mm in height. A Scanning Electron Microscope(SEM) was used to view the vertical growth of the samples. The profiles of the samples were captured and recorded. The profile of one of the black dot on white background, with a dot size of 8 8, is shown in Fig 4. Simple image processing techniques were used to extract the edge information and these were superimposed to see the growth of the dot clusters as a function of cluster size. Fig. 5 shows one-dimensional profiles for squares of sizes 1 1, 2 2,... etc which are printed at a resolution of 100 dpi. The inverse case, which we can observe as white dots in black background, follow a complementary pattern. The growth in both directions were recorded. The average height or depth of the dot profiles were found to follow the curves shown in Fig. 6. The clusters of dots are characterized by the way the height/depth and width of the dots behave after passing through the TIE. The curves of the profiles are modeled as follows:
Fu = Au + Buln(i)
(1)
Fd = Ad (e, i,Bd 2 =Cd , 1)
(2)
(
)
Fu and Fd denote the height and depth parameters respectively, and the constants were determined experimentally to be Au =0.026, Bu = 0:16 and Ad = 0:5, Bd = 0:75, Cd = 1:596 and i denotes the size of the cluster of dots. Thus, the height and depth parameter of the dot clusters where
7
Fig. 4. The Test pattern corresponding to the 8 x 8 pattern as seen through the SEM. 0.025
Height (inches)
0.02
0.015
0.01
0.005
0 −0.08 −0.06 −0.04 −0.02 0 0.02 0.04 Width of clusters of dots (inches)
0.06
(a) 0.005
Depth (inches)
0 −0.005 −0.01 −0.015 −0.02 −0.025
−0.04
−0.02 0 0.02 Width of clusters of dots (inches)
0.04
(b) Fig. 5. Edge variation for the (a) black dots on a white background, (b) white dots on a black background.
have been modeled either as exponential or logarithmic curves.
8
0.5
Height of dots (mm)
0.4
0.3
0.2
0.1
Experimental data Logarithmic curve fit 0 0
1
2 3 Width of dots (mm)
4
5
(a) Curve fit Experimental data
0
Depth of dots (mm)
−0.1 −0.2 −0.3 −0.4 −0.5
1
2
3 4 Width of dots (mm)
5
(b) Fig. 6. (a) Height variation of black dots on white background. (b) Depth variation of white dots on black background.
Consider next the profiles of the dot clusters. On a simple suitable observation, we find that the profile of a single pixel is approximated by a Gaussian surface as shown in the Fig. 7:
G ; = Ae,x2 =B
(3)
11
where the constants were determined experimentally to be
A = 0:00134, B = 0:02752 and x
represents the distance along the width. Larger clusters are modeled as linear combinations of dots. Thus a n n cluster is modeled through an amplitude adjusted convolution model:
Fn;n = n;n(G ; In;n) 11
(4)
9 −4
Height (inches)
15
x 10
10
5
0 −0.015
−0.01
−0.005 0 0.005 Width of dot (inches)
0.01
0.015
Fig. 7. Profile and the model for a unit impulse comprising of dot of size 1x1. −3
Height (inches)
15
x 10
Experimental data Model fit
10
5
0 −0.06
−0.04
−0.02 0 0.02 Width of dot−cluster (inches)
0.04
0.06
Fig. 8. Model fitted curves for the black dots on white background
where Fn;n is the model suitable for an enhanced profile of an n n cluster of dots and n;n is the average attainable height/depth of the profiles for an n n cluster dot, as shown in Fig. 5(a) and
Fig. 5(b). In;n is the collection of impulses representing a cluster of dot of size n n. These have been plotted in Fig. 6. The resulting modeled curves are found to follow the profiles as predicted. The modeled curves superimposed with the measured curves are seen in Fig. 8. This kind of modeling can also be achieved for the white dots on black background.
10
III. TACTILE
RESOLUTION
Images that are enhanced using the micro-capsule paper processed by the TIE are ready to be used as tactile graphics. It is necessary, however, to understand how human subjects respond to the structures on the micro-capsule paper. A group of experiments were carried out to arrive at a mathematical model for the human tactile system and its response to the structures on the micro-capsule paper. The spatial resolution of the dots and the ability to produce a dot profile of substantial distinction as well as the frequency response of the tactile system is investigated in the experiments. A. Review of previous work The discussion on understanding the tactile pattern perception and the factors affecting this perception was started by Loomis [4] and Craig [5]. Johnson et al.
[6], [7], [8] began the
important exploration of the sensory aspects of the human skin. More recently, investigation was carried out by Pawluk et al. [24], [25] to develop an engineering model describing the response of the human peripheral tactile system when an object(a flat indentor) dynamically contacts the finger-pad. Here the system was measured at both mechanically, at the surface of the skin, and neurophysiologically, as the resulting nerve signal heads towards the brain. The investigation made into the spatial neural mechanisms underlying the tactile sensation by Johnson et al. [6], [7], [8] is divided into three parts. In the first part, specifications of the discrimination behavior that depends strictly on spatial neural mechanism is discussed. Experiments are carried out in this regard to define which aspects of the spatial discrimination are based on spatial information in the neural discharge patterns. It was found that human subjects could reliably discriminate between one and two 0.5 mm (0.019 in.) diameter points even when there was no separation between the two points. A high level of spatial resolution was thus demonstrated. Another set of experiments were carried out in which the subjects were required to discriminate between stimuli with and without gaps. A success criteria of 75% was used and it was found on this basis that gap size at the threshold of edge detection was 0.87 mm(0.034 in.). Subjects were also
11
required to feel square-wave gratings that were oriented along or across the axis of the finger. It was found that grating orientation was difficult to decipher until the gaps exceeded 0.5 mm(0.019 in.). The gap size, which confirmed with the success criteria of
75% accuracy, was found to be
0.84 mm(0.033 in.). In the second part of the experiments a similar study was carried out to find the neural responses in the monkey finger pad skin. In the third series of experiments a mechanistic model of skin is developed to predict the stress and strain at mechanoreceptor terminals within the skin. A parallel approach made by Lamotte and Whitehouse [26], discusses the capabilities of humans to detect the presence of a single raised dot of 550m(0.021 in.) diameter on a smooth plate and to judge the magnitude of evoked sensation. The corresponding stimuli on three different kind of mechanoreceptive peripheral nerve fibers of the monkey finger-pad (namely the slowly adapting (SA), the rapidly adapting(RA)and the pacinian (PC) mechanoreceptive peripheral nerve fibers ) was studied for dots of different heights. The mean detection threshold was found to be about 2.10.3 m (9.410,5 in). Also it has
been found that a height of 1 m (3.937 10,5 in ) could be detected if the diameter of dot was 600 m (0.0236 in) or wider. In similar fashion a height of 6 m (2.36
10,
4
in) was required
for detection if the dot diameter was reduced to 40 m (1.5748 10,3 in). In another experiment, stroke velocities of about 10 mm/s (0.393 in/s) were used and the dot detection thresholds were found out to be between 1(3.937 10,5 in) and 3 m (1.181 10,4 in) for all human observers. This again was different for different mechanoreceptive nerve fibers. It was reported that the magnitude of sensation evoked in humans increased with an increase in dot height above the dot detection threshold as defined in the preceding discussion. It was found that the impulses evoked in monkey RAs increased with dot height as did the widths of RA receptive fields. Also, it was hypothesized that the mechanical event responsible for the activating the RA was the lateral deformation of elevated regions of skin. This was found to be true because the number of impulses evoked in RAs by a dot was greater when the dot was stroked across, as opposed to along, the papillary ridges. It is conclusively proved that the responses of RAs alone
12
account for the sensory capacity to detect dot of minimal height on a smooth surface with the finger pad. In all these experiments described above the object to be felt by the finger-pad is an ideal system with flat surfaces and definite structures. The enhanced micro-capsule paper does not suit an ideal system features and hence the results cannot be directly used in its case. B. Discrimination of dots This experiment simulated the work carried out by and Whitehouse [26] on the micro-capsule paper. Test patterns of individual dots of sizes ranging between a single dot and a cluster of 20 20 dots were created as shown in Fig. 3. The human subjects were required to feel the patterns and respond on the detection of the presence or absence of any structures. Sufficient time was provided to each subject to feel the structures and then define the response. The micro-capsule paper and the enhanced structures on it were shielded from the subject by blindfolds. The tactile paper sample containing these test material were made accessible to the human subjects in random order. The data were derived from experiments conducted on 8 subjects aged between 22 and 30 years of age. The discrimination of dots experiment tested the ability of subjects to distinguish the presence or absence of stimuli on the surface of the micro-capsule paper. Both black on white and white on black dot patterns were used. Psychophysics is extensively used to answer questions regarding the way human beings react to external stimuli, such as variation in specified characteristics of environmental stimulation. Experiments usually reveal typical S-shaped curves [27]. The results for the conducted experiments show that the detection curves follow typical S-shaped curves, Fig. 9. The threshold value is typically taken as the stimulus value corresponding to the detection response of 50%. The threshold for the black dot detection in this study was found to be 0.4 mm. The detection of white dots in black background was found to be around 1.9 mm. Hence there exists a difference in the detection threshold of the dots in different background. These are attributed to characteristics and resolution of the tactile paper. The gaps in black background, which are seen as the white dots, do not produce noticeable depth for small dots. This is because the black dots
13
Probability of feeling dots (%)
100
80
60
40
20
Black dots White dots 0 0
0.5
1 1.5 2 Stimulus dimension (mm)
2.5
3
Fig. 9. Detection curves for dots
cause an enhancement that raises the material inside the square block, thereby negating any effect of depth. However for bigger clusters of white dots, the problem is not as pronounced. C. Frequency Response The frequency response of the human tactile system is another way of understanding the way human subjects react to the raised structures on the tactile paper. There has been substantial research in the designing and conducting of psychophysical experiments [28], [29] to analyze the human visual system. The basis of the study was to understand what happens to the spatial information 1 after they are received by the eye. Since the goal is to understand the processing of data flow within the tactile system, the visual experiments provide us with a good model to carry out the tactile system experiments. Visual perception is studied by recording and analyzing the response to sine wave gratings. Spatial sine wave gratings (whose luminance varies sinusoidally across space) are simple building 1
Spatial information refers to the brightness and constant attributes across the space
14
(a)
(b)
Fig. 10. Psychophysical experiments using gratings. (a)Gratings for human visual system tests; (b)Gratings for the human tactile system tests
blocks for complex stimuli and they are influenced by frequency, contrast, phase and orientation. To determine a psychophysical measurement of the sensitivity of humans to the sine wave gratings, human subjects are presented with a gratings of a given spatial frequency, Fig. 10(a). The frequency range is between 3 and 48 cycles per degree of visual angle [29]. Initially the subject cannot see a grating as the image of the grating is below threshold. The contrast of the grating is increased till the subject reports the sighting of the grating. The sensitivity for the particular spatial frequency is the inverse of the contrast threshold. This procedure is repeated for a large number of frequencies. The resulting curve plotting contrast sensitivity as a function of spatial frequency is known as the contrast sensitivity function (CSF). A number of models have been discussed [30] to model the CSF. The dependence of the contrast sensitivity Shvs on the radial spatial frequency, fhvs was developed by Campbell et al. [31], [32]. It can be written as follows:
Shvs = [e, fhvs , e, fhvs ] 2
2
(5)
where ,
and are constants. The constant is proportional to the average illumination. The constants and are set depending on the type of model chosen during the course of the experiment. A similar approach was carried out in the current study to determine the tactile sensitivity func-
15
100 Experimental data Model fit 90
80
Tactile Sensitivity
70
60
50
40
30
20
10
0
0
10
20
30 40 50 60 Spatial Frequency (cycles/paper)
70
80
90
Fig. 11. Tactile Sensitivity Curve for human beings.
tion (TSF). Gratings such as that shown in Fig. 10(b) were utilized. Non-blind subjects were blindfolded and presented with square-wave gratings, which were sufficiently raised after passing through the TIE. These gratings were at a fixed spatial frequency. The spatial frequency used were in the range between 1 to 30 cycles per unit of space. Noise was introduced in the square gratings to simulate the change in signal to noise ratio or the contrast of the gratings. The distribution of the noise varies exponentially with space. At one edge the noise power is very high, so that the subjects could not feel the gratings. The noise level was reduced until a point is reached where the subject reports feeling the gratings. The corresponding noise level was recorded and treated as the sensitivity value for the particular spatial frequency. This experiment was repeated for a number of different spatial frequencies. The resulting spatial sensitivity values are plotted against the width of the gratings, as shown in Fig. 11. The measured human TSF is a band-pass function, as it shows a peak and decreasing sensitivity on either side of this peak. A model for the tactile system utilizing the TSF data generated from the experiments is given
16
75 70
Tactile Sensitivity
65 60 55 50 45 40 0 5 10
20 15 15
10 5 20
0
Fig. 12. Human Tactile Sensitivity model. The x-axis is the horizontal spatial frequency (cycles/mm) and the y-axis is the vertical spatial frequency (cycles/mm)
by:
q
Shts = ( fhts=C ) (e,fhts=D )
(6)
where the constants C and D are calculated from the empirical data to be 4:65 and 43:245, respectively. Also, fhts is the radial spatial frequency in cycles per unit of space. The resulting model is shown in Fig. 12. IV. TACTILE
HALFTONE STRUCTURES
In printing devices where the direct reproduction of all the gray tones is impossible, patterns of binary pixels or halftones are substituted for the original image, in the process creating a visual illusion of similarity. This is possible due to the low-pass nature of the human eye, which averages black dots and white space to perceive a particular gray level. A similar band-pass response of the human touch sense needs to be taken into account when we are trying to devise a tactile imaging scheme based on halftones. In conventional halftoning using model-based techniques, the CSF of the human visual system is utilized to compare the halftoned image and the original continuous tone image. Our intention is to effectively map this idea to the tactile domain.
17
Human Tactile System
Image
Error
Human Tactile
Halftone
TIE
Process
Model
System
TIE Halftoned
Tactile
Image
Image
Fig. 13. Block Diagram for the optimal tactile imaging process.
Figure 13 shows the implementation of the proposed model based halftoning process for tactile imaging. This model is similar to the model-based halftoning method reported in [21], [22], [20]. We have substituted the human tactile system and the TIE for the human visual system and the printer, respectively. The model details the algorithm to produce the halftone patterns that are optimum for a particular halftone process. The original image is passed through an initial halftoning process. The model based halftoning requires a quantitative comparison between the tactile-perceived original image and the tactile-perceived halftoned image. The tactile-perceived original image is obtained by filtering the original image with a band-pass filter modeled according to the tactile sensitivity function, Equation ( 6) in Section III-C. The halftoned original image is passed through a model simulating the TIE. This transforms the distribution of dots to the simulated output of the TIE by taking into consideration the Gaussian surface model (Eq. 4) of the TIE. At this stage, the minimum criteria for recognition of the black and the white dots, as discussed in Section III-B, is used to screen the halftoned image. The halftoned original image is converted to raised halftone structures with the necessary constraints on dots characteristics. Finally, the tactile-perceived halftoned image is the result of output after filtering the raised halftone structures with the tactile sensitivity function band-pass filter model. The comparison of the two perceived images produces an error that is then superimposed on the halftoned image to arrive at an optimal dot arrangement.
18
The successful implementation of the model-based halftoning and the production of the optimum halftone structures requires a choice on the type of halftoning method. The following section discuss the advantages and shortfalls of the common halftoning techniques available. These methods are then compared with each other through a set of experiments. A. Halftoning methods Over the years a number of effective methods have been proposed in the field of halftoning. Ordered dither, or amplitude modulated (AM) halftoning [19], is one of the simplest methods. This method of creating an illusion of a continuous tone image is produced by varying the size of the dots that are printed along a regular lattice. The number of dots, or frequency of the dots, remains fixed. The original continuous-tone image is divided into blocks of smaller sizes that are then compared to a threshold in a pixel by pixel comparison. The dither array is composed of consecutive thresholds clustered together. Floyd and Steinberg [33] developed a halftoning technique, known as error diffusion, that distributes the error generated at each pixel. This became the foundation of the present day frequency modulated (FM) halftoning. In FM halftoning, the required illusion is achieved by varying the distance between printed dots while the dots are held at a constant size. This process is an adaptive algorithm that quantizes each pixel. The quantization error produced in each of the single pixel operations is distributed amongst the neighborhood of yet to be processed pixels. The pattern generated by this process was referred by Robert Ulichney as the blue-noise halftones [19]. This name results from the fact that the dither patterns is composed exclusively of high frequency spectral components. For a given gray level g , the minority pixels are separated by an average distance of b , which is the principle blue noise wavelength. The relation between the two parameters is given by:
8 q < D=( (g)) b = : q D=( (1 , g))
1=2 for 1=2 < g 1 for 0 < g
(7)
where D is the minimum distance between addressable points on the display. The error diffusion algorithm devised by Floyd and Steinberg, shown in Fig. 14, can be mathematically represented
19
X[n]
+
Y[n] + + −
e Y[n]
e X[n] B
Fig. 14. Error diffusion (Floyd and Steinberg, 1976)
(
as follows:
T e (8) Y [n] = 01 ifelse(X [n] , B Y [n]) 0 P b = 1, the output Y [n] = [y[n , 1]; y[n , 2]; :::; y[n , N ]]T where B = [b ; b ; :::; bN ]T , N i i and Y e [n] = [y e [n , 1]; y e[n , 2]; :::; y e [n , N ]]T , which follows from the equation Y e [n] = Y [n] , (X [n] , (B T Y e[n])). The input pixel under consideration in this expression is X [n]. 1
2
=0
An ideal printer outputs patterns that are composed of perfect square black dots. In high quality printing situations, where this effect is true to certain extent, blue-noise halftoning is considered the “optimum technique for minimizing visibility [34] and maximizing the apparent spatial resolution [35]”, as reported in [20]. In practical considerations involving digital printing, printer distortions are relevant. They are summarily categorized as dot-gain and dot-loss [20]. Dot-gain is the increase in size of the printed dot relative to its intended size, and is the main distortion in ink jet printing. The quality of paper also has a very crucial effect, with denser papers causing ink to spread instead of absorbing it completely. Dot loss appears in higher resolution printers where instead of creating a bigger dot, there is a difficulty in printing an isolated black dot. Model based halftoning and clustering of dots are techniques that have been developed to address these problems. The halftoning process that are of particular interest in the present context of clustering are the AM-FM hybrid stochastic halftoning techniques. These create minority pixel clusters that vary, according to tone, in both their size and spacing. The spectral characteristics of such patterns have
20
h X[n] H
X[n]
+
A
+
Y[n]
+
+ + -
e Y[n]
e X[n] B
Fig. 15. Error diffusion with output-dependent feedback: a weighted sum of the previous output pixels is used to vary the threshold. H-hysteresis term, A-hysteresis filter, B-error filter, X[n]-input image, Y[n]-halftoned image.
mid-frequency components. Hence they are collectively known as green-noise halftone patterns. In green-noise, the minority pixel clusters are distributed homogeneously. The average separation (center-to-center) between clusters is termed as g . Its square is inversely proportional to the average number of minority pixel clusters per unit area. The green-noise principle wavelength [20] is given by the following equation.
8 q < D=( (g)=M ) g = : q D=( (1 , g)=M )
1=2 for 1=2 < g 1 for 0 < g
(9)
where D is the minimum distance between addressable points on the display, g is a particular gray
is the average number of minority pixels per cluster in the binary dither pattern. level and M
The green noise halftone is generated by a error diffusion technique proposed by Levien [36]. He referred to it as the error diffusion with output dependent feedback, Fig. 15. In this algorithm a weighted sum of the previous output pixels is used to vary the threshold. This makes the minority pixels more likely to occur in clusters. The amount of clustering is controlled through the hysteresis constant H. Large values of H can cause large clustering and smaller values lead to smaller clusters. Levien’s algorithm is precisely defined as follows:
(
Y [n] = 01
if (X [n] + B T Y e [n] + HAT Y [n]) 0 else
(10)
21
= [a ; a ; :::; aN ]T , B = [b ; b ; :::; bN ]T , PNi ai = 1, PNi bi = 1, the output Y [n] = [y[n , 1]; y[n , 2]; :::; y[n , N ]]T and Y e[n] = [ye[n , 1]; ye[n , 2]; :::; ye[n , N ]]T , which follows from the equation Y e [n] = Y [n] , (X [n] + (B T Y e [n])). The error filter and hysteresis filter can where A
1
2
1
2
=0
=0
take on a wide range of values, including special cases such as Floyd-Steinberg [33], Jarvis [37] and Stucki [38] filter coefficients. Perturbation, in the form of noise, can also be added to these filter weights to create desirable results. B. Results and Evaluation Psychophysical experiments can be used to determine the optimum tactile halftone technique. The measure of goodness is defined as the level of effectiveness in feeling and understanding information through the manual exploration of the structures. As detailed in Fig. 13, the halftoning systems defined by the model transfer functions for AM, FM, and AM-FM hybrid methods are considered. As a first examination, a gray scale ramp test image is used as an original image and the corresponding halftones are generated by each method. These are shown in Fig. 16. A sample image having four gray levels is also processed by the three halftone techniques, Fig. 17. Upon observation of the generated halftone structures, we find certain differences that are particularly important in the case of tactile halftoning. First note that, as expected, the halftoned gray scale ramp produced by the AM halftoning method has a uniform distribution of dots, but varying dot size. Due to the regular spacing of dots, the (micro-capsule) growth of the dots is inhibited. That is, dots begin to merge once they are of a certain size. Thus no discrimination can be made once the dots begin to merge. Also, due to the asymmetry of the micro-capsule enhancement process, black dots and white dots merge at different thresholds. Accordingly, the tactile printing process places significant limitations on the use of AM halftoning. In the case of the blue noise, or FM, halftoning technique, the dots have similar size but are distributed at varying distance depending on the gray level. Once again, the minimum dot size imposed by the tactile printing process imposes a constraint. Since the minimum dot size for black
22
(a)
(b)
(c)
(d) Fig. 16. Test Image: (a) Original Ramp of gray scale; Test image halftoned using the HTS by (b)Amplitude Modulation (AM) halftoning (c) Frequency Modulation (FM) halftoning and (d) AM-FM hybrid halftoning
23
(a)
(b)
(c)
(d)
Fig. 17. Example using a test image (a) Original image (b) AM halftoned (c)FM halftoned (d)AM-FM halftoned
and white are different, both constraints cannot be met simultaneously under the FM structure. This results in merging of white dots and, as in the AM case, leads to a degradation in the quality of the resulting halftone. The green noise, or AM-FM hybrid, halftoning structure allows the minimum black and white dot size to co-exist, and the resulting halftone thus consists of dots within the experimentally defined limits. Also, since both spacing and dot distribution are stochastic processes, this method of generating tactile halftones is more robust to errors in either dot location or size. This results in halftones where the dots are not merged by the printing (micro-capsule paper enhancement) process. Thus, the produced halftones are tactilely distinct and more easily discriminated than either the AM or FM generated halftones. To test the validity of these visual observations, a set of experiments was conducted. The halftone methods discussed in previous sections were utilized to create halftones for the simple
24
(a)
(b)
(c)
(d)
(e)
Fig. 18. Test images halftoned by the AM-FM hybrid method for the edge and region identification experiments: (a) Hexagon (b) Triangle (c) Rectangle (d) Circle (e) image of rock with two gray levels.
geometric figures of hexagon, triangle, rectangle, circle and an image of rock. The structures, with halftoned patterns generated by the AM, FM, and AM-FM methods, constituted the set of stimuli for the experiments. Figure 18 shows the structures generated by the AM-FM hybrid method.
25
10 AM halftoning FM halftoning AM−FM hybrid
9
8
7
Time(secs)
6
5
4
3
2
1
0
1
2
3 Geometric Figures
4
5
Fig. 19. Timed identification of the geometric figures. The figures numbered 1 to 5 are Hexagon, Triangle, Rectangle, Circle, and an image of rock with two gray levels, respectively
The test patterns were enhanced as detailed earlier with the help of the TIE and were presented to blindfolded human subjects. The data were derived from experiments conducted on 8 subjects aged between 22 and 30 years of age, as before. The micro-capsule paper and the enhanced structures on it were shielded from the subject by blindfolds. In the first experiment, the halftones of the geometric figures produced by the three methods were numbered for our identification and presented to the subjects in random order. The subjects were asked to recognize the shape of the geometric figures by manual exploration of the figures on the tactile paper. The time taken by each subject to identify the figure, through manual exploration, was noted and recorded. This experiment was undertaken to understand how quickly the human subjects were able to recognize shapes based on each of the halftoning methods. As each halftoning method results in a different distribution of dots and subsequently different height/depth for the edge dots, this
26
70 AM halftoning FM halftoning AM−FM hybrid 60
Response to structures(%)
50
40
30
20
10
0
1
2
3 Geometric Figures
4
5
Fig. 20. Texture Discrimination.The figures numbered 1 to 5 are Hexagon, Triangle, Rectangle, Circle, and an image of rock with two gray levels, respectively
recording helps in defining which halftoning method produces better recognition. The average identification times for each method are shown in Fig. 19. The compiled results showed that, in all cases, the AM-FM hybrid halftoning method leads to the shortest identification time. Thus it can be noted that the more distinct structures produced by the AM-FM hybrid methods aid in identification. The second experiment tested how well a human subject is able to recognize the details in the textures of the structures. This was aimed at the texture discrimination rather than the shape identification, which was the goal in the first experiment. The human subjects were presented with the three halftones of the same geometrical figure and were asked to make a comparison of the quality of the structures on their surface. As before, the subjects were asked to manually explore the figures and report which of the three halftones presented to them, again in random order, was most distinct i.e. easily identifiable by touch. The experiment recorded which halftone method would give the most tactile perceivable
27
patterns on a surface for a given gray level. Figure 20 reports the responses recorded from this experiment. It was found that, in most cases, the AM halftoning is too weak to produce the necessary discrimination, most likely due to merging of dots. The FM halftoning, on the other hand, enables greater discrimination than the AM halftoning. The AM-FM hybrid halftoning method however, yields the greater discrimination. The psychophysical experiment results are thus in alignment with the analysis of the algorithms. The hybrid AM-FM halftoning, or green noise halftoning, is optimal for the generation of tactile output among the methods available in the literature. The performance of the hybrid method is likely due to its flexibility, in that it can be jointly optimized according to human tactile perception and the characteristics of the printing (microcapsule enhancement) process. Both the AM and FM methods produce suboptimal results, which can be attributed to identifiable shortcomings such as dot merging. Considerable work must still be undertaken to fully optimized and evaluate halftoning methods for tactile representations. For instance, it is not currently known what an appropriate number of tactile gray level representations is. It is unlikely that a user would be able to discriminate amongst, say, the 256 gray levels in an 8–bit image, but discrimination amongst a smaller number of halftoning patterns should be possible. Further studies are needed to clarify this issue. Additionally, the methods must be evaluated on other output medium and displays, such as the TIGER embossing printer. These issues are amongst those that will be the subject of further investigations. V. C ONCLUSIONS This paper introduced the concept of digital halftoning for tactile imaging. To develop appropriate tactile halftoning methods, the characteristics of the Tactile Image Enhancer (TIE) were studied in detail. This study led to an accurate model of the microcapsule paper enhancement process. Additionally, experiments were conducted to determine human tactile sensitivity and frequency response for the case of manual exploration of microcapsule paper. The TIE model and human tactile response were integrated into a general tactile halftoning procedure. This procedure includes as special cases the widely used AM and FM halftoning, as well as the more flexible
28
AM–FM hybrid green noise technique. Each of the methods were analyzed for their applicability in tactile halftoning and sample outputs were generated. Psychophysical experiments on timed identification and discrimination were conducted. The algorithm analysis and experiments indicate that the AM–FM hybrid method produces the most tactilely appropriate halftoning patterns. Further investigations will be conducted to more fully evaluate the developed methods to determine, for instance, the number of halftoning patterns that can be discriminated by users, and additional research will be conducted to optimize the procedures for other tactile display media. VI. ACKNOWLEDGMENT This work was supported by the National Science Foundation under the grants 9800175 and 9875658. R EFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]
S. Millar, Reading by Touch. Routledge, New York, USA, 1997. J. Bliss, “A relatively high-reslution reading aid for the blind,” IEEE Transactions on Man-Machine Systems, vol. 10, pp. 1–9, 1969. J.G.Linvill and J. Bliss, “A direct translation reading aid for the blind,” in Proceedings of the IEEE, vol. 54, pp. 40–51, 1966. J. Loomis, “Tactile pattern perception,” Perception, vol. 10, 1981. J. Craig and C. Sherrick, “Dynamic tactile displays,” in Tactual Perception: A Sourcebook (W. Schiff and E. Foulke, eds.), Cambridge University Press, 1982. J. Phillips and K. Johnson, “Tactile spatial resolution (i): Two-point discrimination, gap detection, grating resolution, and letter recognition,” Journal of Neurophysiology, vol. 46, pp. 1177–1191, 1981. J. Phillips and K. Johnson, “Tactile spatial resolution (ii): Neural representation of bars, edges and gratings in monkey primary afferents,” Journal of Neurophysiology, vol. 46, pp. 1192–1203, December 1981. J. Phillips and K. Johnson, “Tactile spatial resolution (iii):a continuum mechanics model of skin predicting mechanorecptor responses to bars, edges and gratings,” Journal of Neurophysiology, vol. 46, pp. 1204–1225, December 1981. M. Kurze, L. Reichert, and T. Strothotte, “Access to business graphics for blind people,” Proceedings of the RESNA ’94 Annual Conference, June 17-22 1994. R. Klatzky, S. Lederman, and V. Metzger, “Identifying objects by touch: An “expert system”,” Perception and Psychophysics, vol. 37, no. 4, 1985. R. Klatzky, S. Lederman, and C. Reed, “There’s more to touch than meets the eye: The salience of object attributes for haptics with and without vision,” Journal of Experimental Psychology: General, vol. 116, 1987. K. Kokjer, “The information capacity of the human fingertip,” IEEE Transactions on Systems, Man, and Cybernetics, vol. SMC-17, no. 1, 1987. T. P. Way and K. E. Barner, “Automatic visual to tactile translation, part I: Human factors, access methods and image manipulation,” IEEE Transactions on Rehabilitation Engineering, vol. 5, pp. 81–94, Mar. 1997. T. P. Way and K. E. Barner, “Automatic visual to tactile translation, part II: Evaluation of the tactile image creation system,” IEEE Transactions on Rehabilitation Engineering, vol. 5, pp. 95–105, Mar. 1997. S. Hernandez and K. Barner, “Joint region merging criteria for watershed-based image segm entation,” in 2000 International Conference on Image Processing, (Vancouver, BC, Canada), September 10-13 2000. S. Hernandez and K. E. Barner, “Color image segmentation using watersheds and joint region merging criteria,” IEEE Transactions on Image Processing, Aug. 2000. Submitted for publication. S. Hernandez and K. Barner, “Tactile imaging using watershed-based image segmentation,” in ASSETS 2000, (Arlington, Virginia, USA), November 13-15 2000. W. Schiff and E. Foulke, Tactual perception: a sourcebook. Cambridge University Press, 1982. R. A. Ulichney, Digital Halftoning. Cambridge, MA: MIT Press, 1987. D. L. Lau, Modern Digital Halftoning. PhD thesis, University of Delaware, Newark, Delaware, USA, May 1999. T. N. Pappas and D. L. Neuhoff, “Model-based halftoning,” in Proceedings of SPIE, Human Vision, Vision Processing and Digital Display II (B. E. Rogowitz, M. H. Brill, and J. P. Allebach, eds.), vol. 1453, pp. 244–255, June 1991.
29
[22] T. N. Pappas, “Model-based techniques for digital halftoning,” in Proceedings, International Conference on Image Processing (B. Werner, ed.), vol. II, pp. 26–30, IEEE, 1994. [23] J. Gardner, “Tactile graphics: An overview and resource guide,” Available from http://www.rit.edu/ easi/itd/itdv03n4/article2.html, 1999. [24] D. Pawluk, A Viscoelastic Model of the Human Fingerpad and a Holistic Model of the Human Touch. PhD thesis, Harvard University, Cambridge, Massachusetts, USA, May 1997. [25] D. Pawluk and R. Howe, “A holistic model of human touch,” in 5th Annual C.N.S. Meeting, 1996. [26] R. LaMotte and J. Whitehouse, “Tactile detection of a dot on a smooth surface: Peripheral neural events,” Journal of Neurophysiology, vol. 56, no. 4, 1986. [27] H. Schiffman, Sensation and Perception: An Itegrated Approach. John Wiley and Sons, New York, 1976. [28] T. Cornsweet, Visual Perception. New York: Academic press, 1971. [29] F. W. Campbell, “The transmission of spatial information through the visual system,” The Neurosciences Third Study Program, pp. 95–103, 1974. [30] H. R. Kang, Digital Color Halftoning. SPIE Optical Engineering Press and IEEE Press,USA, 1999. [31] F. W. Campbell, R. H. Carpenter, and J. Levinson, “Visibility of aperiodic patterns compared with that of sinusoidal gratings,” The Journal of Physiology, vol. 190, pp. 283–298, 1969. [32] F. W. Campbell, “The human eye as an optical filter,” Proceedings of the IEEE, vol. 56, no. 6, pp. 1009–1014, 1968. [33] R. W. Floyd and L. Steinberg, “An adaptive algorithm for spatial gray-scale,” Proceedings Society Information Display, vol. 17, pp. 75–78, Feb. 1976. [34] J. Sullivan, L. Ray, and R. Miller, “Design of minimum visual modulation halftone patterns,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 21, pp. 33–38, Jan. 1991. [35] M. Rodriguez, “Graphic arts perspective on digital halftoning,” in Proceedings of SPIE, Human Vision, Visual Processing, and Digital Display V (B. E. Rogowitz and J. P. Allebach, eds.), vol. 2179, pp. 144–149, Feb. 1994. [36] R. Levien, “Output dependant feedback in error diffusion halftoning,” in IS&T’s Eighth International Congress on Advances in Non-Impact Printing Technologies, (Williamsburg, Virginia, USA), pp. 280–282, October 25-30 1992. [37] J. F. Jarvis, C. N. Judice, and W. H. Ninke, “A survey of techniques for the display of continuous-tone pictures on bilevel displays,” Computer Graphics and Image Processing, vol. 5, pp. 13–40, 1976. [38] P. Stucki, “Mecca-a multiple-error correcting computation algorithm for bilevel image hardcopy reproduction,” Tech. Rep. RZ1060, IBM Research Laboratory, Zurich, Switzerland, 1981.
