Recognition of Freehand Sketches Using Mean Shift - Semantic Scholar
Recognition of Freehand Sketches Using Mean Shift Bo Yu State Key Laboratory of Software Novel Technology Nanjing University, 210093, China
[email protected]
much work on sketch recognition and some high-level applications based on it (e.g., [11], [7] and [9]). In [11], Sezgin, Stahovich and Davis presented a complete system which could recognize freehand sketches through three phases: approximation, beautification and basic recognition. Their main contribution is the use of both curvature and speed data to analyze strokes, which is based on the assumption that extrema in curvature and speed typically correspond to vertexes. A similar method was also adopted in [2]. This assumption may be valid for polylines, but it will not work on line-arc hybrid curves, because the vertexes are actually located where curvature changes considerably and do not need to be extrema themselves. Therefore, their method cannot handle smooth curves as gracefully as it does to polylines. Moreover, the dependence on speed data will inevitably introduce more noise and may imply that users should slow or stop their pens only at vertexes.
ABSTRACT Freehand sketching is a natural and powerful means of interpersonal communication. But to date, it still cannot be supported effectively by human-computer interface. In this paper, we propose a robust method for sketch recognition. It uses mean shift, a nonparametric technique which can delineate arbitrarily shaped clusters, as a pre-process to analyze the directioncurvature joint space and suppress the severe noise of sketched strokes. Furthermore, it combines the vertex detection and primitive shape approximation into a unified and incremental procedure which, by fully utilizing the visual features, can handle hybrid and smooth curves gracefully. Our method does not rely on any domain-specific knowledge, and therefore it can be easily integrated with other high-level applications.
Categories and Subject Descriptors
In [1], Arvo and Novins described a new sketching interface. Different from traditional ways of processing sketches, their method continuously morphs raw input strokes into ideal geometric shapes, which can apprise users the recognition progress and the appearance of the final shape. However, their method can handle only circles and rectangles drawn with one stroke. This is not adequate for a practical system.
H.5.2 [User Interfaces]: Input Devices and Strategies
General Terms Human Factors
Keywords Sketch Recognition, Mean Shift, Multimodal Interface, HCI
Igarashi, Matsuoka, Kawachiya and Tanaka [8] created an interactive beautification technique which could turn imprecise user sketches into a rigorous drawing just as drafted using a CAD system. It infers any possible geometric constraints such as perpendicularity, congruence, symmetry etc, generates multiple beautification candidates, and then the most plausible one is chosen. But the restriction that each line must be drawn with a separate stroke may bother users.
1. INTRODUCTION Freehand sketching is a very natural and powerful means of interpersonal communication, and people frequently use it to convey information efficiently. It is also a very promising technology of human-computer interface which will release users from a maze of menus, toolbars and many complicated commands. Sketch recognition can be broadly used in the fields of engineering or software design, multimodal user interface, multimedia database retrieval, and so on.
Fonseca and Jorge [3] proposed a method, based on fuzzy logic, for recognizing multi-stroke sketches. It processes one or more strokes collected within a timeout value as a whole, and classifies it into one of several predefined shape kinds. Their system cannot identify the constituent parts of a hybrid shape, and because it relies entirely on global geometric properties such as area and perimeter, it might be difficult to distinguish between similar shapes.
The research on sketch recognition and interpretation has a long history. In [10], Rubine described a trainable gesture recognizer which used a linear evaluation function to discriminate gestures. But it could handle only unistroke symbols which must be drawn in a pre-specified manner.
From the above reviews, we can find two main deficiencies of the existing methods for sketch recognition:
The Design Rationale Group of MIT AI Laboratory has done
Strict restrictions on the drawing manner which do not allow users to draw sketches naturally.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. IUI’03, January 12–15, 2003, Miami, Florida, USA. Copyright 2003 ACM 1-58113-586-6/03/0001…$5.00.
Lack of the ability to analyze hybrid or smooth curves and decompose them into primitive shapes. In this paper, we will describe a robust method based on mean shift to handle these problems. The paper is organized as follows.
204
The next section introduces the mean shift procedure briefly. Then, our method for sketch recognition is explained in detail including vertex detection, primitive shape approximation and the combination of them into a unified and incremental procedure. Finally, the experiments demonstrate the robustness and efficiency of our method.
ˆ f (x) ≡ ∇fˆ ( x) = ∇
=−
2. MEAN SHIFT PROCEDURE The mean shift procedure, which is based on the basic idea of associating each point in the feature space with similar ones, was originally proposed in 1975 by Fukunaga and Hostetler [6], but didn’t draw much attention until Cheng’s paper [3]. Cheng generalized the algorithm in three ways and pointed out that mean shift is a mode-seeking process on a surface constructed with a “shadow” kernel. In [4], Comaniciu and Meer proposed a robust approach based on mean shift, which could analyze a complex multimodal feature space and delineate arbitrarily shaped clusters. They proved the convergence of a recursive mean shift procedure for discrete data. In this section, we will briefly review the mean shift procedure and demonstrate how it can be applied to analyze the clusters in feature space.
x − xi ), K( ∑ h i =1
y j − xi 2 xi k ′ ∑ h i =1 , y j +1 = 2 n y − xi k ′ j ∑ h i =1
(1)
the
T
∫
positive constant which satisfies c ⋅ k ( x ) dx = 1 . Thus the profile of Gaussian kernel will be
x − xi k ∑ h i =1 n
(2) 2
.
1.
Initialize yi,1 with xi, i=1, …, n.
2.
Compute yi,j+1 according to (5) until convergence.
3.
Apply other customized analysis.
As these sequences evolve respectively, we can observe that they are moving to the regions where the majority of the points reside and form different trajectories toward various local peaks of the underlying density function. The final stationary points, to which these sequences converge, will delineate the clusters of the feature space automatically.
and the corresponding density estimator (1) can be rewritten as
c fˆ (x) = d nh
(5)
Let xi and {yi,j}j=1,2…, i=1,…,n, be the input sample points and the evolving sequences respectively. The general clustering algorithm based on mean shift is:
2
k ( x ) = exp( − x / 2) ,
j = 1, 2 ….
converge to zero, i.e. the estimate of the density gradient will be zero at that stationary point. Then we can easily obtain a mode of the feature space by checking whether the corresponding density value is a local maximum.
multivariate normal, , which is also adopted by our K (x) = (2π ) exp(−x x / 2) approach. According to the terminology used in [4], a kernel function can be represented in the form of its profile as 2 K ( x ) = c ⋅ k ( x ) , where k is the profile function and c is a −p/2
(4)
Comaniciu and Meer have proved that if the profile function k is convex and monotonically decreasing, the sequence {yj}j=1,2… will converge to a stationary point [4]. As a result, the mean shift vector, which can be rewritten as m (y ) = y j +1 − y j , will
n
is
n x − xi 2 ∑ x i k ′ 2 x − x i i =1 − x. h ⋅ n h x − xi 2 k ′ ∑ i =1 h
n
∫
kernel
Now we are able to define the mean shift procedure as
where K(z) is termed the kernel function and h is the smoothing parameter (also termed the bandwidth). The kernel function should satisfy K (z) ≥ 0 and K ( z ) dz = 1 . One commonly used
n 2c ⋅ k ′ d +2 ∑ n⋅h i =1
2
The last term of (4) is called the mean shift, i.e. the difference between the weighted mean and x, if we consider the derivative k ′ as a weight function. As k ′ is computed by referring to each sample point xi, it represents the global structure of the whole feature space.
Given a set of n points, {x1, …, xn}, in the d-dimensional space Rd, the multivariate kernel density estimate can be computed by
1 fˆ (x) = d nh
n x − xi 2c (x − x i ) ⋅ k ′ ⋅ d +2 ∑ h n⋅h i =1
(3)
Using the mean shift procedure to analyze clusters has the following two main advantages:
Since the d-dimensional feature space can be regarded, empirically, as a probability density function f(x) with the local maxima corresponding to the dense regions in that space, we can find the centers for each cluster by checking the vectors which satisfy the gradient ∇ f ( x ) = 0 and keeping only those local maxima. By taking the estimate of the density gradient as the gradient of the density estimate, we will have
The method is application independent and does not need any priori assumptions such as the number of clusters or the shape of the clustering regions. Hence it can be easily applied to solve clustering problems across multiple domains. It is efficient and adaptive because, as pointed out by Comaniciu and Meer [4], the mean shift vector always points to the direction of maximum increase in the density and the shift steps are large in low-density regions and small near local maxima. Furthermore, the computation cost is not high compared to other clustering methods. This is a desirable feature to real-time or interactive applications.
205
3. ANALYSIS OF SKETCHED STROKES We think that a practical approach for sketch recognition must have the following attributes: It must allow users to draw in a natural way just as how they do on paper.
(a) Input stroke
(b) Recognition result
It should produce a consistent recognition result, i.e. if a user wants to sketch an arc, no matter it is drawn with a single stroke or a hybrid stroke which contains other primitive shapes, the system should know it is an arc. Our method depends entirely on visual features and puts no particular restrictions on users. In order to handle hybrid and smooth curves properly, the recognition is invoked recursively. During each recognition cycle, our method first tries to remove noise and detect salient vertexes. Then the primitive shape approximation is applied on each stroke segment between two consecutive vertexes. Finally, evaluations are performed on the approximation results. The evaluation serves as a feedback from recognition result to vertex detection and tries to find the missing vertexes according to different strategies.
(c) Direction graph
(d) Curvature graph Figure 1: An example stroke with its direction and curvature graph.
3.1 Some Definitions In this paper, a stroke is defined as the coordinates of each point visited by the pen between a single pair of mouse-down and mouse-up events. Such events can be triggered by any pen-like input device as well as a mouse. Given a stroke or stroke segment (the portion between two consecutive vertexes), there are two kinds of obvious visual features which can guide recognition, i.e. direction and curvature. They can be computed as n+k
dn =
∑θ (i, i + 1)
i =n−k
2k + 1
n + k −1
and c n =
∑φ (d
i =n −k
i +1
(a) Feature area to a line
(b) Feature area to a point
− di )
P(n − k , n + k )
, (c) Feature area to an arc
where dn and cn represent the direction and curvature of the n-th stroke point respectively, θ (i, i + 1) is the angle from the i-th point to its successor, φ is a function which shifts its angle parameter to the range from − π to π , k is a small integer defining the neighborhood size around the n-th point, and P(n-k, n+k) stands for the path length between the (n-k)-th and (n+k)-th stroke point. The periodicity of θ should be noticed when calculating dn. We set k to 1 empirically as a tradeoff between stability and sensitivity for these local geometric features. Fig. 1 shows an example stroke together with its direction and curvature graphs.
Figure 2: Definitions of feature area, where the black dots are stroke points.
3.2 Direction-Curvature Joint Space Freehand sketches are extremely different from carefully drawn engineering diagrams, and we have to deal with considerable or even severe noise which can be seen from Fig. 1c and 1d. The noise mainly comes from the inherent freewill drawing manner of sketching and cannot be predicted or modeled. So we need a robust analysis tool which does not require any priori knowledge about the noise.
Another important concept is feature area. It is defined by a series of stroke points and another reference object which can be a line, an arc or a point. The hatched region in Fig. 2 illustrates this concept. The feature area to a line is computed as the sum area of all the small quadrangles formed by two consecutive stroke points and their foot points on that line. The feature area against an arc can be computed similarly. The feature area to a reference point equals to the sum area of all the small triangles formed by that reference point and two consecutive points on the stroke. Since area is a kind of global geometric attribute which is much more resistant to noise than other ones, we use feature area to judge the validity of recognition results and resolve ambiguities.
We regard the mean shift procedure as an efficient noise cleaner in addition to a clustering tool, because it iteratively shifts each sample point to the average in its neighborhood without destroying the global structure of the feature space. Furthermore, it does not need any priori assumptions. So the mean shift procedure is very suitable to serve as a pre-process for subsequent recognition. Given a stroke with n points, let xi and zi, i=1,…,n, be the 2dimensional input and output vectors containing both direction and curvature information. In order to adjust the smoothing parameters, hd and hc, for direction and curvature respectively, we revise (5), using the normal profile, as
206
y j +1
connection point between a line and an arc. As the mean shift procedure has removed much noise from the curvature data, we can apply this rule and find the vertexes reliably. The algorithm is as follows:
(d j − d i ) 2 (c − ci ) 2 ⋅ exp − j (6) ⋅ exp − 2 2 2hd 2hc d j +1 i =1 = = . 2 2 n c j 1 + − − ( d d ) ( c c ) j i j i ⋅ exp − exp − ∑ 2 2 2hd 2hc i =1 n
∑ x
i
The mean shift procedure is applied as follows: 1.
Initialize yi,1 with xi, i=1, …, n.
2.
Iteratively compute yi,j+1 according to (6) for each stroke point until convergence. The judging criterion for convergence is set to d i , j +1 − d i , j ≤ hd / 100 for
1.
Filter the stroke points and keep the ones with a higher absolute curvature than the mean value.
2.
Mark the i-th point as vertex, if its curvature is either a local maximum or a local minimum and satisfies ci − ci −1 + ci − ci +1 ≥ 2hc , where hc is the smoothing parameter for curvature used in the mean shift procedure.
3.
direction, and c i , j +1 − c i , j ≤ hc / 100 for curvature. 3.
Assign each convergent vector to zi, i=1, …, n.
3.4 Primitive Shape Approximation Our method can recognize the following kinds of primitive shapes: line segments, arcs, circles and ellipses. We do not use spline curves for approximation because we think spline curves are of little use when high-level applications perform symbol recognition, graphic match or other similar operations. Furthermore, it is meaningless to generate a very accurate approximation for sketched strokes which are inherently imprecise.
The smoothing parameters are selected as n −1
hd =
∑φ i =1
2
(d i +1 − d i ) n −1
n −1
and h = c
∑ (c i =1
i +1
− ci ) 2
n −1
Merge two consecutive vertexes, if the path length between them is less than a threshold.
,
which are intended to model the difference between consecutive stroke points. Although direction information has been taken into account when calculating curvature and will not be used in vertex detection, it is still necessary to include direction data and form a joint feature space for analysis. First, direction, which will not be influenced by drawing speed, is more stable then curvature and tends to provide cores for clustering. Second, the direction constraint prevents curvature data from shifting too far away and helps to keep the original features when handling line-arc hybrid strokes. Fig. 3 illustrates the analysis result of the same stroke showed in Fig. 1, and we can see how efficiently the mean shift procedure removes the noise and meanwhile keeps the original features of the curvature curve.
3.4.1 Line Segments Two kinds of clues may suggest an input stroke or stroke segment to be a line. One is from the direction graph after the mean shift procedure, i.e. the direction curve can be fitted with a horizontal line. The other is whether the actual coordinates of the stroke points can be fitted by a line. First, we will compute the least squares best fit line for either the direction curve or the coordinates of stroke points, and check whether the number of deviation points is small using a relatively loose threshold. Then, if succeeds, we hypothesize a line segment from the first stroke point to the last one and judge its validity through the corresponding feature area to that candidate line. Since we give a higher priority to line segments than arcs, if the ratio between the feature area and the candidate line’s standard area, which equals to the product of its length and the input stroke’s width, is less than a strict threshold (1.0 is chosen in our implementation), the system will accept the hypothesis at once and save that line segment. Otherwise, the final decision will be delayed after the same stroke has been tested by the arc recognizer.
(a) Direction graph
We name the procedure of checking the ratio between feature and standard area “feature-area verification”. As a judging criterion based on global geometric features, it serves as a complementary constraint against those local criteria based on direction and curvature information, and makes the recognition more reliable.
(b) Curvature graph
3.4.2 Arcs
Figure 3: The direction and curvature graph after the mean shift procedure, where the vertexes are marked with black squares.
The most salient character of an arc is that the direction graph can be roughly approximated with a non-horizontal line, as can be seen in Fig. 1c. Based on this observation, the algorithm for arc recognition is devised as follows:
3.3 Vertex Detection Based on Curvature Intuitively speaking, the vertexes of a stroke are always located where curvature changes considerably, for example, the
207
1.
Check whether the direction curve can be approximated with a line, and abort if fails.
2.
Find the perpendicular bisector, denoted as M, of the first
and last stroke points, calculate the intersection point A between line M and the stroke path, and then find the circle on which the first and last stroke points as well as point A reside. The circle’s center is taken as the center of the candidate arc. 3.
4.
Take the mean distance from each stroke point to the center as the radius. Although this step sometimes disconnects the arc with its neighbor shapes visually, it can reduce the approximation error.
(a)
Figure 4: (a) A roughly sketched ellipse. (b) Its direction graph.
Compute the ratio between the corresponding feature area against the candidate arc and that arc’s standard area. Check whether the ratio is less than a predefined threshold, and abort if fails.
3.5 Evaluation of Approximation Result On the one hand, the mean shift procedure analyzes only the direction and curvature data, and sometimes it may be difficult to discern the vertexes which are fused with their neighborhood in the joint feature space, even though they are visually obvious. On the other hand, for the purpose of not interpreting the stroke as a number of small fragments and making the recognition more resistant to noise, especially when handling smooth curves, the thresholds related to vertex detection are loosened to detect only the salient ones. Therefore, we must evaluate the approximation result and try to find the possible missing vertexes.
Note that the smooth ascent or descent in direction will be destroyed by the mean shift procedure, as in Fig. 3a. So the above algorithm should be applied on the original direction data. As a line can be viewed as an arc with infinite radius, there is an inherent ambiguity in distinguish between lines and arcs. How to make a reasonable choice? Since there is no domain knowledge available here, we have to resort to feature area. During the approximation process, there are two ratios recorded. One is between the feature area against the candidate line and that line’s standard area, the other is between the feature area against the candidate arc and that arc’s standard area. Our system will compare them and select the hypothesis with a smaller ratio value.
3.5.1 Handling Approximation Failure Approximation failure can be classified according to whether there are vertexes detected. The complete approximation failure is defined as that the whole input stroke, which has no vertex detected, fails to be approximated by any of the primitive shapes. This situation happens mainly because the stroke has a smooth hybrid shape, as in Fig. 5. In such case, the curvature curve is relatively flat (Fig. 5b), and the mean shift procedure makes things even worse (Fig. 5c). As a result, no vertexes can be found and consequently the approximation fails.
3.4.3 Circles and Ellipses According to the drawing manner of most people, circles and ellipses are drawn independently and with one stroke, which means they are not likely to appear in a hybrid stroke. Therefore given an input stroke, our system will check first whether it is a circle or ellipse. If the approximation fails, the normal recognition procedure will be performed then. Similar to arcs, the direction graph of a circle or ellipse can also be roughly approximated with a line. Moreover, its projection on the direction axis covers about a range of 2π . The algorithm for circle recognition is: 1.
Check whether the direction graph of the input stroke can be approximated with the line mentioned above, and abort if fails.
2.
Generate a candidate circle. The center of the stroke’s bounding box is taken as the circle’s center, and the mean distance from each stroke point to the center is taken as the radius.
3.
Calculate the stroke’s feature area against the center and check whether it can match the actual area of the candidate circle.
(b)
(a) Recognition result (two arcs) after applying the rule
(b) Original curvature graph
Ellipses can be handled similarly with one more constraint, i.e. the area should be nearly symmetrical about the major and minor axes respectively. Fig. 4 shows an example.
(c) Curvature graph after the mean shift procedure
Again we have to face the ambiguity between circles and ellipses. We adopt two rules to solve this problem. One is if the major and minor axes of the candidate ellipse have nearly the same length, it should be a circle. The other is based on the comparison of ratios between feature and standard areas, which is analogous to what is used to distinguish between lines and arcs.
Figure 5: Handling complete approximation failure. In order to solve this problem, we introduce the following heuristic rule: select the point, which is farthest from the line passing through the first and last stroke points, as the dividing
208
point, then split the original stroke into two sub-strokes and process them recursively.
3.6 Post-Process After the whole stroke has been recognized, a simple post-process will be applied to merge primitive shapes within one stroke, if they satisfy either of the following conditions:
Because the dividing point usually corresponds to a tangent point (as in Fig. 5a), this rule can decompose a smooth curve efficiently and will not generate the disturbing fragments.
Two connected line segments have nearly the same angle. Two connected arcs have nearly the same center and radius.
The other kind of approximation failure is called partial approximation failure, which is defined as that some segment of the input stroke cannot be approximated by any of the primitive shapes. From the definition, we can know that some salient vertexes have been detected, which implies the feature of the missing vertexes may be suppressed by them. For example, as the smooth parameter hc is computed as a global measurement, it is probably too large for some vertexes with small curvature variance, and consequently those vertexes will be fused with their neighbors. Fig. 6 shows an example.
4. EXPERIMENTS AND EVALUATIONS
Figure 8: Examples of hybrid shapes with explicit vertexes.
Figure 6: An example of partial approximation failure. For the above reason, we extract the failed stroke segment and process it as a new stroke recursively, which means that the stroke points will be re-analyzed using more specific parameters and the relatively vague vertexes will be more obvious this time. From this meaning, the vertexes are detected in an incremental way,
Figure 9: Examples of smooth hybrid shapes.
(a) Improper approximation
(b) Revised approximation
from salient ones to obscure ones.
We have conducted a user study to measure the efficiency of our approach. Three students, one from the Computer Science Department and two from other departments, participated in the study.
3.5.2 Handling Approximation Error
Firstly, they all thought our system is natural to use, and they could draw freely to obtain the desired shapes.
Figure 7: Handling approximation error.
An approximation error means an input stroke or stroke segment is not well fitted by the primitive shape. A typical example is shown in Fig. 7. This problem mainly comes from the inability of feature-area verification to distinguish between shapes with similar area.
The sample strokes for the user study were divided into three classes: primitive shapes, hybrid shapes with explicit vertexes (as in Fig. 8) and smooth hybrid shapes (as in Fig. 9). The experiment results are listed in Table 1 and 2, where the “correctness” means the recognition result exactly matches what is drawn by the user and the “validity” means the main portion and key structure are recognized correctly except for some small disturbances. Both correctness and validity were judged from the participator’s point of view.
To solve this problem, each approximation result will be checked whether there are some portions of the stroke deviating far from the recognized shape, and additional vertexes, especially intersection points, will be inserted if necessary. Then the newgenerated stroke segments will be recognized to replace the original approximation result.
Table 1: Correctness rate for primitive shapes.
Correctness Rate
209
Lines
Arcs
Circles
Ellipses
98%
97%
98%
96%
sketched strokes. Furthermore, it combines the vertex detection and primitive shape approximation into a unified and incremental procedure which, by fully utilizing the visual features, can handle hybrid and smooth curves gracefully. Since our method mainly utilizes low-level geometric features and does not rely on any domain-specific knowledge, it can be integrated easily with other high-level applications and will provide a natural and powerful means of human-computer interaction.
Table 2: Correctness and validity rate for hybrid shapes. Hybrid Shapes with Explicit Vertexes
Smooth Hybrid Shapes
Correctness Rate
84%
78%*
Validity Rate
94%
94%
From Table 1, we can see that our approach can recognize the four types of primitive shapes reliably. The correctness rate for ellipses is a little lower because sometimes a roughly sketched ellipse will be interpreted as a series of lines and arcs. The correctness rate of smooth hybrid shapes is marked with an asterisk because when doing the user study, there was much dispute about to which extent a recognition result could be considered as matching the intention of the user. But from the high validity rate for both kinds of hybrid shapes, we believe that our approach can produce a good approximation for any complex input stroke. In general, our approach is more robust than other existing methods for sketch recognition.
6. REFERENCES
In order to measure the computation cost, we gathered the average number of iteration for each stroke point when analyzing the direction-curvature joint space using the mean shift procedure. The mean value is 7.3 per point, which has little to do with the number of stroke points but is much related with their distribution in the feature space. As most strokes contain no more than 150 points, our method can be viewed as computation-efficient.
[4] Comaniciu, D., and Meer, P. Mean Shift: A Robust
There were also some limitations exposed in the user study. For example, two short connected lines with a small difference angle will be recognized as an arc, if they are drawn in one stoke which contains other more salient vertexes. In addition, we feel that treating wave lines as one kind of primitive shape is necessary, because our method encountered some difficulties when processing them in the user study.
[6] Fukunaga, K., and Hostetler, L.D. The Estimation of the
We are now building a query system for CAD drawings which allows users to sketch desired symbols and then fetches the corresponding drawings from a central database. It adopts the input method based on this paper and measures the similarities between input and model graphs through elastic graph matching.
[8] Igarashi, T., Matsuoka, S., Kawachiya, S. and Tanaka, H.
[1] Arvo, J., and Novins, K. Fluid Sketches: Continuous Recognition and Morphing of Simple Hand-Drawn Shapes. Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, 2000.
[2] Calhoun, C., Stahovich, T.F., Kurtoglu, T., and Kara, L.B. Recognizing Multi-Stroke Symbols. AAAI Spring Symposium on Sketch Understanding, 2002.
[3] Cheng, Y. Mean Shift, Mode Seeking, and Clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, No. 8, 1995. Approach Toward Feature Space Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 5, 2002.
[5] Fonseca, M.J., and Jorge, J.A. Using Fuzzy Logic to Recognize Geometric Shapes Interactively. Proceedings of the 9th International Conference on Fuzzy Systems, 2000. Gradient of a Density Function. IEEE Transactions on Information Theory, Vol. 21, 1975, 32-40.
[7] Hammond, T., and Davis, R. Tahuti: A Sketch Recognition System for UML Class Diagrams. AAAI Spring Symposium on Sketch Understanding, 2002. Interactive Beautification: A Technique for Rapid Geometric Design. UIST'97, 105-114.
[9] Oltmans,
M., and Davis, R. Naturally Conveyed Explanations of Device Behavior. Proceedings of PUI-2001, November 2001.
5. CONCLUSION
[10] Rubine, D. Specifying gestures by example. Computer
In this paper, we have proposed a robust method for freehand sketch recognition. It uses the mean shift procedure, an efficient tool which can delineate arbitrarily shaped clusters in a complex multimodal feature space, as a pre-process to analyze the direction-curvature joint space and suppress the severe noise of
Graphics (SIGGRAPH'91 Proceedings), 329-337.
[11] Sezgin, T.M., Stahovich, T., and Davis, R. Sketch Based Interfaces: Early Processing for Sketch Understanding. Proceedings of PUI-2001, November 2001.
210
[email protected]
much work on sketch recognition and some high-level applications based on it (e.g., [11], [7] and [9]). In [11], Sezgin, Stahovich and Davis presented a complete system which could recognize freehand sketches through three phases: approximation, beautification and basic recognition. Their main contribution is the use of both curvature and speed data to analyze strokes, which is based on the assumption that extrema in curvature and speed typically correspond to vertexes. A similar method was also adopted in [2]. This assumption may be valid for polylines, but it will not work on line-arc hybrid curves, because the vertexes are actually located where curvature changes considerably and do not need to be extrema themselves. Therefore, their method cannot handle smooth curves as gracefully as it does to polylines. Moreover, the dependence on speed data will inevitably introduce more noise and may imply that users should slow or stop their pens only at vertexes.
ABSTRACT Freehand sketching is a natural and powerful means of interpersonal communication. But to date, it still cannot be supported effectively by human-computer interface. In this paper, we propose a robust method for sketch recognition. It uses mean shift, a nonparametric technique which can delineate arbitrarily shaped clusters, as a pre-process to analyze the directioncurvature joint space and suppress the severe noise of sketched strokes. Furthermore, it combines the vertex detection and primitive shape approximation into a unified and incremental procedure which, by fully utilizing the visual features, can handle hybrid and smooth curves gracefully. Our method does not rely on any domain-specific knowledge, and therefore it can be easily integrated with other high-level applications.
Categories and Subject Descriptors
In [1], Arvo and Novins described a new sketching interface. Different from traditional ways of processing sketches, their method continuously morphs raw input strokes into ideal geometric shapes, which can apprise users the recognition progress and the appearance of the final shape. However, their method can handle only circles and rectangles drawn with one stroke. This is not adequate for a practical system.
H.5.2 [User Interfaces]: Input Devices and Strategies
General Terms Human Factors
Keywords Sketch Recognition, Mean Shift, Multimodal Interface, HCI
Igarashi, Matsuoka, Kawachiya and Tanaka [8] created an interactive beautification technique which could turn imprecise user sketches into a rigorous drawing just as drafted using a CAD system. It infers any possible geometric constraints such as perpendicularity, congruence, symmetry etc, generates multiple beautification candidates, and then the most plausible one is chosen. But the restriction that each line must be drawn with a separate stroke may bother users.
1. INTRODUCTION Freehand sketching is a very natural and powerful means of interpersonal communication, and people frequently use it to convey information efficiently. It is also a very promising technology of human-computer interface which will release users from a maze of menus, toolbars and many complicated commands. Sketch recognition can be broadly used in the fields of engineering or software design, multimodal user interface, multimedia database retrieval, and so on.
Fonseca and Jorge [3] proposed a method, based on fuzzy logic, for recognizing multi-stroke sketches. It processes one or more strokes collected within a timeout value as a whole, and classifies it into one of several predefined shape kinds. Their system cannot identify the constituent parts of a hybrid shape, and because it relies entirely on global geometric properties such as area and perimeter, it might be difficult to distinguish between similar shapes.
The research on sketch recognition and interpretation has a long history. In [10], Rubine described a trainable gesture recognizer which used a linear evaluation function to discriminate gestures. But it could handle only unistroke symbols which must be drawn in a pre-specified manner.
From the above reviews, we can find two main deficiencies of the existing methods for sketch recognition:
The Design Rationale Group of MIT AI Laboratory has done
Strict restrictions on the drawing manner which do not allow users to draw sketches naturally.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. IUI’03, January 12–15, 2003, Miami, Florida, USA. Copyright 2003 ACM 1-58113-586-6/03/0001…$5.00.
Lack of the ability to analyze hybrid or smooth curves and decompose them into primitive shapes. In this paper, we will describe a robust method based on mean shift to handle these problems. The paper is organized as follows.
204
The next section introduces the mean shift procedure briefly. Then, our method for sketch recognition is explained in detail including vertex detection, primitive shape approximation and the combination of them into a unified and incremental procedure. Finally, the experiments demonstrate the robustness and efficiency of our method.
ˆ f (x) ≡ ∇fˆ ( x) = ∇
=−
2. MEAN SHIFT PROCEDURE The mean shift procedure, which is based on the basic idea of associating each point in the feature space with similar ones, was originally proposed in 1975 by Fukunaga and Hostetler [6], but didn’t draw much attention until Cheng’s paper [3]. Cheng generalized the algorithm in three ways and pointed out that mean shift is a mode-seeking process on a surface constructed with a “shadow” kernel. In [4], Comaniciu and Meer proposed a robust approach based on mean shift, which could analyze a complex multimodal feature space and delineate arbitrarily shaped clusters. They proved the convergence of a recursive mean shift procedure for discrete data. In this section, we will briefly review the mean shift procedure and demonstrate how it can be applied to analyze the clusters in feature space.
x − xi ), K( ∑ h i =1
y j − xi 2 xi k ′ ∑ h i =1 , y j +1 = 2 n y − xi k ′ j ∑ h i =1
(1)
the
T
∫
positive constant which satisfies c ⋅ k ( x ) dx = 1 . Thus the profile of Gaussian kernel will be
x − xi k ∑ h i =1 n
(2) 2
.
1.
Initialize yi,1 with xi, i=1, …, n.
2.
Compute yi,j+1 according to (5) until convergence.
3.
Apply other customized analysis.
As these sequences evolve respectively, we can observe that they are moving to the regions where the majority of the points reside and form different trajectories toward various local peaks of the underlying density function. The final stationary points, to which these sequences converge, will delineate the clusters of the feature space automatically.
and the corresponding density estimator (1) can be rewritten as
c fˆ (x) = d nh
(5)
Let xi and {yi,j}j=1,2…, i=1,…,n, be the input sample points and the evolving sequences respectively. The general clustering algorithm based on mean shift is:
2
k ( x ) = exp( − x / 2) ,
j = 1, 2 ….
converge to zero, i.e. the estimate of the density gradient will be zero at that stationary point. Then we can easily obtain a mode of the feature space by checking whether the corresponding density value is a local maximum.
multivariate normal, , which is also adopted by our K (x) = (2π ) exp(−x x / 2) approach. According to the terminology used in [4], a kernel function can be represented in the form of its profile as 2 K ( x ) = c ⋅ k ( x ) , where k is the profile function and c is a −p/2
(4)
Comaniciu and Meer have proved that if the profile function k is convex and monotonically decreasing, the sequence {yj}j=1,2… will converge to a stationary point [4]. As a result, the mean shift vector, which can be rewritten as m (y ) = y j +1 − y j , will
n
is
n x − xi 2 ∑ x i k ′ 2 x − x i i =1 − x. h ⋅ n h x − xi 2 k ′ ∑ i =1 h
n
∫
kernel
Now we are able to define the mean shift procedure as
where K(z) is termed the kernel function and h is the smoothing parameter (also termed the bandwidth). The kernel function should satisfy K (z) ≥ 0 and K ( z ) dz = 1 . One commonly used
n 2c ⋅ k ′ d +2 ∑ n⋅h i =1
2
The last term of (4) is called the mean shift, i.e. the difference between the weighted mean and x, if we consider the derivative k ′ as a weight function. As k ′ is computed by referring to each sample point xi, it represents the global structure of the whole feature space.
Given a set of n points, {x1, …, xn}, in the d-dimensional space Rd, the multivariate kernel density estimate can be computed by
1 fˆ (x) = d nh
n x − xi 2c (x − x i ) ⋅ k ′ ⋅ d +2 ∑ h n⋅h i =1
(3)
Using the mean shift procedure to analyze clusters has the following two main advantages:
Since the d-dimensional feature space can be regarded, empirically, as a probability density function f(x) with the local maxima corresponding to the dense regions in that space, we can find the centers for each cluster by checking the vectors which satisfy the gradient ∇ f ( x ) = 0 and keeping only those local maxima. By taking the estimate of the density gradient as the gradient of the density estimate, we will have
The method is application independent and does not need any priori assumptions such as the number of clusters or the shape of the clustering regions. Hence it can be easily applied to solve clustering problems across multiple domains. It is efficient and adaptive because, as pointed out by Comaniciu and Meer [4], the mean shift vector always points to the direction of maximum increase in the density and the shift steps are large in low-density regions and small near local maxima. Furthermore, the computation cost is not high compared to other clustering methods. This is a desirable feature to real-time or interactive applications.
205
3. ANALYSIS OF SKETCHED STROKES We think that a practical approach for sketch recognition must have the following attributes: It must allow users to draw in a natural way just as how they do on paper.
(a) Input stroke
(b) Recognition result
It should produce a consistent recognition result, i.e. if a user wants to sketch an arc, no matter it is drawn with a single stroke or a hybrid stroke which contains other primitive shapes, the system should know it is an arc. Our method depends entirely on visual features and puts no particular restrictions on users. In order to handle hybrid and smooth curves properly, the recognition is invoked recursively. During each recognition cycle, our method first tries to remove noise and detect salient vertexes. Then the primitive shape approximation is applied on each stroke segment between two consecutive vertexes. Finally, evaluations are performed on the approximation results. The evaluation serves as a feedback from recognition result to vertex detection and tries to find the missing vertexes according to different strategies.
(c) Direction graph
(d) Curvature graph Figure 1: An example stroke with its direction and curvature graph.
3.1 Some Definitions In this paper, a stroke is defined as the coordinates of each point visited by the pen between a single pair of mouse-down and mouse-up events. Such events can be triggered by any pen-like input device as well as a mouse. Given a stroke or stroke segment (the portion between two consecutive vertexes), there are two kinds of obvious visual features which can guide recognition, i.e. direction and curvature. They can be computed as n+k
dn =
∑θ (i, i + 1)
i =n−k
2k + 1
n + k −1
and c n =
∑φ (d
i =n −k
i +1
(a) Feature area to a line
(b) Feature area to a point
− di )
P(n − k , n + k )
, (c) Feature area to an arc
where dn and cn represent the direction and curvature of the n-th stroke point respectively, θ (i, i + 1) is the angle from the i-th point to its successor, φ is a function which shifts its angle parameter to the range from − π to π , k is a small integer defining the neighborhood size around the n-th point, and P(n-k, n+k) stands for the path length between the (n-k)-th and (n+k)-th stroke point. The periodicity of θ should be noticed when calculating dn. We set k to 1 empirically as a tradeoff between stability and sensitivity for these local geometric features. Fig. 1 shows an example stroke together with its direction and curvature graphs.
Figure 2: Definitions of feature area, where the black dots are stroke points.
3.2 Direction-Curvature Joint Space Freehand sketches are extremely different from carefully drawn engineering diagrams, and we have to deal with considerable or even severe noise which can be seen from Fig. 1c and 1d. The noise mainly comes from the inherent freewill drawing manner of sketching and cannot be predicted or modeled. So we need a robust analysis tool which does not require any priori knowledge about the noise.
Another important concept is feature area. It is defined by a series of stroke points and another reference object which can be a line, an arc or a point. The hatched region in Fig. 2 illustrates this concept. The feature area to a line is computed as the sum area of all the small quadrangles formed by two consecutive stroke points and their foot points on that line. The feature area against an arc can be computed similarly. The feature area to a reference point equals to the sum area of all the small triangles formed by that reference point and two consecutive points on the stroke. Since area is a kind of global geometric attribute which is much more resistant to noise than other ones, we use feature area to judge the validity of recognition results and resolve ambiguities.
We regard the mean shift procedure as an efficient noise cleaner in addition to a clustering tool, because it iteratively shifts each sample point to the average in its neighborhood without destroying the global structure of the feature space. Furthermore, it does not need any priori assumptions. So the mean shift procedure is very suitable to serve as a pre-process for subsequent recognition. Given a stroke with n points, let xi and zi, i=1,…,n, be the 2dimensional input and output vectors containing both direction and curvature information. In order to adjust the smoothing parameters, hd and hc, for direction and curvature respectively, we revise (5), using the normal profile, as
206
y j +1
connection point between a line and an arc. As the mean shift procedure has removed much noise from the curvature data, we can apply this rule and find the vertexes reliably. The algorithm is as follows:
(d j − d i ) 2 (c − ci ) 2 ⋅ exp − j (6) ⋅ exp − 2 2 2hd 2hc d j +1 i =1 = = . 2 2 n c j 1 + − − ( d d ) ( c c ) j i j i ⋅ exp − exp − ∑ 2 2 2hd 2hc i =1 n
∑ x
i
The mean shift procedure is applied as follows: 1.
Initialize yi,1 with xi, i=1, …, n.
2.
Iteratively compute yi,j+1 according to (6) for each stroke point until convergence. The judging criterion for convergence is set to d i , j +1 − d i , j ≤ hd / 100 for
1.
Filter the stroke points and keep the ones with a higher absolute curvature than the mean value.
2.
Mark the i-th point as vertex, if its curvature is either a local maximum or a local minimum and satisfies ci − ci −1 + ci − ci +1 ≥ 2hc , where hc is the smoothing parameter for curvature used in the mean shift procedure.
3.
direction, and c i , j +1 − c i , j ≤ hc / 100 for curvature. 3.
Assign each convergent vector to zi, i=1, …, n.
3.4 Primitive Shape Approximation Our method can recognize the following kinds of primitive shapes: line segments, arcs, circles and ellipses. We do not use spline curves for approximation because we think spline curves are of little use when high-level applications perform symbol recognition, graphic match or other similar operations. Furthermore, it is meaningless to generate a very accurate approximation for sketched strokes which are inherently imprecise.
The smoothing parameters are selected as n −1
hd =
∑φ i =1
2
(d i +1 − d i ) n −1
n −1
and h = c
∑ (c i =1
i +1
− ci ) 2
n −1
Merge two consecutive vertexes, if the path length between them is less than a threshold.
,
which are intended to model the difference between consecutive stroke points. Although direction information has been taken into account when calculating curvature and will not be used in vertex detection, it is still necessary to include direction data and form a joint feature space for analysis. First, direction, which will not be influenced by drawing speed, is more stable then curvature and tends to provide cores for clustering. Second, the direction constraint prevents curvature data from shifting too far away and helps to keep the original features when handling line-arc hybrid strokes. Fig. 3 illustrates the analysis result of the same stroke showed in Fig. 1, and we can see how efficiently the mean shift procedure removes the noise and meanwhile keeps the original features of the curvature curve.
3.4.1 Line Segments Two kinds of clues may suggest an input stroke or stroke segment to be a line. One is from the direction graph after the mean shift procedure, i.e. the direction curve can be fitted with a horizontal line. The other is whether the actual coordinates of the stroke points can be fitted by a line. First, we will compute the least squares best fit line for either the direction curve or the coordinates of stroke points, and check whether the number of deviation points is small using a relatively loose threshold. Then, if succeeds, we hypothesize a line segment from the first stroke point to the last one and judge its validity through the corresponding feature area to that candidate line. Since we give a higher priority to line segments than arcs, if the ratio between the feature area and the candidate line’s standard area, which equals to the product of its length and the input stroke’s width, is less than a strict threshold (1.0 is chosen in our implementation), the system will accept the hypothesis at once and save that line segment. Otherwise, the final decision will be delayed after the same stroke has been tested by the arc recognizer.
(a) Direction graph
We name the procedure of checking the ratio between feature and standard area “feature-area verification”. As a judging criterion based on global geometric features, it serves as a complementary constraint against those local criteria based on direction and curvature information, and makes the recognition more reliable.
(b) Curvature graph
3.4.2 Arcs
Figure 3: The direction and curvature graph after the mean shift procedure, where the vertexes are marked with black squares.
The most salient character of an arc is that the direction graph can be roughly approximated with a non-horizontal line, as can be seen in Fig. 1c. Based on this observation, the algorithm for arc recognition is devised as follows:
3.3 Vertex Detection Based on Curvature Intuitively speaking, the vertexes of a stroke are always located where curvature changes considerably, for example, the
207
1.
Check whether the direction curve can be approximated with a line, and abort if fails.
2.
Find the perpendicular bisector, denoted as M, of the first
and last stroke points, calculate the intersection point A between line M and the stroke path, and then find the circle on which the first and last stroke points as well as point A reside. The circle’s center is taken as the center of the candidate arc. 3.
4.
Take the mean distance from each stroke point to the center as the radius. Although this step sometimes disconnects the arc with its neighbor shapes visually, it can reduce the approximation error.
(a)
Figure 4: (a) A roughly sketched ellipse. (b) Its direction graph.
Compute the ratio between the corresponding feature area against the candidate arc and that arc’s standard area. Check whether the ratio is less than a predefined threshold, and abort if fails.
3.5 Evaluation of Approximation Result On the one hand, the mean shift procedure analyzes only the direction and curvature data, and sometimes it may be difficult to discern the vertexes which are fused with their neighborhood in the joint feature space, even though they are visually obvious. On the other hand, for the purpose of not interpreting the stroke as a number of small fragments and making the recognition more resistant to noise, especially when handling smooth curves, the thresholds related to vertex detection are loosened to detect only the salient ones. Therefore, we must evaluate the approximation result and try to find the possible missing vertexes.
Note that the smooth ascent or descent in direction will be destroyed by the mean shift procedure, as in Fig. 3a. So the above algorithm should be applied on the original direction data. As a line can be viewed as an arc with infinite radius, there is an inherent ambiguity in distinguish between lines and arcs. How to make a reasonable choice? Since there is no domain knowledge available here, we have to resort to feature area. During the approximation process, there are two ratios recorded. One is between the feature area against the candidate line and that line’s standard area, the other is between the feature area against the candidate arc and that arc’s standard area. Our system will compare them and select the hypothesis with a smaller ratio value.
3.5.1 Handling Approximation Failure Approximation failure can be classified according to whether there are vertexes detected. The complete approximation failure is defined as that the whole input stroke, which has no vertex detected, fails to be approximated by any of the primitive shapes. This situation happens mainly because the stroke has a smooth hybrid shape, as in Fig. 5. In such case, the curvature curve is relatively flat (Fig. 5b), and the mean shift procedure makes things even worse (Fig. 5c). As a result, no vertexes can be found and consequently the approximation fails.
3.4.3 Circles and Ellipses According to the drawing manner of most people, circles and ellipses are drawn independently and with one stroke, which means they are not likely to appear in a hybrid stroke. Therefore given an input stroke, our system will check first whether it is a circle or ellipse. If the approximation fails, the normal recognition procedure will be performed then. Similar to arcs, the direction graph of a circle or ellipse can also be roughly approximated with a line. Moreover, its projection on the direction axis covers about a range of 2π . The algorithm for circle recognition is: 1.
Check whether the direction graph of the input stroke can be approximated with the line mentioned above, and abort if fails.
2.
Generate a candidate circle. The center of the stroke’s bounding box is taken as the circle’s center, and the mean distance from each stroke point to the center is taken as the radius.
3.
Calculate the stroke’s feature area against the center and check whether it can match the actual area of the candidate circle.
(b)
(a) Recognition result (two arcs) after applying the rule
(b) Original curvature graph
Ellipses can be handled similarly with one more constraint, i.e. the area should be nearly symmetrical about the major and minor axes respectively. Fig. 4 shows an example.
(c) Curvature graph after the mean shift procedure
Again we have to face the ambiguity between circles and ellipses. We adopt two rules to solve this problem. One is if the major and minor axes of the candidate ellipse have nearly the same length, it should be a circle. The other is based on the comparison of ratios between feature and standard areas, which is analogous to what is used to distinguish between lines and arcs.
Figure 5: Handling complete approximation failure. In order to solve this problem, we introduce the following heuristic rule: select the point, which is farthest from the line passing through the first and last stroke points, as the dividing
208
point, then split the original stroke into two sub-strokes and process them recursively.
3.6 Post-Process After the whole stroke has been recognized, a simple post-process will be applied to merge primitive shapes within one stroke, if they satisfy either of the following conditions:
Because the dividing point usually corresponds to a tangent point (as in Fig. 5a), this rule can decompose a smooth curve efficiently and will not generate the disturbing fragments.
Two connected line segments have nearly the same angle. Two connected arcs have nearly the same center and radius.
The other kind of approximation failure is called partial approximation failure, which is defined as that some segment of the input stroke cannot be approximated by any of the primitive shapes. From the definition, we can know that some salient vertexes have been detected, which implies the feature of the missing vertexes may be suppressed by them. For example, as the smooth parameter hc is computed as a global measurement, it is probably too large for some vertexes with small curvature variance, and consequently those vertexes will be fused with their neighbors. Fig. 6 shows an example.
4. EXPERIMENTS AND EVALUATIONS
Figure 8: Examples of hybrid shapes with explicit vertexes.
Figure 6: An example of partial approximation failure. For the above reason, we extract the failed stroke segment and process it as a new stroke recursively, which means that the stroke points will be re-analyzed using more specific parameters and the relatively vague vertexes will be more obvious this time. From this meaning, the vertexes are detected in an incremental way,
Figure 9: Examples of smooth hybrid shapes.
(a) Improper approximation
(b) Revised approximation
from salient ones to obscure ones.
We have conducted a user study to measure the efficiency of our approach. Three students, one from the Computer Science Department and two from other departments, participated in the study.
3.5.2 Handling Approximation Error
Firstly, they all thought our system is natural to use, and they could draw freely to obtain the desired shapes.
Figure 7: Handling approximation error.
An approximation error means an input stroke or stroke segment is not well fitted by the primitive shape. A typical example is shown in Fig. 7. This problem mainly comes from the inability of feature-area verification to distinguish between shapes with similar area.
The sample strokes for the user study were divided into three classes: primitive shapes, hybrid shapes with explicit vertexes (as in Fig. 8) and smooth hybrid shapes (as in Fig. 9). The experiment results are listed in Table 1 and 2, where the “correctness” means the recognition result exactly matches what is drawn by the user and the “validity” means the main portion and key structure are recognized correctly except for some small disturbances. Both correctness and validity were judged from the participator’s point of view.
To solve this problem, each approximation result will be checked whether there are some portions of the stroke deviating far from the recognized shape, and additional vertexes, especially intersection points, will be inserted if necessary. Then the newgenerated stroke segments will be recognized to replace the original approximation result.
Table 1: Correctness rate for primitive shapes.
Correctness Rate
209
Lines
Arcs
Circles
Ellipses
98%
97%
98%
96%
sketched strokes. Furthermore, it combines the vertex detection and primitive shape approximation into a unified and incremental procedure which, by fully utilizing the visual features, can handle hybrid and smooth curves gracefully. Since our method mainly utilizes low-level geometric features and does not rely on any domain-specific knowledge, it can be integrated easily with other high-level applications and will provide a natural and powerful means of human-computer interaction.
Table 2: Correctness and validity rate for hybrid shapes. Hybrid Shapes with Explicit Vertexes
Smooth Hybrid Shapes
Correctness Rate
84%
78%*
Validity Rate
94%
94%
From Table 1, we can see that our approach can recognize the four types of primitive shapes reliably. The correctness rate for ellipses is a little lower because sometimes a roughly sketched ellipse will be interpreted as a series of lines and arcs. The correctness rate of smooth hybrid shapes is marked with an asterisk because when doing the user study, there was much dispute about to which extent a recognition result could be considered as matching the intention of the user. But from the high validity rate for both kinds of hybrid shapes, we believe that our approach can produce a good approximation for any complex input stroke. In general, our approach is more robust than other existing methods for sketch recognition.
6. REFERENCES
In order to measure the computation cost, we gathered the average number of iteration for each stroke point when analyzing the direction-curvature joint space using the mean shift procedure. The mean value is 7.3 per point, which has little to do with the number of stroke points but is much related with their distribution in the feature space. As most strokes contain no more than 150 points, our method can be viewed as computation-efficient.
[4] Comaniciu, D., and Meer, P. Mean Shift: A Robust
There were also some limitations exposed in the user study. For example, two short connected lines with a small difference angle will be recognized as an arc, if they are drawn in one stoke which contains other more salient vertexes. In addition, we feel that treating wave lines as one kind of primitive shape is necessary, because our method encountered some difficulties when processing them in the user study.
[6] Fukunaga, K., and Hostetler, L.D. The Estimation of the
We are now building a query system for CAD drawings which allows users to sketch desired symbols and then fetches the corresponding drawings from a central database. It adopts the input method based on this paper and measures the similarities between input and model graphs through elastic graph matching.
[8] Igarashi, T., Matsuoka, S., Kawachiya, S. and Tanaka, H.
[1] Arvo, J., and Novins, K. Fluid Sketches: Continuous Recognition and Morphing of Simple Hand-Drawn Shapes. Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, 2000.
[2] Calhoun, C., Stahovich, T.F., Kurtoglu, T., and Kara, L.B. Recognizing Multi-Stroke Symbols. AAAI Spring Symposium on Sketch Understanding, 2002.
[3] Cheng, Y. Mean Shift, Mode Seeking, and Clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, No. 8, 1995. Approach Toward Feature Space Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 5, 2002.
[5] Fonseca, M.J., and Jorge, J.A. Using Fuzzy Logic to Recognize Geometric Shapes Interactively. Proceedings of the 9th International Conference on Fuzzy Systems, 2000. Gradient of a Density Function. IEEE Transactions on Information Theory, Vol. 21, 1975, 32-40.
[7] Hammond, T., and Davis, R. Tahuti: A Sketch Recognition System for UML Class Diagrams. AAAI Spring Symposium on Sketch Understanding, 2002. Interactive Beautification: A Technique for Rapid Geometric Design. UIST'97, 105-114.
[9] Oltmans,
M., and Davis, R. Naturally Conveyed Explanations of Device Behavior. Proceedings of PUI-2001, November 2001.
5. CONCLUSION
[10] Rubine, D. Specifying gestures by example. Computer
In this paper, we have proposed a robust method for freehand sketch recognition. It uses the mean shift procedure, an efficient tool which can delineate arbitrarily shaped clusters in a complex multimodal feature space, as a pre-process to analyze the direction-curvature joint space and suppress the severe noise of
Graphics (SIGGRAPH'91 Proceedings), 329-337.
[11] Sezgin, T.M., Stahovich, T., and Davis, R. Sketch Based Interfaces: Early Processing for Sketch Understanding. Proceedings of PUI-2001, November 2001.
210
