Effective Handwriting Recognition System Using Geometrical ...
Effective Handwriting Recognition System Using Geometrical Character Analysis Algorithms Wojciech Kacalak and Maciej Majewski Koszalin University of Technology, Department of Mechanical Engineering Raclawicka 15-17, Koszalin, Poland {wojciech.kacalak,maciej.majewski}@tu.koszalin.pl
Abstract. We propose a new method for natural writing recognition that utilizes geometric features of letters. The paper deals with recognition of isolated handwritten characters using an artificial neural network. As a result of the geometrical analysis realized, graphical representations of recognized characters are obtained in the form of pattern descriptions of isolated characters. The radius measurements of the characters obtained are inputs to the neural network for natural writing recognition which is font independent. In this paper, we present a new method for off-line natural writing recognition and also describe our research and tests performed on the neural network. Keywords: handwriting recognition, artificial neural networks, artificial intelligence, human-computer interaction, natural writing processing.
1
Introduction
Natural writing recognition has been studied for nearly forty years and there have been many proposed approaches. The problem is quite complex, and even now there is no single approach that solves it both efficiently and completely in all contexts. In written language recognition processes, an image containing text must be appropriately supplied and preprocessed. Then the text must either undergo segmentation or feature extraction. Small processed pieces of the text will be the result, and these must undergo recognition by the system. Finally, contextual information should be applied to the recognized symbols to verify the result. Artificial neural networks, applied in handwriting recognition, allow for high generalization ability and do not require deep background knowledge and formalization to be able to solve the written language recognition problem. Handwriting recognition can be divided by its input method into two categories: off-line handwriting recognition and on-line handwriting recognition. For off-line recognition, the writing is usually captured optically by a scanner. For on-line recognition, a digitizer samples the handwriting to time-sequenced pixels as it is being written. Hence, the on-line handwriting signal contains additional time information which is not present in the off-line signal. T. Huang et al. (Eds.): ICONIP 2012, Part IV, LNCS 7666, pp. 248–255, 2012. c Springer-Verlag Berlin Heidelberg 2012
Effective Handwriting Recognition System
PREPROCESSING SUBSYSTEM natural written text
digitization binarization noise elimination thinning normalizing
FUZZY LOGIC SUBSYSTEM
GEOMETRICAL ANALYSIS SUBSYSTEM for feature extraction by character Module of searching of centre of mass
Module of detection and extraction of geometrical features
Module of pattern description
segmentation natural writing recognised text
249
Module for recognizing phrases using artificial neural networks
Module for recognizing words using artificial neural networks
ANN training patterns of phrases (contextual knowledge from linguistics)
ANN training patterns of words (vocabulary verification)
Letter string recognition module NEURAL NETWORK SUBSYSTEM for vocabulary and linguistics
Module of input value normalization Module of fuzzy logic rules
Module for recognizing isolated characters using artificial neural networks NEURAL NETWORK SUBSYSTEM for isolated characters
Fig. 1. Scheme of the proposed natural writing recognition system
In the proposed new method of natural writing recognition in Fig. 1, the handwritten text is produced subject to the following preprocessing: digitization, binarization, noise elimination, thinning, normalizing and segmentation. The next step is to find the center of mass of the character image. With the center of mass as a reference point, radiuses are drawn, creating a set of points describing the contour of the character so that its pattern description is made. In the proposed hybrid system, the pattern description of each isolated character, after the process of input value normalization and application of letter description rules using fuzzy logic, are the input signals for the neural networks for isolated character recognition. The recognized characters are grouped into more quantitative units with the letter string recognition module, which are coded as binary images of vectors and then become inputs of the module for recognizing words. The module uses a 3-layer Hamming neural network. The network of this module uses a training file containing patterns of words. The recognized vocabulary words represented by the output neurons are processed by the module for recognizing phrases which uses the Hamming Maxnet network equipped with a training file containing phrases built with contextual knowledge from linguistics.
2
The State of the Art
The state of the art of automatic recognition of handwriting at the beginning of the new millennium is that as a field it is no longer an esoteric topic on the fringes of information technology, but a mature discipline that has found many commercial uses. On-line systems for handwriting recognition are available in hand-held computers such as personal digital assistants. Their performance is acceptable for processing handprinted symbols, and when combined with keyboard entry, a powerful method for data entry has been created. Off-line systems
250
W. Kacalak and M. Majewski
are less accurate than on-line systems. However, they are now good enough that they have a significant economic impact on specialized domains such as interpreting handwritten postal addresses on envelopes and reading courtesy amounts on bank checks [1,2,3,12]. The success of on-line systems makes it attractive to consider developing off-line systems that first estimate the trajectory of the writing from off-line data and then use on-line recognition algorithms [11]. However, the difficulty of recreating the temporal data [4] has led to few such feature extraction systems so far [1]. Research on automated written language recognition dates back several decades. Today, cleanly machine-printed text documents with simple layouts can be recognized reliably by OCR software. There is also some success with handwriting recognition, particularly for isolated handprinted characters and words. For example, in the on-line case, the recently introduced personal digital assistants have practical value. Similarly, some online signature verification systems have been marketed over the last few years and instructional tools to help children learn to write are beginning to emerge. Most of the off-line successes have come in constrained domains, such as postal addresses, bank checks, and census forms. The analysis of documents with complex layouts, recognition of degraded printed text, and the recognition of running handwriting continue to remain largely in the research arena. Some of the major research challenges in on-line or off-line processing of handwriting are in word and line separation, segmentation of words into characters, recognition of words when lexicons are large, and the use of language models in aiding preprocessing and recognition. In most applications, machine performance is far from being acceptable, although potential users often forget that human subjects generally make reading mistakes [2,3]. The design of human-computer interfaces [5,6,7,8,9] based on handwriting is part of a tremendous research effort together with speech recognition, language processing and translation to facilitate communication of people with computers. From this perspective, any successes or failures in these fields will have an important impact on the evolution of languages [10].
3
Description of the Method
The proposed system attempts to combine two methods for natural writing recognition, neural networks and preprocessing for geometric features extraction. The system consists of the preprocessing subsystem, geometrical analysis subsystem, fuzzy logic subsystem, neural network subsystem for isolated characters as well as neural network subsystem for vocabulary and linguistics, as shown in Fig. 2. The motivation behind that preprocessor is to reduce the dimensionality of the neural network input. However, another benefit given by the preprocessor is immunity against image translation, because all the information is relative to the image’s center of mass. The extraction process of the selected geometrical features of letters is based on application of the center of mass of a letter with a method of data clustering. The selected Fuzzy C-Means algorithm (Fig. 3) is described with typical denotations of data clustering algorithms and can be aliased as unsupervised
Effective Handwriting Recognition System Natural writing recognised text Phrase recognition module using neural networks Classification to phrase classes (Contextual knowledge from linguistics) WYRAZ 1 WYRAZ 2 WYRAZ 3
y
b
x
WYRAZ b
y
WYRAZ 1 WYRAZ 2 WYRAZ 3
Obraz binarny frazy
Mass centre search module using Fuzzy C-Means algorithm
b
x xc
WYRAZ b
Geometrical feature detection and extraction module
r11 r10
r2
r yc
r4
yc
xc
r8 y
r2 r3
Δr yc
yc
r4 xc
lf1
r9
xc
r5 lf3
r8
r6
r7
lf2
x
x
binary image of recognised word
n
y1 = å w 1i x i
å
i =1 n
y 2 = å w 2i x i
å
i =1
n
y j = å w ji x i
å
i =1
n
y M = å w Mi x i
å
i =1
Classification to word classes (Vocabulary patterns)
r10
x xc y
r1
r12 r11
x
r6
r7
ROZPOZNAWANIE
r5
r9
Pattern description module number of points
a
Word recognition module using neural networks
r3
l
Numer wyrazu
y
Rywalizacja, polaryzacja, ...
GEOMETRICAL ANALYSIS SUBSYSTEM
r1
r12
y
Network input: N=a*b
NEURAL NETWORK SUBSYSTEM for vocabulary and linguistics
binarization thinning segmentation
Obraz binarny rozpoznanej frazy
PREPROCESSING SUBSYSTEM
natural written text digitization noise elimination normalizing
251
14 12 10 8 6 4
OUTPUT: binary images of the recognised words
Network input: N=26*a
2 0 1
2
3
4
5
6
7
8
9
10
11
NETWORK INPUT: binary images as isolated components of the text created by the letter string recognition module
12
r Nr promienia
number of points
14 12 10
Letter string recognition module
8 6 4 2 0 1
2
3
4
5
6
7
8
9
FUZZY LOGIC SUBSYSTEM
r Nr promienia
10
11
12
Isolated character recognition module using artificial neural networks
Normalization module using fuzzy logic
NEURAL NETWORK SUBSYSTEM for isolated characters
Fig. 2. Algorithm of the proposed system of effective handwriting recognition
252
W. Kacalak and M. Majewski
Fig. 3. The Fuzzy C-Means algorithm to find the center of mass of an isolated character y
A)
C)
E)
r1
r12
y
f=12 f=1 r11 yc
r10
xc
r8
l
r7
r5
r10
y
D) y
r10
f=12
r11 yc
f=1 1 f=10 f=9
f=8
r9 r8
r12
r1
r12
rfn
r9
x
r6
f
r7
r6
r5
f=1 f=2
r11
r3
f=3 f=4
r10
r4
f=7 f=6
xc
r12
rfn
r1
r11
r3 r4
f=4 rf
r9 x
f=2 f=3
rf1 ¸ rfn rf1
r5 r6
r1 yc
r8
rfx
x
f=7 xc r8
r7
lf1
r7
y
r12 r3
r11 r4
r6
r10
r2 f=2
Δr
r3
f=3
rfn
r9 x
r1
yc
r4
f=4
rf1 ¸ rfn rf1
r5
lf3
x
r6
Δrf = rf +1 - rf
f=4
r8
r5
r9
x
r2
xc lf2
r6
Δr
H)
f=3
rfn
r9
r5 xc
r7
r10
r4
yc
rf1 ¸ rfn
f=2 yc
rf1 ¸ rfn rf1
r10
= lf1 + lf2 + K + lfn y
r2
rf1
r2
s f=5
r7
xc
r8
F)
r1
r3
r11 r4
yc
ål B)
r3 f=3
r4 r12 r1
xc
r11
r2
f=12 f=1
f=3
yc
r9
x
r3
r1
r12
y
r2
f=1
r12 r1
r10
s
G)
r1
r12
y
r2
Δr r8
xc
r7
r5 r6
x
Fig. 4. Geometrical analyses of characters: A) determination of the center of the mass for exemplary letter s; B) determination of intersection points of the letter and the radiuses for exemplary letter o; C) measurement of the length of line segments l created by letter points in fragments f for letter e; D) summation of measurements in fragments f containing n radiuses for letter l; E) measurement of the length of line segments of each radius for exemplary letter e and letter l (F); G) measurements of differences of the radius lengths in each fragment f for exemplary letter e and letter l (H).
learning. After the first partitioning of letter points into clusters and obtaining their cluster centers, a new clustering is performed with the algorithm, which is a partitioning of the obtained cluster centers. The clustering is repeated with the algorithm until two clusters are obtained. The center of the line segment created by the last two cluster centers is the center of mass of the letter. The developed geometrical analysis is based on the processing of the images of letter shapes into their graphical representations in the form of pattern descriptions. The process of the geometrical analysis begins with determining of the center of mass of the letter in order to find the initial point of the analysis. The next step of the algorithm is based on drawing radiuses from the initial point, the lengths of which are equal to the length of the line segment created by the initial point and the point on the letter furthest from this point. The
Effective Handwriting Recognition System y
y y Lo r1 r12
r11 r10 r9
rn r2 r r4 f=4
xc r8
r11 r3
Lfi
yc r7
r1 f=1
r6
r5
p1
pn
r
r11
r5
xc f=7 r7
r6
Lo
Lo
r7
r6
r5
r10
Lfi
yc
y Lo
o1 r3
o2
p11
r r f=4 4 xc
r2 f=12 f=1 p2
r11
r2 r3
yc r8
x
r1 Lfi
r9
Lfi
p7
r9
r10
r4
s
yc
x
r8
r12
r3
p3
s(xc, yc)
r10
y Lo
o1
o2
Δr
r1
rn r2
r
s(xc, yc)
r12 r4
s
x
p8
r9 r8
253
r5
f=7 xc
r7
x
r1
r2
Lfi
r11 yc r10 r9
r3
xc r8
r7
r r f=4 4 r6
r5
x
r6
Fig. 5. Geometrical analyses of characters for exemplary letters
creation of a circle of that radius makes it visible that the analysis covers the whole letter. The precision of this geometrical analysis method is proportional to the number of radiuses. Where the radiuses intersect with the letter, points are obtained, which makes it possible to obtain the measures of the line segment created by the initial point and the letter intersection point. The lengths of the created line segments obtained are represented in the form of pattern descriptions of isolated characters which are inputs of the neural network. Geometrical analyses of characters for exemplary letters are shown in Fig. 4 and Fig. 5.
4
Experimental Results
n eu r al n e t w o r k er r o r [ % ]
The research on the developed method concerns the ability of the neural network to learn to recognize specific letters. The neural networks are trained with the model of isolated written language characters. 100 90 80 70 60 50 40 30 20 10 0 0
100
200 300 time of learning [epochs]
400
500
Fig. 6. The error rate of the neural network for recognition of isolated handwritten characters
The ability of the neural network to learn to recognize specific letters depends on the number of learning epochs. The specified time of learning enables the network to minimize the error so that it can work more efficiently. Based on the research, the error rate achieved is as shown in Fig. 6.
254
W. Kacalak and M. Majewski y
y
y
y
yc
yc
yc
yc
y
yc
100
80 60 40 20
80 60 40 20
1
2
3
4
5
6
7
8
9
10
11
40 20
2
3
4
5
6
7
8
9
10
11
1
12
2
3
4
5
6
yc
40 20
7
8
9
10
11
80 60 40 20 0
1
12
2
3
4
5
6
7
8
9
10
11
12
1
y
3
4
5
6
7
8
9
10
11
12
r
y
y
yc yc
yc
yc
80 60 40 20
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
7
r
8
9
10
11
xc
xc
6
80 60 40 20 0
7
8
9
10
11
8
9
10
11
12
2
3
4
5
6
2
3
4
5
6
7
8
9
10
11
7
8
9
10
11
1
5
6
7
8
9
10
11
8
9
10
11
12
20
12
x
xc
80 60 40
80 60 40 20 0
1
2
3
4
5
6
r
7
8
9
10
11
1
12
2
3
4
5
6
r
y
7
8
9
10
11
12
r
y
y
yc
yc
yc
xc
x
radius length [%]
0
80 60 40 20 0
2
3
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
r
7
8
9
10
11
yc
6
7
r
6
8
9
10
11
12
7
8
9
10
11
20
20 0
1
12
2
3
4
5
6
7
8
9
10
11
12
1
yc
yc
xc
x
xc
x
100
radius length [%]
80
40 20
2
3
4
5
6
7
r
8
9
10
11
12
x
40 20
2
3
4
5
6
5
6
7
8
9
10
11
r
12
7
8
9
10
11
12
x
100
80 60 40 20 0
1
4
xc
100
80 60
0 1
3
r
yc
60
2
r
yc
0
0 5
5
60 40
y
radius length [%]
40
4
4
80
y
xc
20
3
3
r
100
2
2
100
y
x
xc 80 60
60 40
0
1
x
120
80
y
100
1
10 0
12
r
y
xc
x
xc 100
radius length [%]
1
x
70 60 50 40 30 20
radius length [%]
20
radius length [%]
100
60 40
radius length [%]
xc
x
80
radius length [%]
xc 100
radius length [%]
7
yc
yc
radius length [%]
6
100
100
0 4
5
x
xc
20
3
4
r
120
60 40
2
3
yc
x
80
1
2
y
yc
12
40 20
12
y
r
y
80 60
0 1
0 1
r
y
40 20
12
r
xc
100
0 6
7
100
radius length [%]
radius length [%]
20 5
5
x
120
4
4
y
x
80 60 40
3
3
yc
120 100
80 60
0
2
r
yc
yc
2
40 20
1
y
y
1
80 60
12
r
x
100
radius length [%]
3
radius length [%]
2
xc
x
xc 100
0
0 1
x
100
radius length [%]
radius length [%]
radius length [%]
xc
x
100
radius length [%]
xc
x
80 70 60 50 40 30 20 10 0
radius length [%]
xc
radius length [%]
2
r
r
r
y
x
xc 100
80 60
0
0
1
12
r
y
x
100
80 60
0
0
xc
x
100
radius length [%]
radius length [%]
radius length [%]
100
xc
x
xc
radius length [%]
x
radius length [%]
xc
80 60 40 20 0
1
2
3
4
5
6
7
r
8
9
10
11
12
1
2
3
4
5
6
7
8
9
10
11
12
r
Fig. 7. Geometrical analysis and pattern description of isolated characters
Error rate is about 20% at learning time equals 50 epochs and 5% at 100 epochs. The error rate dropped by about 90% after training with 60 series of all patterns. Several geometrical analyses of isolated characters and their pattern description were realized, which made it possible to draw significant conclusions (Fig. 7) and apply them in the proposed algorithms.
5
Conclusions and Perspectives
Many advances and changes have occurred in the field of automated written language recognition, over the last decade. The different sources of variability in psychophysical aspects of the generation and perception of written language make handwriting processing difficult.
Effective Handwriting Recognition System
255
Considerable progress has been made in natural writing recognition technology. Written language recognition systems have been limited to small and medium vocabulary applications, since most of them often rely on a lexicon during the recognition process. The capability of dealing with large lexicons, however, opens up many more applications. The advantages of this new method of natural writing recognition are flexibility with regards to writing style, geometrical analysis enabling font independent character recognition, possibility of application of other types of neural networks, extension of the range of geometrical analysis and other possibilities for further development.
References 1. Artieres, T., Gauthier, N., Gallinari, P., Dorizzi, B.: A Hidden Markov Models combination framework for handwriting recognition. International Journal on Document Analysis and Recognition 5(4), 233–243 (2003) 2. Besner, D., Humphreys, G.W.: Basic Processes in Reading: Visual Word Recognition. Lawrence Erlbaum Associates, Hillsdale (1991) 3. Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press Inc., New York (2004) 4. Dori, D., Bruckstein, A.: Shape, Structure and Pattern Recognition. World Scientific Publishing Co., New Jersey (1995) 5. Kacalak, W., Majewski, M.: Intelligent System for Automatic Recognition and Evaluation of Speech Commands. In: King, I., Wang, J., Chan, L.-W., Wang, D. (eds.) ICONIP 2006. LNCS, vol. 4232, pp. 298–305. Springer, Heidelberg (2006) 6. Kacalak, W., Majewski, M.: E-Learning Systems with Artificial Intelligence in Engineering. In: Huang, D.-S., Jo, K.-H., Lee, H.-H., Kang, H.-J., Bevilacqua, V. (eds.) ICIC 2009. LNCS, vol. 5754, pp. 918–927. Springer, Heidelberg (2009) 7. Majewski, M., Kacalak, W.: Intelligent System for Natural Language Processing. In: Huang, D.-S., Li, K., Irwin, G.W. (eds.) ICIC 2006. LNCS (LNAI), vol. 4114, pp. 742–747. Springer, Heidelberg (2006) 8. Kacalak, W., Majewski, M., Zurada, J.M.: Intelligent E-Learning Systems for Evaluation of User’s Knowledge and Skills with Efficient Information Processing. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010, Part II. LNCS, vol. 6114, pp. 508–515. Springer, Heidelberg (2010) 9. Majewski, M., Zurada, J.M.: Sentence recognition using artificial neural networks. Elsevier Knowledge-Based Systems 21(7), 629–635 (2008) 10. Mori, S., Nishida, H., Yamada, H.: Optical Character Recognition. John Wiley & Sons, Inc., New York (1999) 11. Nishida, H.: An Approach to Integration of Off-Line and On-Line Recognition of Handwriting. Pattern Recognition Letters 16(11), 1213–1219 (1995) 12. Zhou, J., Krzyzak, A., Suen, C.Y.: Verification-a Method of Enhancing the Recognizers of Isolated and Touching Handwritten Numerals. Pattern Recognition 35(5), 1179–1189 (2002)
Abstract. We propose a new method for natural writing recognition that utilizes geometric features of letters. The paper deals with recognition of isolated handwritten characters using an artificial neural network. As a result of the geometrical analysis realized, graphical representations of recognized characters are obtained in the form of pattern descriptions of isolated characters. The radius measurements of the characters obtained are inputs to the neural network for natural writing recognition which is font independent. In this paper, we present a new method for off-line natural writing recognition and also describe our research and tests performed on the neural network. Keywords: handwriting recognition, artificial neural networks, artificial intelligence, human-computer interaction, natural writing processing.
1
Introduction
Natural writing recognition has been studied for nearly forty years and there have been many proposed approaches. The problem is quite complex, and even now there is no single approach that solves it both efficiently and completely in all contexts. In written language recognition processes, an image containing text must be appropriately supplied and preprocessed. Then the text must either undergo segmentation or feature extraction. Small processed pieces of the text will be the result, and these must undergo recognition by the system. Finally, contextual information should be applied to the recognized symbols to verify the result. Artificial neural networks, applied in handwriting recognition, allow for high generalization ability and do not require deep background knowledge and formalization to be able to solve the written language recognition problem. Handwriting recognition can be divided by its input method into two categories: off-line handwriting recognition and on-line handwriting recognition. For off-line recognition, the writing is usually captured optically by a scanner. For on-line recognition, a digitizer samples the handwriting to time-sequenced pixels as it is being written. Hence, the on-line handwriting signal contains additional time information which is not present in the off-line signal. T. Huang et al. (Eds.): ICONIP 2012, Part IV, LNCS 7666, pp. 248–255, 2012. c Springer-Verlag Berlin Heidelberg 2012
Effective Handwriting Recognition System
PREPROCESSING SUBSYSTEM natural written text
digitization binarization noise elimination thinning normalizing
FUZZY LOGIC SUBSYSTEM
GEOMETRICAL ANALYSIS SUBSYSTEM for feature extraction by character Module of searching of centre of mass
Module of detection and extraction of geometrical features
Module of pattern description
segmentation natural writing recognised text
249
Module for recognizing phrases using artificial neural networks
Module for recognizing words using artificial neural networks
ANN training patterns of phrases (contextual knowledge from linguistics)
ANN training patterns of words (vocabulary verification)
Letter string recognition module NEURAL NETWORK SUBSYSTEM for vocabulary and linguistics
Module of input value normalization Module of fuzzy logic rules
Module for recognizing isolated characters using artificial neural networks NEURAL NETWORK SUBSYSTEM for isolated characters
Fig. 1. Scheme of the proposed natural writing recognition system
In the proposed new method of natural writing recognition in Fig. 1, the handwritten text is produced subject to the following preprocessing: digitization, binarization, noise elimination, thinning, normalizing and segmentation. The next step is to find the center of mass of the character image. With the center of mass as a reference point, radiuses are drawn, creating a set of points describing the contour of the character so that its pattern description is made. In the proposed hybrid system, the pattern description of each isolated character, after the process of input value normalization and application of letter description rules using fuzzy logic, are the input signals for the neural networks for isolated character recognition. The recognized characters are grouped into more quantitative units with the letter string recognition module, which are coded as binary images of vectors and then become inputs of the module for recognizing words. The module uses a 3-layer Hamming neural network. The network of this module uses a training file containing patterns of words. The recognized vocabulary words represented by the output neurons are processed by the module for recognizing phrases which uses the Hamming Maxnet network equipped with a training file containing phrases built with contextual knowledge from linguistics.
2
The State of the Art
The state of the art of automatic recognition of handwriting at the beginning of the new millennium is that as a field it is no longer an esoteric topic on the fringes of information technology, but a mature discipline that has found many commercial uses. On-line systems for handwriting recognition are available in hand-held computers such as personal digital assistants. Their performance is acceptable for processing handprinted symbols, and when combined with keyboard entry, a powerful method for data entry has been created. Off-line systems
250
W. Kacalak and M. Majewski
are less accurate than on-line systems. However, they are now good enough that they have a significant economic impact on specialized domains such as interpreting handwritten postal addresses on envelopes and reading courtesy amounts on bank checks [1,2,3,12]. The success of on-line systems makes it attractive to consider developing off-line systems that first estimate the trajectory of the writing from off-line data and then use on-line recognition algorithms [11]. However, the difficulty of recreating the temporal data [4] has led to few such feature extraction systems so far [1]. Research on automated written language recognition dates back several decades. Today, cleanly machine-printed text documents with simple layouts can be recognized reliably by OCR software. There is also some success with handwriting recognition, particularly for isolated handprinted characters and words. For example, in the on-line case, the recently introduced personal digital assistants have practical value. Similarly, some online signature verification systems have been marketed over the last few years and instructional tools to help children learn to write are beginning to emerge. Most of the off-line successes have come in constrained domains, such as postal addresses, bank checks, and census forms. The analysis of documents with complex layouts, recognition of degraded printed text, and the recognition of running handwriting continue to remain largely in the research arena. Some of the major research challenges in on-line or off-line processing of handwriting are in word and line separation, segmentation of words into characters, recognition of words when lexicons are large, and the use of language models in aiding preprocessing and recognition. In most applications, machine performance is far from being acceptable, although potential users often forget that human subjects generally make reading mistakes [2,3]. The design of human-computer interfaces [5,6,7,8,9] based on handwriting is part of a tremendous research effort together with speech recognition, language processing and translation to facilitate communication of people with computers. From this perspective, any successes or failures in these fields will have an important impact on the evolution of languages [10].
3
Description of the Method
The proposed system attempts to combine two methods for natural writing recognition, neural networks and preprocessing for geometric features extraction. The system consists of the preprocessing subsystem, geometrical analysis subsystem, fuzzy logic subsystem, neural network subsystem for isolated characters as well as neural network subsystem for vocabulary and linguistics, as shown in Fig. 2. The motivation behind that preprocessor is to reduce the dimensionality of the neural network input. However, another benefit given by the preprocessor is immunity against image translation, because all the information is relative to the image’s center of mass. The extraction process of the selected geometrical features of letters is based on application of the center of mass of a letter with a method of data clustering. The selected Fuzzy C-Means algorithm (Fig. 3) is described with typical denotations of data clustering algorithms and can be aliased as unsupervised
Effective Handwriting Recognition System Natural writing recognised text Phrase recognition module using neural networks Classification to phrase classes (Contextual knowledge from linguistics) WYRAZ 1 WYRAZ 2 WYRAZ 3
y
b
x
WYRAZ b
y
WYRAZ 1 WYRAZ 2 WYRAZ 3
Obraz binarny frazy
Mass centre search module using Fuzzy C-Means algorithm
b
x xc
WYRAZ b
Geometrical feature detection and extraction module
r11 r10
r2
r yc
r4
yc
xc
r8 y
r2 r3
Δr yc
yc
r4 xc
lf1
r9
xc
r5 lf3
r8
r6
r7
lf2
x
x
binary image of recognised word
n
y1 = å w 1i x i
å
i =1 n
y 2 = å w 2i x i
å
i =1
n
y j = å w ji x i
å
i =1
n
y M = å w Mi x i
å
i =1
Classification to word classes (Vocabulary patterns)
r10
x xc y
r1
r12 r11
x
r6
r7
ROZPOZNAWANIE
r5
r9
Pattern description module number of points
a
Word recognition module using neural networks
r3
l
Numer wyrazu
y
Rywalizacja, polaryzacja, ...
GEOMETRICAL ANALYSIS SUBSYSTEM
r1
r12
y
Network input: N=a*b
NEURAL NETWORK SUBSYSTEM for vocabulary and linguistics
binarization thinning segmentation
Obraz binarny rozpoznanej frazy
PREPROCESSING SUBSYSTEM
natural written text digitization noise elimination normalizing
251
14 12 10 8 6 4
OUTPUT: binary images of the recognised words
Network input: N=26*a
2 0 1
2
3
4
5
6
7
8
9
10
11
NETWORK INPUT: binary images as isolated components of the text created by the letter string recognition module
12
r Nr promienia
number of points
14 12 10
Letter string recognition module
8 6 4 2 0 1
2
3
4
5
6
7
8
9
FUZZY LOGIC SUBSYSTEM
r Nr promienia
10
11
12
Isolated character recognition module using artificial neural networks
Normalization module using fuzzy logic
NEURAL NETWORK SUBSYSTEM for isolated characters
Fig. 2. Algorithm of the proposed system of effective handwriting recognition
252
W. Kacalak and M. Majewski
Fig. 3. The Fuzzy C-Means algorithm to find the center of mass of an isolated character y
A)
C)
E)
r1
r12
y
f=12 f=1 r11 yc
r10
xc
r8
l
r7
r5
r10
y
D) y
r10
f=12
r11 yc
f=1 1 f=10 f=9
f=8
r9 r8
r12
r1
r12
rfn
r9
x
r6
f
r7
r6
r5
f=1 f=2
r11
r3
f=3 f=4
r10
r4
f=7 f=6
xc
r12
rfn
r1
r11
r3 r4
f=4 rf
r9 x
f=2 f=3
rf1 ¸ rfn rf1
r5 r6
r1 yc
r8
rfx
x
f=7 xc r8
r7
lf1
r7
y
r12 r3
r11 r4
r6
r10
r2 f=2
Δr
r3
f=3
rfn
r9 x
r1
yc
r4
f=4
rf1 ¸ rfn rf1
r5
lf3
x
r6
Δrf = rf +1 - rf
f=4
r8
r5
r9
x
r2
xc lf2
r6
Δr
H)
f=3
rfn
r9
r5 xc
r7
r10
r4
yc
rf1 ¸ rfn
f=2 yc
rf1 ¸ rfn rf1
r10
= lf1 + lf2 + K + lfn y
r2
rf1
r2
s f=5
r7
xc
r8
F)
r1
r3
r11 r4
yc
ål B)
r3 f=3
r4 r12 r1
xc
r11
r2
f=12 f=1
f=3
yc
r9
x
r3
r1
r12
y
r2
f=1
r12 r1
r10
s
G)
r1
r12
y
r2
Δr r8
xc
r7
r5 r6
x
Fig. 4. Geometrical analyses of characters: A) determination of the center of the mass for exemplary letter s; B) determination of intersection points of the letter and the radiuses for exemplary letter o; C) measurement of the length of line segments l created by letter points in fragments f for letter e; D) summation of measurements in fragments f containing n radiuses for letter l; E) measurement of the length of line segments of each radius for exemplary letter e and letter l (F); G) measurements of differences of the radius lengths in each fragment f for exemplary letter e and letter l (H).
learning. After the first partitioning of letter points into clusters and obtaining their cluster centers, a new clustering is performed with the algorithm, which is a partitioning of the obtained cluster centers. The clustering is repeated with the algorithm until two clusters are obtained. The center of the line segment created by the last two cluster centers is the center of mass of the letter. The developed geometrical analysis is based on the processing of the images of letter shapes into their graphical representations in the form of pattern descriptions. The process of the geometrical analysis begins with determining of the center of mass of the letter in order to find the initial point of the analysis. The next step of the algorithm is based on drawing radiuses from the initial point, the lengths of which are equal to the length of the line segment created by the initial point and the point on the letter furthest from this point. The
Effective Handwriting Recognition System y
y y Lo r1 r12
r11 r10 r9
rn r2 r r4 f=4
xc r8
r11 r3
Lfi
yc r7
r1 f=1
r6
r5
p1
pn
r
r11
r5
xc f=7 r7
r6
Lo
Lo
r7
r6
r5
r10
Lfi
yc
y Lo
o1 r3
o2
p11
r r f=4 4 xc
r2 f=12 f=1 p2
r11
r2 r3
yc r8
x
r1 Lfi
r9
Lfi
p7
r9
r10
r4
s
yc
x
r8
r12
r3
p3
s(xc, yc)
r10
y Lo
o1
o2
Δr
r1
rn r2
r
s(xc, yc)
r12 r4
s
x
p8
r9 r8
253
r5
f=7 xc
r7
x
r1
r2
Lfi
r11 yc r10 r9
r3
xc r8
r7
r r f=4 4 r6
r5
x
r6
Fig. 5. Geometrical analyses of characters for exemplary letters
creation of a circle of that radius makes it visible that the analysis covers the whole letter. The precision of this geometrical analysis method is proportional to the number of radiuses. Where the radiuses intersect with the letter, points are obtained, which makes it possible to obtain the measures of the line segment created by the initial point and the letter intersection point. The lengths of the created line segments obtained are represented in the form of pattern descriptions of isolated characters which are inputs of the neural network. Geometrical analyses of characters for exemplary letters are shown in Fig. 4 and Fig. 5.
4
Experimental Results
n eu r al n e t w o r k er r o r [ % ]
The research on the developed method concerns the ability of the neural network to learn to recognize specific letters. The neural networks are trained with the model of isolated written language characters. 100 90 80 70 60 50 40 30 20 10 0 0
100
200 300 time of learning [epochs]
400
500
Fig. 6. The error rate of the neural network for recognition of isolated handwritten characters
The ability of the neural network to learn to recognize specific letters depends on the number of learning epochs. The specified time of learning enables the network to minimize the error so that it can work more efficiently. Based on the research, the error rate achieved is as shown in Fig. 6.
254
W. Kacalak and M. Majewski y
y
y
y
yc
yc
yc
yc
y
yc
100
80 60 40 20
80 60 40 20
1
2
3
4
5
6
7
8
9
10
11
40 20
2
3
4
5
6
7
8
9
10
11
1
12
2
3
4
5
6
yc
40 20
7
8
9
10
11
80 60 40 20 0
1
12
2
3
4
5
6
7
8
9
10
11
12
1
y
3
4
5
6
7
8
9
10
11
12
r
y
y
yc yc
yc
yc
80 60 40 20
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
7
r
8
9
10
11
xc
xc
6
80 60 40 20 0
7
8
9
10
11
8
9
10
11
12
2
3
4
5
6
2
3
4
5
6
7
8
9
10
11
7
8
9
10
11
1
5
6
7
8
9
10
11
8
9
10
11
12
20
12
x
xc
80 60 40
80 60 40 20 0
1
2
3
4
5
6
r
7
8
9
10
11
1
12
2
3
4
5
6
r
y
7
8
9
10
11
12
r
y
y
yc
yc
yc
xc
x
radius length [%]
0
80 60 40 20 0
2
3
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
r
7
8
9
10
11
yc
6
7
r
6
8
9
10
11
12
7
8
9
10
11
20
20 0
1
12
2
3
4
5
6
7
8
9
10
11
12
1
yc
yc
xc
x
xc
x
100
radius length [%]
80
40 20
2
3
4
5
6
7
r
8
9
10
11
12
x
40 20
2
3
4
5
6
5
6
7
8
9
10
11
r
12
7
8
9
10
11
12
x
100
80 60 40 20 0
1
4
xc
100
80 60
0 1
3
r
yc
60
2
r
yc
0
0 5
5
60 40
y
radius length [%]
40
4
4
80
y
xc
20
3
3
r
100
2
2
100
y
x
xc 80 60
60 40
0
1
x
120
80
y
100
1
10 0
12
r
y
xc
x
xc 100
radius length [%]
1
x
70 60 50 40 30 20
radius length [%]
20
radius length [%]
100
60 40
radius length [%]
xc
x
80
radius length [%]
xc 100
radius length [%]
7
yc
yc
radius length [%]
6
100
100
0 4
5
x
xc
20
3
4
r
120
60 40
2
3
yc
x
80
1
2
y
yc
12
40 20
12
y
r
y
80 60
0 1
0 1
r
y
40 20
12
r
xc
100
0 6
7
100
radius length [%]
radius length [%]
20 5
5
x
120
4
4
y
x
80 60 40
3
3
yc
120 100
80 60
0
2
r
yc
yc
2
40 20
1
y
y
1
80 60
12
r
x
100
radius length [%]
3
radius length [%]
2
xc
x
xc 100
0
0 1
x
100
radius length [%]
radius length [%]
radius length [%]
xc
x
100
radius length [%]
xc
x
80 70 60 50 40 30 20 10 0
radius length [%]
xc
radius length [%]
2
r
r
r
y
x
xc 100
80 60
0
0
1
12
r
y
x
100
80 60
0
0
xc
x
100
radius length [%]
radius length [%]
radius length [%]
100
xc
x
xc
radius length [%]
x
radius length [%]
xc
80 60 40 20 0
1
2
3
4
5
6
7
r
8
9
10
11
12
1
2
3
4
5
6
7
8
9
10
11
12
r
Fig. 7. Geometrical analysis and pattern description of isolated characters
Error rate is about 20% at learning time equals 50 epochs and 5% at 100 epochs. The error rate dropped by about 90% after training with 60 series of all patterns. Several geometrical analyses of isolated characters and their pattern description were realized, which made it possible to draw significant conclusions (Fig. 7) and apply them in the proposed algorithms.
5
Conclusions and Perspectives
Many advances and changes have occurred in the field of automated written language recognition, over the last decade. The different sources of variability in psychophysical aspects of the generation and perception of written language make handwriting processing difficult.
Effective Handwriting Recognition System
255
Considerable progress has been made in natural writing recognition technology. Written language recognition systems have been limited to small and medium vocabulary applications, since most of them often rely on a lexicon during the recognition process. The capability of dealing with large lexicons, however, opens up many more applications. The advantages of this new method of natural writing recognition are flexibility with regards to writing style, geometrical analysis enabling font independent character recognition, possibility of application of other types of neural networks, extension of the range of geometrical analysis and other possibilities for further development.
References 1. Artieres, T., Gauthier, N., Gallinari, P., Dorizzi, B.: A Hidden Markov Models combination framework for handwriting recognition. International Journal on Document Analysis and Recognition 5(4), 233–243 (2003) 2. Besner, D., Humphreys, G.W.: Basic Processes in Reading: Visual Word Recognition. Lawrence Erlbaum Associates, Hillsdale (1991) 3. Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press Inc., New York (2004) 4. Dori, D., Bruckstein, A.: Shape, Structure and Pattern Recognition. World Scientific Publishing Co., New Jersey (1995) 5. Kacalak, W., Majewski, M.: Intelligent System for Automatic Recognition and Evaluation of Speech Commands. In: King, I., Wang, J., Chan, L.-W., Wang, D. (eds.) ICONIP 2006. LNCS, vol. 4232, pp. 298–305. Springer, Heidelberg (2006) 6. Kacalak, W., Majewski, M.: E-Learning Systems with Artificial Intelligence in Engineering. In: Huang, D.-S., Jo, K.-H., Lee, H.-H., Kang, H.-J., Bevilacqua, V. (eds.) ICIC 2009. LNCS, vol. 5754, pp. 918–927. Springer, Heidelberg (2009) 7. Majewski, M., Kacalak, W.: Intelligent System for Natural Language Processing. In: Huang, D.-S., Li, K., Irwin, G.W. (eds.) ICIC 2006. LNCS (LNAI), vol. 4114, pp. 742–747. Springer, Heidelberg (2006) 8. Kacalak, W., Majewski, M., Zurada, J.M.: Intelligent E-Learning Systems for Evaluation of User’s Knowledge and Skills with Efficient Information Processing. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010, Part II. LNCS, vol. 6114, pp. 508–515. Springer, Heidelberg (2010) 9. Majewski, M., Zurada, J.M.: Sentence recognition using artificial neural networks. Elsevier Knowledge-Based Systems 21(7), 629–635 (2008) 10. Mori, S., Nishida, H., Yamada, H.: Optical Character Recognition. John Wiley & Sons, Inc., New York (1999) 11. Nishida, H.: An Approach to Integration of Off-Line and On-Line Recognition of Handwriting. Pattern Recognition Letters 16(11), 1213–1219 (1995) 12. Zhou, J., Krzyzak, A., Suen, C.Y.: Verification-a Method of Enhancing the Recognizers of Isolated and Touching Handwritten Numerals. Pattern Recognition 35(5), 1179–1189 (2002)
