Inexact graph recognition matching for structural pattern
Pattern Recognition Letters 1 (1983) 245-253 North-Holland
May 1983
Inexact graph matching for structural pattern recognition H. B u n k e Lehrstuhl fffr Informatik 5 (Mustererkennung), Universitiit Erlangen-Niirnberg, F.R. Germany
G. A l l e r m a n n Mannesmann Forschungsinstitut, Duisburg, F.R. Germany Received 25 January 1983; revised 10 March 1983
Abstract: This paper is concerned with the inexact matching of attributed, relational graphs for structural pattern recognition. The matching procedure is based on a state space search utilizing heuristic information. Some experimental results are reported.
Key words: Attributed graphs, inexact matching, heuristic search, structural pattern recognition, silhouette recognition.
1. Introduction In the structural approach, patterns are characterized as consisting of sub-patterns among which various relations exist (Fu (1982), Pavlidis (1977)). There are several ways for pattern description and representation in structural pattern recognition. A basic way is by means of strings. Due to the two-dimensionality of pictorial patterns, however, a two-dimensional description formalism seems to be more adequate for certain applications. Consequently, the use of trees for pattern representation has been proposed (Fu and Bhargava (1973)). A natural generalization of trees is graphs. Their use allows the description of complex structures in a straightforward way on a high, problem-oriented level. Besides representation of single patterns, the description of pattern classes is crucial. A method which has gained much importance over the last decade is the use of formal grammars (Fu (1982)). Grammars are favourable when a large number of different pattern classes is involved. At the other hand, there exist cases where the use of a grammar is not feasible, for example, when the set of sample
patterns is too small for the inference of a representative grammar. In this case, the patterns in the sample set can directly be used as prototypes, representing a class each. For the classification a n d / o r description of an unknown pattern, one can match the input pattern against the prototypes. Pattern matching is equivalent to graph matching when the patterns in question are represented by graphs. There are several papers in literature addressing the domain of matching by finding a graph isomorphism, e.g. Corneil and Gottlieb (1970). In practical applications, however, there is noise and distortions causing variation in both the input graph and the prototype. Therefore, our problem turns into a problem of inexact matching. This means given an input graph ~o and a set of prototypes ~ ..... ~ , all possibly different from co, we want to find out which one of the prototypes most closely resembles the input graph. As a consequence, we have to define some measure of similarity on graphs. Several approaches to inexact graph matching have been proposed in literature, e.g. Cheng and Huang (1981), Sanfeliu (1980), Shapiro (1981), Tsai and Fu (1979), Tsai and
0167-8655/83/53.00 ~ 1983, Elsevier Science Publishers B.V. (North-Holland)
245
Volume 1, Number 4
PATTERN RECOGNITION LETTERS
Chang (1982). The approach proposed in this paper is closey related to the work of Tsai and Fu (1979) and Tsai and Chang (1982), i.e. their definition of relational graphs is adopted and matching is based on a state space search. Following are differences: inexact matching is generalized to arbitrary graphs; an extension to attributed graphs is proposed; the cost functions are defined in such a way that the graph distance fulfils the properties of a metric under certain conditions.
2. Basic definitions Let Vs and Vs be a set of node and branch labels, respectively. Each symbol belonging to VN or VB is of the form (s,x) where s is the syntactic c o m p o n e n t and x = (xl . . . . . xn) is a semantic vector consisting of attribute values associated with s. The following definition is due to Tsai and Fu (1979). Definition 1. An a t t r i b u t e d r e l a t i o n a l g r a p h over V= VN U Vs is a 4-tupel ta = (N, B, a, e) where (1) N is a finite, n o n e m p t y set of nodes, (2) B c_N x N is the set of branches, (3) a : N - - , VN is a function for labeling the nodes, (4) c:B--. VB is a function for labeling the edges.
May 1983
DELNODE(0")(INSNODE(0")).The costs for substituting a node with syntactic label a by a node with syntactic label r are given by SUBNODE (e, r). Similarly, SUBBRANCH(a,r) are costs for branch substitution. Defining costs for branch deletion and insertion is slightly more difficult since these transformations are dependent on the bordering nodes. When a node is deleted then all its outgoing and incoming branches vanish, too. Conversely, when inserting a node, there are usually branches to be inserted, too, for properly embedding this node in the rest of the graph. As a consequence, we define DELBRANCH(O')and INSBRANCH(O') as costs for insertion and deletion of a branch with syntactic label (7 while none of the bordering nodes is inserted or deleted, respectively. In addition, DELBRANCH'(0")and INSBRANCH'(O')are corresponding costs for branch deletion and insertion when at least one of the bordering nodes has been deleted or inserted, respectively. While the above considerations are addressed to distortions on only syntactic symbols, one has to take into account also distortions on attributes. For deletion and insertion, the costs for attributes are assumed to be included in the costs concerning the syntactic symbol. In the case of node substitution, however, the total costs for the syntactic symbol and the semantic vector are given by SUBNODE ((7, r ) + ~ gilxi--Yil i=I
An element (x, y ) e B is interpreted as a directed edge from node x to node y. Nodes x and y are called the bordering nodes o f the edge. An unattributed relational graph results when the semantic vector is empty, i.e. n = 0 , for each syntactic symbol. Using attributes besides syntactic labels has been proposed by Tsai and Fu (1979). It is a generalization of the formal graph models used by Tsai and Chang (1982), resp. Cheng and Huang (1981). In this paper, we are concerned with distortions on graphs. Any such distortion can be described in terms of basic transformations, namely deletion, insertion, and substitution of nodes and edges. There are costs associated with each basic transformation. We denote the costs for deleting (inserting) a node with syntactic label a e VN by 2,16
where s = (a, (x, . . . . . x , ) )
and t = (r, (y~ . . . . . yn))
are the labels o f the nodes involved and gi are weighting factors. The costs for edge substitution are analogously defined. The costs as well as the weighting factors gi are some means for the user to express a priori knowledge about the problem domain. Low costs should indicate high likelihood for a particular transformation and vice versa. Remark that costs depend only on labels and attributes, not on particular nodes or branches. Costs have to range between zero and infinite where infinitely high costs
Volume 1, Number 4
PATTERN RECOGNITION LETTERS
are to be defined for impossible transformations (with respect to a particular problem domain). When an inexact match between graphs o~ and a~2 is established, three cases can occur: a node n ~ N ~ can be deleted, or a node n t ~N~ can be substituted by a node n_, e N 2, or a node n 2 e N 2 can be inserted. Following is a formal definition for inexact matching taking into account these cases. Formally, we augment both N l and N 2 by the symbol $ for indicating node deletions and insertions. A node in co L, resp. w 2, which is mapped to the symbol $ is deleted from ~ l , resp. inserted in a~2.
;7
T
7
×
Q
×
tM
Q
tJ 2 : 7
b)
o) Fig. 1. Two graphs.
tion and substitution of nodes and branches. Note that mapping nodes by means of f induces a mapping of branches, too. A branch (n, n')~B~ is substituted when there exists a branch
Definition 2. Let
091=(Ni, Bt,l~l,et) and ~2=(N2, B2,I.ta,e2) be attributed relational graphs. Furthermore, let NI=NIU{$},
May 1983
.N'2=N2U{$},
$ ~ N I U N 2.
An inexact match between co~ and co2 is a partial function f:/Vl -* N2 where (l) f ( $ ) ~ $. (2) n ~ n ' = f ( n ) ¢ f ( n ' ) for all n , n ' e N l and
f ( n ) , f ( n ' ) ~ N 2. (3) For each n2eN2, there exists an nl e~rl with f ( n i ) = n 2 and for each n l ~ N l , there exists an n 2 e ~ : 2 with f ( n l ) = n 2. A node nleNl with f ( n l ) = n z e N 2 is substituted by a node n 2 in co2. A node n~eN~ with f - t ( n j ) = = $ is deleted while a node n 2 e N 2 with f - 1(n2) = $ is inserted. The property that f is a partial function is only addressed to the element $ e N1, i.e. when there are no nodes to be inserted in w2 then f ( $ ) is undefined. Condition (3) means that each node in w~ (~o2) must be either substituted or deleted (inserted). Notice that Definition 2 covers inexact matches of any type. Particularly, it contains, as a special case, errorcorrecting isomorphisms introduced by Tsai and Fu (1979).
Definition 3. Let co I and tz)2 be attributed relational graphs and f:.K: l -'/if/2 an inexact match. The costs o f that match, c o s t ( f ) , are obtained by summing up all costs arising from deletion, inser-
(f(n), f ( n ' ) ) e B2; a branch (n, n ' ) e B l is deleted when there is no branch
( f ( n ) , f ( n ' ) ) e B2; a branch (n, n ' ) e B z is inserted when there is no branch
(.f - l ( n ) , f - l(n')) e Bl. Example 1. Two graphs co I and co2 are shown in Figure la,b. An inexact match between m~ and oo2 is given by f1:2~4,
3--5, 1--$.
Let DELNODE(CQ=2
for a e { a , r } ,
SUBNODE(a, r ) = SUBNODE(r, O') = 1, DELBRANCH(X) = 2 ,
DELBRANCH '(X) = l ,
XNSaRANCH(x) = 2, then c o s t ( f 1 ) = 2 + 1 + 1 + 1 = 5 by deleting a node with label a , deleting two edges (where the costs are DELBRANCH'(X) in this case), and substituting one node label. Another inexact match is given, for example, by f2:1--4, 2--5, 3--$.
Here, we have c o s t ( fz)= 6. There are no attributes in this example. 247
Volume 1, Number 4
PATTERN RECOGNITION LETTERS
Definition 4. Let fl . . . . . fn be all inexact matches which are possible between an attributed relational graph co! and co_,. The optimal inexact match f * between col and co2 is defined by c o s t ( f * ) = min {cost(f)}. fl ..... fn
Lemma 5. Let cost(cot, co2) denote the costs of the optimal inexact match between col and co,.; cost(wt, co2) is a metric t if the following holds: (1) SUBNODE (O', O') = SUBBRANCH(O', O') = 0 for all syntactic symbols a, and SUBNODE(a, r), SUBBRANCH(a, r ) > 0 for ere 1., and the costs for any deletion or insertion are greater than zero. (2) SUBNODE(O', 1.) = SUBNODE(1., O'), DELNODE(O') = INSNODE (O'),
DELBRANCH'(O') = INSBRANCH'(O')
for any symbols er, 1.; the analogous conditions hold true for branches. (3) Let c2 and c3 be the costs for two basic transformations yielding the same result as one basic transformation with costs cl, then always Cl~C2+C
3•
The proof is straightforward by independently concluding all metric properties from condition (1)-(3). An example for condition (3) is SUBNODE (O', l") __<SUBNODE(O', ~O) + SUBNODE (Q, l').
Notice that none of the approaches to inexact graph matching cited in the introduction is a metric.
3. Finding the optimal inexact match Referring back to the problem stated in the introduction of finding that prototype col among ml . . . . . con which most closely resembles an input graph co, a straightforward solution is to deter-
1 This means (a) cost(w,, oaz) -> 0 and cost(oat, oa,) = 0 iff oat=w,; (b) cost(w, co,)=cost(oa2, w,. oat); (c) cost(oat, w3) -
May 1983
Inexact graph matching for structural pattern recognition H. B u n k e Lehrstuhl fffr Informatik 5 (Mustererkennung), Universitiit Erlangen-Niirnberg, F.R. Germany
G. A l l e r m a n n Mannesmann Forschungsinstitut, Duisburg, F.R. Germany Received 25 January 1983; revised 10 March 1983
Abstract: This paper is concerned with the inexact matching of attributed, relational graphs for structural pattern recognition. The matching procedure is based on a state space search utilizing heuristic information. Some experimental results are reported.
Key words: Attributed graphs, inexact matching, heuristic search, structural pattern recognition, silhouette recognition.
1. Introduction In the structural approach, patterns are characterized as consisting of sub-patterns among which various relations exist (Fu (1982), Pavlidis (1977)). There are several ways for pattern description and representation in structural pattern recognition. A basic way is by means of strings. Due to the two-dimensionality of pictorial patterns, however, a two-dimensional description formalism seems to be more adequate for certain applications. Consequently, the use of trees for pattern representation has been proposed (Fu and Bhargava (1973)). A natural generalization of trees is graphs. Their use allows the description of complex structures in a straightforward way on a high, problem-oriented level. Besides representation of single patterns, the description of pattern classes is crucial. A method which has gained much importance over the last decade is the use of formal grammars (Fu (1982)). Grammars are favourable when a large number of different pattern classes is involved. At the other hand, there exist cases where the use of a grammar is not feasible, for example, when the set of sample
patterns is too small for the inference of a representative grammar. In this case, the patterns in the sample set can directly be used as prototypes, representing a class each. For the classification a n d / o r description of an unknown pattern, one can match the input pattern against the prototypes. Pattern matching is equivalent to graph matching when the patterns in question are represented by graphs. There are several papers in literature addressing the domain of matching by finding a graph isomorphism, e.g. Corneil and Gottlieb (1970). In practical applications, however, there is noise and distortions causing variation in both the input graph and the prototype. Therefore, our problem turns into a problem of inexact matching. This means given an input graph ~o and a set of prototypes ~ ..... ~ , all possibly different from co, we want to find out which one of the prototypes most closely resembles the input graph. As a consequence, we have to define some measure of similarity on graphs. Several approaches to inexact graph matching have been proposed in literature, e.g. Cheng and Huang (1981), Sanfeliu (1980), Shapiro (1981), Tsai and Fu (1979), Tsai and
0167-8655/83/53.00 ~ 1983, Elsevier Science Publishers B.V. (North-Holland)
245
Volume 1, Number 4
PATTERN RECOGNITION LETTERS
Chang (1982). The approach proposed in this paper is closey related to the work of Tsai and Fu (1979) and Tsai and Chang (1982), i.e. their definition of relational graphs is adopted and matching is based on a state space search. Following are differences: inexact matching is generalized to arbitrary graphs; an extension to attributed graphs is proposed; the cost functions are defined in such a way that the graph distance fulfils the properties of a metric under certain conditions.
2. Basic definitions Let Vs and Vs be a set of node and branch labels, respectively. Each symbol belonging to VN or VB is of the form (s,x) where s is the syntactic c o m p o n e n t and x = (xl . . . . . xn) is a semantic vector consisting of attribute values associated with s. The following definition is due to Tsai and Fu (1979). Definition 1. An a t t r i b u t e d r e l a t i o n a l g r a p h over V= VN U Vs is a 4-tupel ta = (N, B, a, e) where (1) N is a finite, n o n e m p t y set of nodes, (2) B c_N x N is the set of branches, (3) a : N - - , VN is a function for labeling the nodes, (4) c:B--. VB is a function for labeling the edges.
May 1983
DELNODE(0")(INSNODE(0")).The costs for substituting a node with syntactic label a by a node with syntactic label r are given by SUBNODE (e, r). Similarly, SUBBRANCH(a,r) are costs for branch substitution. Defining costs for branch deletion and insertion is slightly more difficult since these transformations are dependent on the bordering nodes. When a node is deleted then all its outgoing and incoming branches vanish, too. Conversely, when inserting a node, there are usually branches to be inserted, too, for properly embedding this node in the rest of the graph. As a consequence, we define DELBRANCH(O')and INSBRANCH(O') as costs for insertion and deletion of a branch with syntactic label (7 while none of the bordering nodes is inserted or deleted, respectively. In addition, DELBRANCH'(0")and INSBRANCH'(O')are corresponding costs for branch deletion and insertion when at least one of the bordering nodes has been deleted or inserted, respectively. While the above considerations are addressed to distortions on only syntactic symbols, one has to take into account also distortions on attributes. For deletion and insertion, the costs for attributes are assumed to be included in the costs concerning the syntactic symbol. In the case of node substitution, however, the total costs for the syntactic symbol and the semantic vector are given by SUBNODE ((7, r ) + ~ gilxi--Yil i=I
An element (x, y ) e B is interpreted as a directed edge from node x to node y. Nodes x and y are called the bordering nodes o f the edge. An unattributed relational graph results when the semantic vector is empty, i.e. n = 0 , for each syntactic symbol. Using attributes besides syntactic labels has been proposed by Tsai and Fu (1979). It is a generalization of the formal graph models used by Tsai and Chang (1982), resp. Cheng and Huang (1981). In this paper, we are concerned with distortions on graphs. Any such distortion can be described in terms of basic transformations, namely deletion, insertion, and substitution of nodes and edges. There are costs associated with each basic transformation. We denote the costs for deleting (inserting) a node with syntactic label a e VN by 2,16
where s = (a, (x, . . . . . x , ) )
and t = (r, (y~ . . . . . yn))
are the labels o f the nodes involved and gi are weighting factors. The costs for edge substitution are analogously defined. The costs as well as the weighting factors gi are some means for the user to express a priori knowledge about the problem domain. Low costs should indicate high likelihood for a particular transformation and vice versa. Remark that costs depend only on labels and attributes, not on particular nodes or branches. Costs have to range between zero and infinite where infinitely high costs
Volume 1, Number 4
PATTERN RECOGNITION LETTERS
are to be defined for impossible transformations (with respect to a particular problem domain). When an inexact match between graphs o~ and a~2 is established, three cases can occur: a node n ~ N ~ can be deleted, or a node n t ~N~ can be substituted by a node n_, e N 2, or a node n 2 e N 2 can be inserted. Following is a formal definition for inexact matching taking into account these cases. Formally, we augment both N l and N 2 by the symbol $ for indicating node deletions and insertions. A node in co L, resp. w 2, which is mapped to the symbol $ is deleted from ~ l , resp. inserted in a~2.
;7
T
7
×
Q
×
tM
Q
tJ 2 : 7
b)
o) Fig. 1. Two graphs.
tion and substitution of nodes and branches. Note that mapping nodes by means of f induces a mapping of branches, too. A branch (n, n')~B~ is substituted when there exists a branch
Definition 2. Let
091=(Ni, Bt,l~l,et) and ~2=(N2, B2,I.ta,e2) be attributed relational graphs. Furthermore, let NI=NIU{$},
May 1983
.N'2=N2U{$},
$ ~ N I U N 2.
An inexact match between co~ and co2 is a partial function f:/Vl -* N2 where (l) f ( $ ) ~ $. (2) n ~ n ' = f ( n ) ¢ f ( n ' ) for all n , n ' e N l and
f ( n ) , f ( n ' ) ~ N 2. (3) For each n2eN2, there exists an nl e~rl with f ( n i ) = n 2 and for each n l ~ N l , there exists an n 2 e ~ : 2 with f ( n l ) = n 2. A node nleNl with f ( n l ) = n z e N 2 is substituted by a node n 2 in co2. A node n~eN~ with f - t ( n j ) = = $ is deleted while a node n 2 e N 2 with f - 1(n2) = $ is inserted. The property that f is a partial function is only addressed to the element $ e N1, i.e. when there are no nodes to be inserted in w2 then f ( $ ) is undefined. Condition (3) means that each node in w~ (~o2) must be either substituted or deleted (inserted). Notice that Definition 2 covers inexact matches of any type. Particularly, it contains, as a special case, errorcorrecting isomorphisms introduced by Tsai and Fu (1979).
Definition 3. Let co I and tz)2 be attributed relational graphs and f:.K: l -'/if/2 an inexact match. The costs o f that match, c o s t ( f ) , are obtained by summing up all costs arising from deletion, inser-
(f(n), f ( n ' ) ) e B2; a branch (n, n ' ) e B l is deleted when there is no branch
( f ( n ) , f ( n ' ) ) e B2; a branch (n, n ' ) e B z is inserted when there is no branch
(.f - l ( n ) , f - l(n')) e Bl. Example 1. Two graphs co I and co2 are shown in Figure la,b. An inexact match between m~ and oo2 is given by f1:2~4,
3--5, 1--$.
Let DELNODE(CQ=2
for a e { a , r } ,
SUBNODE(a, r ) = SUBNODE(r, O') = 1, DELBRANCH(X) = 2 ,
DELBRANCH '(X) = l ,
XNSaRANCH(x) = 2, then c o s t ( f 1 ) = 2 + 1 + 1 + 1 = 5 by deleting a node with label a , deleting two edges (where the costs are DELBRANCH'(X) in this case), and substituting one node label. Another inexact match is given, for example, by f2:1--4, 2--5, 3--$.
Here, we have c o s t ( fz)= 6. There are no attributes in this example. 247
Volume 1, Number 4
PATTERN RECOGNITION LETTERS
Definition 4. Let fl . . . . . fn be all inexact matches which are possible between an attributed relational graph co! and co_,. The optimal inexact match f * between col and co2 is defined by c o s t ( f * ) = min {cost(f)}. fl ..... fn
Lemma 5. Let cost(cot, co2) denote the costs of the optimal inexact match between col and co,.; cost(wt, co2) is a metric t if the following holds: (1) SUBNODE (O', O') = SUBBRANCH(O', O') = 0 for all syntactic symbols a, and SUBNODE(a, r), SUBBRANCH(a, r ) > 0 for ere 1., and the costs for any deletion or insertion are greater than zero. (2) SUBNODE(O', 1.) = SUBNODE(1., O'), DELNODE(O') = INSNODE (O'),
DELBRANCH'(O') = INSBRANCH'(O')
for any symbols er, 1.; the analogous conditions hold true for branches. (3) Let c2 and c3 be the costs for two basic transformations yielding the same result as one basic transformation with costs cl, then always Cl~C2+C
3•
The proof is straightforward by independently concluding all metric properties from condition (1)-(3). An example for condition (3) is SUBNODE (O', l") __<SUBNODE(O', ~O) + SUBNODE (Q, l').
Notice that none of the approaches to inexact graph matching cited in the introduction is a metric.
3. Finding the optimal inexact match Referring back to the problem stated in the introduction of finding that prototype col among ml . . . . . con which most closely resembles an input graph co, a straightforward solution is to deter-
1 This means (a) cost(w,, oaz) -> 0 and cost(oat, oa,) = 0 iff oat=w,; (b) cost(w, co,)=cost(oa2, w,. oat); (c) cost(oat, w3) -
