On the Hardness of Approximating the Minimum ... - Semantic Scholar
On the Hardness of Approximating the Minimum Consistent OBDD Problem Kouichi Hirata1
Shinichi Shimozono2
Ayumi Shinohara3
1
2
Department of Arti cial Intelligence Department of Control Engineering and Science Kyushu Institute of Technology, Kawazu 680-4, Iizuka, 820 JAPAN
E-mail: [email protected], [email protected] 3
Research Institute of Fundamental Information Science Kyushu University, Fukuoka 812 JAPAN E-mail: [email protected]
Abstract. Ordered binary decision diagrams (OBDD, for short) represent Boolean functions as directed acyclic graphs. The minimum consistent OBDD problem is, given an incomplete truth table of a function, to nd the smallest OBDD that is consistent with the truth table with respect to a xed variable ordering. We show that this problem is NP-hard, and prove that there is a constant > 0 such that no polynomial time algorithm can approximate the minimum consistent OBDD within the ratio n unless P=NP, where n is the number of variables. This result suggests that OBDDs are unlikely to be polynomial time learnable in PAC-learning model. Furthermore, we give a polynomial time learnable subclass of OBDDs representing symmetric functions.
1 Introduction For a class of representations of languages, the minimum consistent problem is to nd a representation that is as small size as possible and is consistent with given positive and negative examples. The computational complexity of the problem closely relates to the eciency of learning with the target class. Intuitively, the polynomial-time learnability of a class is provided by the explicit or inplicit existence of a polynomial-time algorithm which can produce a consistent representation whose size is smallest, or at most polynomial times the smallest one. Therefore, for the minimum consistent problem that is intractable, a polynomial-time algorithm that can nd an approximately small representation is undoubtely important. One of the most frequenlty studied minimum consistent problems is that for deterministic nite automata (DFAs for short). The problem was rst shown
to be NP-hard [1, 13]. This negative result was enhanced to the lower bound on approximation ratios due to Li and Vazirani [15]. They showed that, unless P=NP, the minimum consistent DFA cannot be approximated within the ratio 9 in polynomial time. Pitt and Warmuth [18] improved to the ratio opt k , where 8 opt is the minimum number of states and k is any positive integer. Hancock et al. [14] investigated the minimum consistent problem for decision lists and decision trees that represent n-ary Boolean functions. They showed that decision lists cannot be approximated in polynomial time within a factor of nc for some constant c > 0, unless P=NP. They also showed that decision trees cannot be approximated in polynomial time within a factor of nc for any c > 0 unless NP is included in DTIME[2poly log n]. This paper deals with the minimum consistent problem for ordered binary decision diagrams. An ordered binary decision diagram (OBDD for short) [9] represents a Boolean function as a directed acyclic graph whose internal nodes correspond to the input variables and terminal nodes indicate the output values. Many useful Boolean functions, such as symmetric functions and threshold functions, can be expressed succinctly by OBDDs [9]. As another important aspect from machine learning, OBDDs can be considered as acyclic DFAs whose size are counted by only the brancing state [12]. The size of an OBDD strongly depends on the order of variables [9], and it is known that nding the optimal order that realizes the minimum size OBDD representing the function is intractable [8]. However, to concentrate on the hardness of the minimum consistent problem of OBDDs, we x the order of variables with respect to the order of bits in input strings, and discuss the complexity to deal with an incomplete truth table. Our non-approximability result on the minimum consistent OBDD problem is in sharp contrast to the well-known fact that the minimum size OBDD representing the function can be computed from the complete truth table in linear time with respect to the size of it [20]. This paper is organized as follows: First, we show that the minimum consistent OBDD problem is NP-hard, even the number of positive examples is only one. Then, we show that there is a constant > 0 such that no polynomial time algorithm can approximate the minimum consistent OBDD within the ratio n unless P=NP. The proof employed in our result does not use cyclic transitions, which is the essence of the hardness for DFA in [18]. Our results suggest that any ecient algorithm for learning OBDDs would have to produce very large hypotheses.
2 Minimum Consistent OBDD Problem We rst give de nitions and notations for OBDDs. Then we introduce the minimum consistent problem of OBDD as a combinatorial optimization problem, and show that the problem is NP-hard. A binary decision diagram over a set X = fx1 ; : : : ; xn g of Boolean variables is a rooted, directed acyclic graph whose nodes are labeled with variables in X or Boolean constants, 0 or 1. A node labeled with a variable is called an internal node and has two outgoing arcs that are labeled with 0 and 1, respectively. A node labeled with 1 or 0 is called a terminal and has no outgoing arcs. The root node is the only node with no incoming arcs. A binary decision diagram D over n variables represents a Boolan function as following way. An input w = w[1] 1 1 1 w [n] 2 f0; 1gn determins a computation path on D from the root to a terminal: it starts from the root, and if a reached node is labeled with xi then it follows the arc labeled with w[i]. Then the output of D for w is the label of the terminal reached by the computation path for w . We call terminals labeled with 1 and 0 by 1-terminal and 0-terminal, respectively. Let be an order on X . We say a binary decision diagram D is an ordered binary decision diagram (OBDD for short) with respect to if on every path from the root to a terminal the variables are tested according to only once. In this paper, we deal with the order x1 < 1 1 1 < xn . We consider the following problem:
Minimum Consistent OBDD Two disjoint sets P; N f0; 1gn . The strings in P
De nition 1. :
Instance
Solution
and N are called positive examples and negative exampls, respectively.
: An OBDD D over n variables that is consistent with P and N , i.e., outputs 1 for all positive examples and 0 for all negative examples.
Cost: The size jD j of D , i.e., the number of internal nodes of D . The goal is to nd a solution whose cost is the minimum.
For P and N provided as to form a complete truth table P [ N = f0; 1gn , a minimum consistent OBDD can be computed in polynomial time by constructing an OBDD like a complete binary tree and then applying to it the reduction procedure introduced by Bryant [9, 20]. This is one of the reasons why OBDDs are conveniently used in circuit designs of well-de ned Boolean functions. However, the problem becomes intractable for incomplete examples:
Lemma 2.
Minimum Consistent OBDD
is NP-hard, even either the set of
positive examples or the set of negative examples consists of only one string. Proof. This can be shown by a log-space reduction involving the translation of instances from Minimum Cover [11] to sets P; N of strings presented by Hancock et al. [14]. Minimum Cover is, given a collection C of subsets over U , to nd a subcollection C 0 C that covers U and is as small as possible. Given an instance C , construct P; N as follows: (i) For every i 2 U , P includes the string which is obtained by placing 1s on 0n at all positions k for ck 2 C such that i 2 ck , and (ii) N consists of only one string 0n . Then by an OBDD D consistent with P and N we obtain a cover C 0 = fck 2 C j xk 2 X 0 g for U with the size jC 0 j jD j, where X 0 X is the set of variables labeled on the nodes on the computation path for 0n in D . Notince that only the nodes in X 0 are relevant to the computations for inputs in P [ N . 2
3 Approximate the Minimum Consistent OBDD Since Minimum Consistent OBDD problem is intractable, it is natural to ask whether there is an ecient approximation algorithm that produces reasonably small OBDDs for given examples. Deviations of approximate solutions from the optimum solution can be measured by the ratio of their costs. For Minimum Consistent OBDD problem, the performance ratio of a solution D with respect to an instance P; N is the ratio jjDD3jj , where D 3 is a minimum OBDD consistent with P and N . The guaranteed performance ratio of an approximation algorithm is the ratio r such that, for any instance, the algorithm always produces an OBDD D satisfying jjDD3jj r . In PAC learning criteria, it is crucial whether there is a polynomial-time algorithm that achieves the performance ratio bounded by a constant or a slowly growing function, since Occam's razor requires inequality r
n (jP j + jN j)
holds if the target domain is PAC learnable. Notice that the reduction referred in Lemma 2 preserves performance ratios between Minimum Cover and Minimum Consistent OBDD. Hence, even for an instance with either only one positive or one negative example, we do not have any polynomial-time algorithm whose performance ratio is better than O(log n), unless NP DTIME[npoly log n ] [16]. The aim of this section is to show that this
lower bound can be ampli ed by a reduction involving many both positive and negative examples. To obtain our result, we invoke the non-approximability of Graph Coloring (Chromatic Number) [11, 16]. Let G = (V; E ) be an undirected graph. We say that G is k-colorable if there is a k-coloring of G, that is, a mapping f : V ! f1; : : : ; k g such that f (u) 6= f (v ) whenever (u; v ) 2 E . The chromatic number of G is the minimum number k such that G is k-colorable. We denote the chromatic number of G by K (G). Then the problem Graph Coloring is, given a graph G, to nd a k -coloring for G such that k is as small as possible. It is known in [16] that there is a constant c > 0 such that Graph Coloring cannot be approximated within the ratio nc unless P=NP. Thus, there is c > 0 such that no polynomial-time algorithm can always nd a k -coloring satisfying k=K (G) < nc for all G. Let us start with introducing a translation from a graph to sets of strings for dealing with OBDDs. Let G = (V; E ) be a graph with the set V = f1; : : : ; ng of nodes. For each node i 2 V , we de ne the adjacency sets Pi and Ni as follows: (i) Pi consists of 1n and pij = 1j 01 0 1n0j for every (i; j ) 2 E , and (ii) Ni consists of only one string qi = 1i01 0 1n0i . Note that, for any i 2 V , Pi and Ni are disjoint. S S Let U be a subset of V . We denote the union i2U Pi by PU and i2U Ni by NU . We identify a mapping f : V ! f1; : : : ; kg with the partition of nodes U1 ; : : : ; Uk V de ned by Ui = fv 2 V j f (v ) = ig.
Lemma 3. and
NUi
A partition
U1 ; : : : ; U k
are disjoint for all
of
V
1 i k.
is a
k-coloring
of
G
if and only if
PUi
Proof. This holds since there is (i; j ) 2 E with i; j 2 Uh for some 1 h k if and only if there are the same strings pij 2 Pi and qj 2 Nj .
Lemma 4.
Let
G
= (V; E )
the adjacency sets for
PU
1n .
and
NU
U
has exactly
be a graph and let
V jU j
of
G.
PU
and
NU
be disjoint unions of
Then the minimum OBDD consistent with
nodes all of which are on the computation path for
Every string in PU [ NU except 1n includes exactly one 0. This implies that any internal node can separate either one negative string or one positive string from 1n and other strings. Since there are jU j negative strings in NU , the minimum OBDD must have exactly jU j nodes. Thus the lemma holds. Proof.
The translation of graphs to strings presented here is di erent from those in the reductions to other problem, such as k -term DNF and k -Decision List [14, 17]. This is because OBDDs can count easily the number of 0s (or 1s) in input strings.
Theorem 5.
OBDD
There is a constant
0
<
Shinichi Shimozono2
Ayumi Shinohara3
1
2
Department of Arti cial Intelligence Department of Control Engineering and Science Kyushu Institute of Technology, Kawazu 680-4, Iizuka, 820 JAPAN
E-mail: [email protected], [email protected] 3
Research Institute of Fundamental Information Science Kyushu University, Fukuoka 812 JAPAN E-mail: [email protected]
Abstract. Ordered binary decision diagrams (OBDD, for short) represent Boolean functions as directed acyclic graphs. The minimum consistent OBDD problem is, given an incomplete truth table of a function, to nd the smallest OBDD that is consistent with the truth table with respect to a xed variable ordering. We show that this problem is NP-hard, and prove that there is a constant > 0 such that no polynomial time algorithm can approximate the minimum consistent OBDD within the ratio n unless P=NP, where n is the number of variables. This result suggests that OBDDs are unlikely to be polynomial time learnable in PAC-learning model. Furthermore, we give a polynomial time learnable subclass of OBDDs representing symmetric functions.
1 Introduction For a class of representations of languages, the minimum consistent problem is to nd a representation that is as small size as possible and is consistent with given positive and negative examples. The computational complexity of the problem closely relates to the eciency of learning with the target class. Intuitively, the polynomial-time learnability of a class is provided by the explicit or inplicit existence of a polynomial-time algorithm which can produce a consistent representation whose size is smallest, or at most polynomial times the smallest one. Therefore, for the minimum consistent problem that is intractable, a polynomial-time algorithm that can nd an approximately small representation is undoubtely important. One of the most frequenlty studied minimum consistent problems is that for deterministic nite automata (DFAs for short). The problem was rst shown
to be NP-hard [1, 13]. This negative result was enhanced to the lower bound on approximation ratios due to Li and Vazirani [15]. They showed that, unless P=NP, the minimum consistent DFA cannot be approximated within the ratio 9 in polynomial time. Pitt and Warmuth [18] improved to the ratio opt k , where 8 opt is the minimum number of states and k is any positive integer. Hancock et al. [14] investigated the minimum consistent problem for decision lists and decision trees that represent n-ary Boolean functions. They showed that decision lists cannot be approximated in polynomial time within a factor of nc for some constant c > 0, unless P=NP. They also showed that decision trees cannot be approximated in polynomial time within a factor of nc for any c > 0 unless NP is included in DTIME[2poly log n]. This paper deals with the minimum consistent problem for ordered binary decision diagrams. An ordered binary decision diagram (OBDD for short) [9] represents a Boolean function as a directed acyclic graph whose internal nodes correspond to the input variables and terminal nodes indicate the output values. Many useful Boolean functions, such as symmetric functions and threshold functions, can be expressed succinctly by OBDDs [9]. As another important aspect from machine learning, OBDDs can be considered as acyclic DFAs whose size are counted by only the brancing state [12]. The size of an OBDD strongly depends on the order of variables [9], and it is known that nding the optimal order that realizes the minimum size OBDD representing the function is intractable [8]. However, to concentrate on the hardness of the minimum consistent problem of OBDDs, we x the order of variables with respect to the order of bits in input strings, and discuss the complexity to deal with an incomplete truth table. Our non-approximability result on the minimum consistent OBDD problem is in sharp contrast to the well-known fact that the minimum size OBDD representing the function can be computed from the complete truth table in linear time with respect to the size of it [20]. This paper is organized as follows: First, we show that the minimum consistent OBDD problem is NP-hard, even the number of positive examples is only one. Then, we show that there is a constant > 0 such that no polynomial time algorithm can approximate the minimum consistent OBDD within the ratio n unless P=NP. The proof employed in our result does not use cyclic transitions, which is the essence of the hardness for DFA in [18]. Our results suggest that any ecient algorithm for learning OBDDs would have to produce very large hypotheses.
2 Minimum Consistent OBDD Problem We rst give de nitions and notations for OBDDs. Then we introduce the minimum consistent problem of OBDD as a combinatorial optimization problem, and show that the problem is NP-hard. A binary decision diagram over a set X = fx1 ; : : : ; xn g of Boolean variables is a rooted, directed acyclic graph whose nodes are labeled with variables in X or Boolean constants, 0 or 1. A node labeled with a variable is called an internal node and has two outgoing arcs that are labeled with 0 and 1, respectively. A node labeled with 1 or 0 is called a terminal and has no outgoing arcs. The root node is the only node with no incoming arcs. A binary decision diagram D over n variables represents a Boolan function as following way. An input w = w[1] 1 1 1 w [n] 2 f0; 1gn determins a computation path on D from the root to a terminal: it starts from the root, and if a reached node is labeled with xi then it follows the arc labeled with w[i]. Then the output of D for w is the label of the terminal reached by the computation path for w . We call terminals labeled with 1 and 0 by 1-terminal and 0-terminal, respectively. Let be an order on X . We say a binary decision diagram D is an ordered binary decision diagram (OBDD for short) with respect to if on every path from the root to a terminal the variables are tested according to only once. In this paper, we deal with the order x1 < 1 1 1 < xn . We consider the following problem:
Minimum Consistent OBDD Two disjoint sets P; N f0; 1gn . The strings in P
De nition 1. :
Instance
Solution
and N are called positive examples and negative exampls, respectively.
: An OBDD D over n variables that is consistent with P and N , i.e., outputs 1 for all positive examples and 0 for all negative examples.
Cost: The size jD j of D , i.e., the number of internal nodes of D . The goal is to nd a solution whose cost is the minimum.
For P and N provided as to form a complete truth table P [ N = f0; 1gn , a minimum consistent OBDD can be computed in polynomial time by constructing an OBDD like a complete binary tree and then applying to it the reduction procedure introduced by Bryant [9, 20]. This is one of the reasons why OBDDs are conveniently used in circuit designs of well-de ned Boolean functions. However, the problem becomes intractable for incomplete examples:
Lemma 2.
Minimum Consistent OBDD
is NP-hard, even either the set of
positive examples or the set of negative examples consists of only one string. Proof. This can be shown by a log-space reduction involving the translation of instances from Minimum Cover [11] to sets P; N of strings presented by Hancock et al. [14]. Minimum Cover is, given a collection C of subsets over U , to nd a subcollection C 0 C that covers U and is as small as possible. Given an instance C , construct P; N as follows: (i) For every i 2 U , P includes the string which is obtained by placing 1s on 0n at all positions k for ck 2 C such that i 2 ck , and (ii) N consists of only one string 0n . Then by an OBDD D consistent with P and N we obtain a cover C 0 = fck 2 C j xk 2 X 0 g for U with the size jC 0 j jD j, where X 0 X is the set of variables labeled on the nodes on the computation path for 0n in D . Notince that only the nodes in X 0 are relevant to the computations for inputs in P [ N . 2
3 Approximate the Minimum Consistent OBDD Since Minimum Consistent OBDD problem is intractable, it is natural to ask whether there is an ecient approximation algorithm that produces reasonably small OBDDs for given examples. Deviations of approximate solutions from the optimum solution can be measured by the ratio of their costs. For Minimum Consistent OBDD problem, the performance ratio of a solution D with respect to an instance P; N is the ratio jjDD3jj , where D 3 is a minimum OBDD consistent with P and N . The guaranteed performance ratio of an approximation algorithm is the ratio r such that, for any instance, the algorithm always produces an OBDD D satisfying jjDD3jj r . In PAC learning criteria, it is crucial whether there is a polynomial-time algorithm that achieves the performance ratio bounded by a constant or a slowly growing function, since Occam's razor requires inequality r
n (jP j + jN j)
holds if the target domain is PAC learnable. Notice that the reduction referred in Lemma 2 preserves performance ratios between Minimum Cover and Minimum Consistent OBDD. Hence, even for an instance with either only one positive or one negative example, we do not have any polynomial-time algorithm whose performance ratio is better than O(log n), unless NP DTIME[npoly log n ] [16]. The aim of this section is to show that this
lower bound can be ampli ed by a reduction involving many both positive and negative examples. To obtain our result, we invoke the non-approximability of Graph Coloring (Chromatic Number) [11, 16]. Let G = (V; E ) be an undirected graph. We say that G is k-colorable if there is a k-coloring of G, that is, a mapping f : V ! f1; : : : ; k g such that f (u) 6= f (v ) whenever (u; v ) 2 E . The chromatic number of G is the minimum number k such that G is k-colorable. We denote the chromatic number of G by K (G). Then the problem Graph Coloring is, given a graph G, to nd a k -coloring for G such that k is as small as possible. It is known in [16] that there is a constant c > 0 such that Graph Coloring cannot be approximated within the ratio nc unless P=NP. Thus, there is c > 0 such that no polynomial-time algorithm can always nd a k -coloring satisfying k=K (G) < nc for all G. Let us start with introducing a translation from a graph to sets of strings for dealing with OBDDs. Let G = (V; E ) be a graph with the set V = f1; : : : ; ng of nodes. For each node i 2 V , we de ne the adjacency sets Pi and Ni as follows: (i) Pi consists of 1n and pij = 1j 01 0 1n0j for every (i; j ) 2 E , and (ii) Ni consists of only one string qi = 1i01 0 1n0i . Note that, for any i 2 V , Pi and Ni are disjoint. S S Let U be a subset of V . We denote the union i2U Pi by PU and i2U Ni by NU . We identify a mapping f : V ! f1; : : : ; kg with the partition of nodes U1 ; : : : ; Uk V de ned by Ui = fv 2 V j f (v ) = ig.
Lemma 3. and
NUi
A partition
U1 ; : : : ; U k
are disjoint for all
of
V
1 i k.
is a
k-coloring
of
G
if and only if
PUi
Proof. This holds since there is (i; j ) 2 E with i; j 2 Uh for some 1 h k if and only if there are the same strings pij 2 Pi and qj 2 Nj .
Lemma 4.
Let
G
= (V; E )
the adjacency sets for
PU
1n .
and
NU
U
has exactly
be a graph and let
V jU j
of
G.
PU
and
NU
be disjoint unions of
Then the minimum OBDD consistent with
nodes all of which are on the computation path for
Every string in PU [ NU except 1n includes exactly one 0. This implies that any internal node can separate either one negative string or one positive string from 1n and other strings. Since there are jU j negative strings in NU , the minimum OBDD must have exactly jU j nodes. Thus the lemma holds. Proof.
The translation of graphs to strings presented here is di erent from those in the reductions to other problem, such as k -term DNF and k -Decision List [14, 17]. This is because OBDDs can count easily the number of 0s (or 1s) in input strings.
Theorem 5.
OBDD
There is a constant
0
<
