The Computational Complexity of Probabilistic ... - Semantic Scholar
The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks Flemming Jensen
October 2008
1 / 72
Purpose
To show that probabilistic inference using multiply connected Bayesian networks is N P-hard.
2 / 72
Multiply Connected Bayesian Networks Multiply connected networks contains at least one pair of variables that have more than one undirected path connecting them.
3 / 72
Multiply Connected Bayesian Networks Multiply connected networks contains at least one pair of variables that have more than one undirected path connecting them. Example:
A
B E
C F
D G
H
4 / 72
Representation of Bayesian Networks
We represent a Bayesian network BN as a triple BN = (V , A, P), where V is the set of variables, A the set of arcs between variables, and P the set of probabilities.
5 / 72
Representation of Bayesian Networks
We represent a Bayesian network BN as a triple BN = (V , A, P), where V is the set of variables, A the set of arcs between variables, and P the set of probabilities. We assume that sp(Vi ) = {T , F } for all Vi ∈ V .
6 / 72
Probabilistic Inference
We focus on the form of probabilistic inference needed to determine the probability P(Y = T ) for some variable Y .
7 / 72
Probabilistic Inference
We focus on the form of probabilistic inference needed to determine the probability P(Y = T ) for some variable Y . By proving that computing P(Y = T ) is N P-hard we will prove that other more general forms of probabilistic inference are N P-hard as well.
8 / 72
Proof Idea
We define PIBNET as the probabilistic inference function problem of determining P(Y = T ) for some variable Y in a Bayesian network.
9 / 72
Proof Idea
We define PIBNET as the probabilistic inference function problem of determining P(Y = T ) for some variable Y in a Bayesian network. We define PIBNETD as a decision problem version of PIBNET , that for some variable Y determines whether P(Y = T ) > 0 and returns ”yes” if so and ”no” otherwise.
10 / 72
Proof Idea
We define PIBNET as the probabilistic inference function problem of determining P(Y = T ) for some variable Y in a Bayesian network. We define PIBNETD as a decision problem version of PIBNET , that for some variable Y determines whether P(Y = T ) > 0 and returns ”yes” if so and ”no” otherwise. We prove that PIBNET is N P-hard by giving a polynomial time reduction from the well known N P-complete problem 3SAT to PIBNETD. A polynomial time reduction from PIBNETD to PIBNET concludes the proof.
11 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables.
12 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U.
13 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U. The literal ui is true iff the variable u is true (T), and the literal ¬ui is true iff the variable u is false (F).
14 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U. The literal ui is true iff the variable u is true (T), and the literal ¬ui is true iff the variable u is false (F). Each clause ci contains a disjunction of three literals over U.
15 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U. The literal ui is true iff the variable u is true (T), and the literal ¬ui is true iff the variable u is false (F). Each clause ci contains a disjunction of three literals over U. A set C of clauses over U is satisfiable iff there exists some truth assignment for U that satisfies all the clauses in C .
16 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U. The literal ui is true iff the variable u is true (T), and the literal ¬ui is true iff the variable u is false (F). Each clause ci contains a disjunction of three literals over U. A set C of clauses over U is satisfiable iff there exists some truth assignment for U that satisfies all the clauses in C . The 3SAT problem determines whether a set C of clauses over a set of variables U is satisfiable. 17 / 72
3SAT
Example of a 3SAT instance: U = {u1 , u2 , u3 , u4 } C = {(u1 ∨ u2 ∨ u3 ), (¬u1 ∨ ¬u2 ∨ u3 ), (u2 ∨ ¬u3 ∨ u4 )} 3SAT would answer ”yes” in this situation using, for example, the truth assignments: u1 = F , u2 = T , u3 = T , and u4 = F .
18 / 72
Proof - from 3SAT to PIBNETD
Let U = {u1 , u2 , · · · , un } and C = {c1 , c2 , · · · , cm } be any instance of 3SAT .
19 / 72
Proof - from 3SAT to PIBNETD
Let U = {u1 , u2 , · · · , un } and C = {c1 , c2 , · · · , cm } be any instance of 3SAT . We construct a BN containing variable Y such that: P(T = Y ) > 0 ⇐⇒ C is satisfiable.
20 / 72
Proof - from 3SAT to PIBNETD
The BN will consists of three components:
21 / 72
Proof - from 3SAT to PIBNETD
The BN will consists of three components: A truth-setting component - sets the truth values of U.
22 / 72
Proof - from 3SAT to PIBNETD
The BN will consists of three components: A truth-setting component - sets the truth values of U. A clause-satisfaction-testing component - tests whether each clause is satisfied.
23 / 72
Proof - from 3SAT to PIBNETD
The BN will consists of three components: A truth-setting component - sets the truth values of U. A clause-satisfaction-testing component - tests whether each clause is satisfied. An overall-satisfaction-testing component - tests whether C is satisfied.
24 / 72
Proof - from 3SAT to PIBNETD Given the 3SAT instance from the example before: U = {u1 , u2 , u3 , u4 } C = {(u1 ∨ u2 ∨ u3 ), (¬u1 ∨ ¬u2 ∨ u3 ), (u2 ∨ ¬u3 ∨ u4 )} we want to end up with a Bayesian network like this:
u1
u2 C1
u3 C2 Y
u4 C3
}
}
}
truth-setting component clause-satisfaction-testing component
overall-satisfaction-testing component
25 / 72
Proof - from 3SAT to PIBNETD
The truth-setting component (V t , P t ) is constructed as:
26 / 72
Proof - from 3SAT to PIBNETD
The truth-setting component (V t , P t ) is constructed as: V t = U.
27 / 72
Proof - from 3SAT to PIBNETD
The truth-setting component (V t , P t ) is constructed as: V t = U. P t = {P(u1 = T ) = 12 , P(u2 = T ) = 21 , · · · , P(un = T ) = 12 }.
28 / 72
Proof - from 3SAT to PIBNETD The clause-satisfaction-testing component (V s , As , P s ) is the union of m subcomponents; one for each clause ci ∈ C . Each subcomponent (Vis , Asi , Pis ) is constructed as:
29 / 72
Proof - from 3SAT to PIBNETD The clause-satisfaction-testing component (V s , As , P s ) is the union of m subcomponents; one for each clause ci ∈ C . Each subcomponent (Vis , Asi , Pis ) is constructed as: Vis = {wi1 , wi2 , wi3 , Ci }, where wi1 , wi2 , and wi3 are the variables corresponding to the literals in clause ci , and the variable Ci represents the truth value of clause ci .
30 / 72
Proof - from 3SAT to PIBNETD The clause-satisfaction-testing component (V s , As , P s ) is the union of m subcomponents; one for each clause ci ∈ C . Each subcomponent (Vis , Asi , Pis ) is constructed as: Vis = {wi1 , wi2 , wi3 , Ci }, where wi1 , wi2 , and wi3 are the variables corresponding to the literals in clause ci , and the variable Ci represents the truth value of clause ci . Asi = {(wi1 , Ci ), (wi2 , Ci ), (wi3 , Ci )}.
31 / 72
Proof - from 3SAT to PIBNETD The clause-satisfaction-testing component (V s , As , P s ) is the union of m subcomponents; one for each clause ci ∈ C . Each subcomponent (Vis , Asi , Pis ) is constructed as: Vis = {wi1 , wi2 , wi3 , Ci }, where wi1 , wi2 , and wi3 are the variables corresponding to the literals in clause ci , and the variable Ci represents the truth value of clause ci . Asi = {(wi1 , Ci ), (wi2 , Ci ), (wi3 , Ci )}. Pis = {P(Ci = T |πci )}, where πci represents the conjunction of the three variables and 1 if gi (πci ) = T , P(Ci = T |πci ) = 0 if gi (πci ) = F , where gi is the truth function for clause ci . 32 / 72
Proof - from 3SAT to PIBNETD The overall-satisfaction-testing component (V o , Ao , P o ) is constructed as:
33 / 72
Proof - from 3SAT to PIBNETD The overall-satisfaction-testing component (V o , Ao , P o ) is constructed as: V o = {Y }.
34 / 72
Proof - from 3SAT to PIBNETD The overall-satisfaction-testing component (V o , Ao , P o ) is constructed as: V o = {Y }. Ao = {(C1 , Y ), (C2 , Y ), · · · , (Cm , Y )}.
35 / 72
Proof - from 3SAT to PIBNETD The overall-satisfaction-testing component (V o , Ao , P o ) is constructed as: V o = {Y }. Ao = {(C1 , Y ), (C2 , Y ), · · · , (Cm , Y )}. P o = {P(Y = T |C1 , C2 , · · · , Cm )}, where P(Y = T |C1 , C2 , · · · , Cm ) =
1 if C1 ∧ C2 ∧ · · · ∧ Cm = T , 0 if otherwise.
36 / 72
Proof - from 3SAT to PIBNETD When the number of clauses increase so does the number of parents of the node Y . u1
u2 C1
u3 C2
u4 C3
Y
37 / 72
Proof - from 3SAT to PIBNETD When the number of clauses increase so does the number of parents of the node Y . u1
u2 C1
u3 C2
u4 C3
Y
To avoid having a node with many parents, dummy variables functioning as AND-gates, can be inserted. u1
u2
u3
u4
C1
C2
C3
D1
D2
Y
38 / 72
Proof - from 3SAT to PIBNETD When the number of clauses increase so does the number of parents of the node Y . u1
u2 C1
u3 C2
u4 C3
Y
To avoid having a node with many parents, dummy variables functioning as AND-gates, can be inserted. u1
u2
u3
u4
C1
C2
C3
D1
D2
Y
The use of dummy variables keeps the indegree of any node in the network below four. 39 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po
40 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po The time complexity of constructing the Bayesian network is polynomial in the size of the 3SAT instance:
41 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po The time complexity of constructing the Bayesian network is polynomial in the size of the 3SAT instance: O(n) - the construction of the truth-setting component
42 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po The time complexity of constructing the Bayesian network is polynomial in the size of the 3SAT instance: O(n) - the construction of the truth-setting component O(m) - the construction of the clause-satisfaction-testing component
43 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po The time complexity of constructing the Bayesian network is polynomial in the size of the 3SAT instance: O(n) - the construction of the truth-setting component O(m) - the construction of the clause-satisfaction-testing component O(m) - the construction of the overall-satisfaction-testing component 44 / 72
Proof - from 3SAT to PIBNETD
We now only need to show that: P(T = Y ) > 0 ⇐⇒ C is satisfiable.
45 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T .
46 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T . For example U5 represents the instantiation: u4 = F , u3 = T , u2 = F , and u1 = T , because the binary representation of 5 is 0101.
47 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T . For example U5 represents the instantiation: u4 = F , u3 = T , u2 = F , and u1 = T , because the binary representation of 5 is 0101. Let Cβ denote an instantiation of the variables in C similar to Uα .
48 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T . For example U5 represents the instantiation: u4 = F , u3 = T , u2 = F , and u1 = T , because the binary representation of 5 is 0101. Let Cβ denote an instantiation of the variables in C similar to Uα . Let Us denote a truth assignment to the variables in U that satisfies all the clauses in C .
49 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T . For example U5 represents the instantiation: u4 = F , u3 = T , u2 = F , and u1 = T , because the binary representation of 5 is 0101. Let Cβ denote an instantiation of the variables in C similar to Uα . Let Us denote a truth assignment to the variables in U that satisfies all the clauses in C . Let Cs denote the set of instantiated variables that results from assigning the value T to each variable Ci . 50 / 72
Proof - from 3SAT to PIBNETD We can compute P(Y = T ) as: P(Y = T ) =
n −1 2m −1 2X X
P(Y = T |Cβ )P(Cβ |Uα )P(Uα )
α=0 β=0
51 / 72
Proof - from 3SAT to PIBNETD We can compute P(Y = T ) as: P(Y = T ) =
n −1 2m −1 2X X
P(Y = T |Cβ )P(Cβ |Uα )P(Uα )
α=0 β=0
Now we assume that C is satisfiable.
52 / 72
Proof - from 3SAT to PIBNETD We can compute P(Y = T ) as: P(Y = T ) =
n −1 2m −1 2X X
P(Y = T |Cβ )P(Cβ |Uα )P(Uα )
α=0 β=0
Now we assume that C is satisfiable. Since C is satisfiable, one term of the sum is P(Y = T |Cs )P(Cs |Us )P(Us ) and we have that: P(Y = T ) ≥ P(Y = T |Cs )P(Cs |Us )P(Us )
53 / 72
Proof - from 3SAT to PIBNETD We can compute P(Y = T ) as: P(Y = T ) =
n −1 2m −1 2X X
P(Y = T |Cβ )P(Cβ |Uα )P(Uα )
α=0 β=0
Now we assume that C is satisfiable. Since C is satisfiable, one term of the sum is P(Y = T |Cs )P(Cs |Us )P(Us ) and we have that: P(Y = T ) ≥ P(Y = T |Cs )P(Cs |Us )P(Us ) We know that P(Y = T |Cs ) = 1 and P(Us ) = ( 12 )n so we only need to show that P(Cs |Us ) > 0. 54 / 72
Proof - from 3SAT to PIBNETD P(Cs |Us ) can be expressed as: 1 , w 2 , w 3 ), P(Cs |Us ) = P(C1 = T |w11 , w12 , w13 ) · · · P(Cm = T |wm m m
where wi1 , wi2 , and wi3 denote the instantiation of the variables wi1 , wi2 , and wi3 as designated by Us .
55 / 72
Proof - from 3SAT to PIBNETD P(Cs |Us ) can be expressed as: 1 , w 2 , w 3 ), P(Cs |Us ) = P(C1 = T |w11 , w12 , w13 ) · · · P(Cm = T |wm m m
where wi1 , wi2 , and wi3 denote the instantiation of the variables wi1 , wi2 , and wi3 as designated by Us . Because Us represents a satisfying truth assignment for C , it also represent a satisfying truth assignment for each clause ci in C .
56 / 72
Proof - from 3SAT to PIBNETD P(Cs |Us ) can be expressed as: 1 , w 2 , w 3 ), P(Cs |Us ) = P(C1 = T |w11 , w12 , w13 ) · · · P(Cm = T |wm m m
where wi1 , wi2 , and wi3 denote the instantiation of the variables wi1 , wi2 , and wi3 as designated by Us . Because Us represents a satisfying truth assignment for C , it also represent a satisfying truth assignment for each clause ci in C . We therefore have: P(Ci = T |wi1 , wi2 , wi3 ) = 1 for 1 ≤ i ≤ m,
57 / 72
Proof - from 3SAT to PIBNETD P(Cs |Us ) can be expressed as: 1 , w 2 , w 3 ), P(Cs |Us ) = P(C1 = T |w11 , w12 , w13 ) · · · P(Cm = T |wm m m
where wi1 , wi2 , and wi3 denote the instantiation of the variables wi1 , wi2 , and wi3 as designated by Us . Because Us represents a satisfying truth assignment for C , it also represent a satisfying truth assignment for each clause ci in C . We therefore have: P(Ci = T |wi1 , wi2 , wi3 ) = 1 for 1 ≤ i ≤ m, and hence: P(Cs |Us ) = 1 which means P(Y = T ) > 0. 58 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable.
59 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0
60 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0 and hence: P(Cs |Uα ) = 0 for all Uα .
61 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0 and hence: P(Cs |Uα ) = 0 for all Uα . This implies that the only possible positive term in the sum must exist when Cβ 6= Cs . However, when Cβ 6= Cs , we have that P(Y = T |Cβ ) = 0,
62 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0 and hence: P(Cs |Uα ) = 0 for all Uα . This implies that the only possible positive term in the sum must exist when Cβ 6= Cs . However, when Cβ 6= Cs , we have that P(Y = T |Cβ ) = 0, and therefore that: P(Y = T ) = 0.
63 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0 and hence: P(Cs |Uα ) = 0 for all Uα . This implies that the only possible positive term in the sum must exist when Cβ 6= Cs . However, when Cβ 6= Cs , we have that P(Y = T |Cβ ) = 0, and therefore that: P(Y = T ) = 0. This implies that PIBNETD is N P-hard. 64 / 72
Proof - from PIBNETD to PIBNET
PIBNET is the probabilistic inference function problem of determining P(Y = T ).
65 / 72
Proof - from PIBNETD to PIBNET
PIBNET is the probabilistic inference function problem of determining P(Y = T ). The reduction from PIBNETD to PIBNET is very simple.
66 / 72
Proof - from PIBNETD to PIBNET
PIBNET is the probabilistic inference function problem of determining P(Y = T ). The reduction from PIBNETD to PIBNET is very simple. We simply let PIBNET determine P(Y = T ) and answer ”yes” if P(Y = T ) > 0 and ”no” if P(Y = T ) = 0.
67 / 72
Proof - from PIBNETD to PIBNET
PIBNET is the probabilistic inference function problem of determining P(Y = T ). The reduction from PIBNETD to PIBNET is very simple. We simply let PIBNET determine P(Y = T ) and answer ”yes” if P(Y = T ) > 0 and ”no” if P(Y = T ) = 0. Thus, we have proven that PIBNET is N P-hard.
68 / 72
Additional Complexity Results Together, the truth-setting component and the clause-satisfaction-testing component defines a bipartite Bayesian network, which can be used as one type of restricted diagnostic system, where each node ui corresponds to a disease and each node Ci corresponds to a symptom.
69 / 72
Additional Complexity Results Together, the truth-setting component and the clause-satisfaction-testing component defines a bipartite Bayesian network, which can be used as one type of restricted diagnostic system, where each node ui corresponds to a disease and each node Ci corresponds to a symptom. The probability of a disease ui given a set of evidence variables {C1 , C2 , · · · , Cm } can be calculated as: P(ui = T |C1 , C2 , · · · , Cm ) =
P(C1 ,C2 ,··· ,Cm |ui =T )P(ui =T ) P(C1 ,C2 ,··· ,Cm )
70 / 72
Additional Complexity Results Together, the truth-setting component and the clause-satisfaction-testing component defines a bipartite Bayesian network, which can be used as one type of restricted diagnostic system, where each node ui corresponds to a disease and each node Ci corresponds to a symptom. The probability of a disease ui given a set of evidence variables {C1 , C2 , · · · , Cm } can be calculated as: P(ui = T |C1 , C2 , · · · , Cm ) =
P(C1 ,C2 ,··· ,Cm |ui =T )P(ui =T ) P(C1 ,C2 ,··· ,Cm )
Recall that Cs is the set of instantiated variables that results from instantiating each Ci in {C1 , C2 , · · · , Cm } to the value T.
71 / 72
Additional Complexity Results Together, the truth-setting component and the clause-satisfaction-testing component defines a bipartite Bayesian network, which can be used as one type of restricted diagnostic system, where each node ui corresponds to a disease and each node Ci corresponds to a symptom. The probability of a disease ui given a set of evidence variables {C1 , C2 , · · · , Cm } can be calculated as: P(ui = T |C1 , C2 , · · · , Cm ) =
P(C1 ,C2 ,··· ,Cm |ui =T )P(ui =T ) P(C1 ,C2 ,··· ,Cm )
Recall that Cs is the set of instantiated variables that results from instantiating each Ci in {C1 , C2 , · · · , Cm } to the value T. Inferring the value of P(Cs ) using a bipartite Bayesian network answers the PIBNETD decision problem and therefore this inference task is N P-hard as well. 72 / 72
October 2008
1 / 72
Purpose
To show that probabilistic inference using multiply connected Bayesian networks is N P-hard.
2 / 72
Multiply Connected Bayesian Networks Multiply connected networks contains at least one pair of variables that have more than one undirected path connecting them.
3 / 72
Multiply Connected Bayesian Networks Multiply connected networks contains at least one pair of variables that have more than one undirected path connecting them. Example:
A
B E
C F
D G
H
4 / 72
Representation of Bayesian Networks
We represent a Bayesian network BN as a triple BN = (V , A, P), where V is the set of variables, A the set of arcs between variables, and P the set of probabilities.
5 / 72
Representation of Bayesian Networks
We represent a Bayesian network BN as a triple BN = (V , A, P), where V is the set of variables, A the set of arcs between variables, and P the set of probabilities. We assume that sp(Vi ) = {T , F } for all Vi ∈ V .
6 / 72
Probabilistic Inference
We focus on the form of probabilistic inference needed to determine the probability P(Y = T ) for some variable Y .
7 / 72
Probabilistic Inference
We focus on the form of probabilistic inference needed to determine the probability P(Y = T ) for some variable Y . By proving that computing P(Y = T ) is N P-hard we will prove that other more general forms of probabilistic inference are N P-hard as well.
8 / 72
Proof Idea
We define PIBNET as the probabilistic inference function problem of determining P(Y = T ) for some variable Y in a Bayesian network.
9 / 72
Proof Idea
We define PIBNET as the probabilistic inference function problem of determining P(Y = T ) for some variable Y in a Bayesian network. We define PIBNETD as a decision problem version of PIBNET , that for some variable Y determines whether P(Y = T ) > 0 and returns ”yes” if so and ”no” otherwise.
10 / 72
Proof Idea
We define PIBNET as the probabilistic inference function problem of determining P(Y = T ) for some variable Y in a Bayesian network. We define PIBNETD as a decision problem version of PIBNET , that for some variable Y determines whether P(Y = T ) > 0 and returns ”yes” if so and ”no” otherwise. We prove that PIBNET is N P-hard by giving a polynomial time reduction from the well known N P-complete problem 3SAT to PIBNETD. A polynomial time reduction from PIBNETD to PIBNET concludes the proof.
11 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables.
12 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U.
13 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U. The literal ui is true iff the variable u is true (T), and the literal ¬ui is true iff the variable u is false (F).
14 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U. The literal ui is true iff the variable u is true (T), and the literal ¬ui is true iff the variable u is false (F). Each clause ci contains a disjunction of three literals over U.
15 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U. The literal ui is true iff the variable u is true (T), and the literal ¬ui is true iff the variable u is false (F). Each clause ci contains a disjunction of three literals over U. A set C of clauses over U is satisfiable iff there exists some truth assignment for U that satisfies all the clauses in C .
16 / 72
3SAT Let C = {c1 , c2 , · · · , cm } be a set of clauses on a set U = {u1 , u2 , · · · , un } of boolean variables. If ui is a variable in U, then ui and ¬ui are literals over U. The literal ui is true iff the variable u is true (T), and the literal ¬ui is true iff the variable u is false (F). Each clause ci contains a disjunction of three literals over U. A set C of clauses over U is satisfiable iff there exists some truth assignment for U that satisfies all the clauses in C . The 3SAT problem determines whether a set C of clauses over a set of variables U is satisfiable. 17 / 72
3SAT
Example of a 3SAT instance: U = {u1 , u2 , u3 , u4 } C = {(u1 ∨ u2 ∨ u3 ), (¬u1 ∨ ¬u2 ∨ u3 ), (u2 ∨ ¬u3 ∨ u4 )} 3SAT would answer ”yes” in this situation using, for example, the truth assignments: u1 = F , u2 = T , u3 = T , and u4 = F .
18 / 72
Proof - from 3SAT to PIBNETD
Let U = {u1 , u2 , · · · , un } and C = {c1 , c2 , · · · , cm } be any instance of 3SAT .
19 / 72
Proof - from 3SAT to PIBNETD
Let U = {u1 , u2 , · · · , un } and C = {c1 , c2 , · · · , cm } be any instance of 3SAT . We construct a BN containing variable Y such that: P(T = Y ) > 0 ⇐⇒ C is satisfiable.
20 / 72
Proof - from 3SAT to PIBNETD
The BN will consists of three components:
21 / 72
Proof - from 3SAT to PIBNETD
The BN will consists of three components: A truth-setting component - sets the truth values of U.
22 / 72
Proof - from 3SAT to PIBNETD
The BN will consists of three components: A truth-setting component - sets the truth values of U. A clause-satisfaction-testing component - tests whether each clause is satisfied.
23 / 72
Proof - from 3SAT to PIBNETD
The BN will consists of three components: A truth-setting component - sets the truth values of U. A clause-satisfaction-testing component - tests whether each clause is satisfied. An overall-satisfaction-testing component - tests whether C is satisfied.
24 / 72
Proof - from 3SAT to PIBNETD Given the 3SAT instance from the example before: U = {u1 , u2 , u3 , u4 } C = {(u1 ∨ u2 ∨ u3 ), (¬u1 ∨ ¬u2 ∨ u3 ), (u2 ∨ ¬u3 ∨ u4 )} we want to end up with a Bayesian network like this:
u1
u2 C1
u3 C2 Y
u4 C3
}
}
}
truth-setting component clause-satisfaction-testing component
overall-satisfaction-testing component
25 / 72
Proof - from 3SAT to PIBNETD
The truth-setting component (V t , P t ) is constructed as:
26 / 72
Proof - from 3SAT to PIBNETD
The truth-setting component (V t , P t ) is constructed as: V t = U.
27 / 72
Proof - from 3SAT to PIBNETD
The truth-setting component (V t , P t ) is constructed as: V t = U. P t = {P(u1 = T ) = 12 , P(u2 = T ) = 21 , · · · , P(un = T ) = 12 }.
28 / 72
Proof - from 3SAT to PIBNETD The clause-satisfaction-testing component (V s , As , P s ) is the union of m subcomponents; one for each clause ci ∈ C . Each subcomponent (Vis , Asi , Pis ) is constructed as:
29 / 72
Proof - from 3SAT to PIBNETD The clause-satisfaction-testing component (V s , As , P s ) is the union of m subcomponents; one for each clause ci ∈ C . Each subcomponent (Vis , Asi , Pis ) is constructed as: Vis = {wi1 , wi2 , wi3 , Ci }, where wi1 , wi2 , and wi3 are the variables corresponding to the literals in clause ci , and the variable Ci represents the truth value of clause ci .
30 / 72
Proof - from 3SAT to PIBNETD The clause-satisfaction-testing component (V s , As , P s ) is the union of m subcomponents; one for each clause ci ∈ C . Each subcomponent (Vis , Asi , Pis ) is constructed as: Vis = {wi1 , wi2 , wi3 , Ci }, where wi1 , wi2 , and wi3 are the variables corresponding to the literals in clause ci , and the variable Ci represents the truth value of clause ci . Asi = {(wi1 , Ci ), (wi2 , Ci ), (wi3 , Ci )}.
31 / 72
Proof - from 3SAT to PIBNETD The clause-satisfaction-testing component (V s , As , P s ) is the union of m subcomponents; one for each clause ci ∈ C . Each subcomponent (Vis , Asi , Pis ) is constructed as: Vis = {wi1 , wi2 , wi3 , Ci }, where wi1 , wi2 , and wi3 are the variables corresponding to the literals in clause ci , and the variable Ci represents the truth value of clause ci . Asi = {(wi1 , Ci ), (wi2 , Ci ), (wi3 , Ci )}. Pis = {P(Ci = T |πci )}, where πci represents the conjunction of the three variables and 1 if gi (πci ) = T , P(Ci = T |πci ) = 0 if gi (πci ) = F , where gi is the truth function for clause ci . 32 / 72
Proof - from 3SAT to PIBNETD The overall-satisfaction-testing component (V o , Ao , P o ) is constructed as:
33 / 72
Proof - from 3SAT to PIBNETD The overall-satisfaction-testing component (V o , Ao , P o ) is constructed as: V o = {Y }.
34 / 72
Proof - from 3SAT to PIBNETD The overall-satisfaction-testing component (V o , Ao , P o ) is constructed as: V o = {Y }. Ao = {(C1 , Y ), (C2 , Y ), · · · , (Cm , Y )}.
35 / 72
Proof - from 3SAT to PIBNETD The overall-satisfaction-testing component (V o , Ao , P o ) is constructed as: V o = {Y }. Ao = {(C1 , Y ), (C2 , Y ), · · · , (Cm , Y )}. P o = {P(Y = T |C1 , C2 , · · · , Cm )}, where P(Y = T |C1 , C2 , · · · , Cm ) =
1 if C1 ∧ C2 ∧ · · · ∧ Cm = T , 0 if otherwise.
36 / 72
Proof - from 3SAT to PIBNETD When the number of clauses increase so does the number of parents of the node Y . u1
u2 C1
u3 C2
u4 C3
Y
37 / 72
Proof - from 3SAT to PIBNETD When the number of clauses increase so does the number of parents of the node Y . u1
u2 C1
u3 C2
u4 C3
Y
To avoid having a node with many parents, dummy variables functioning as AND-gates, can be inserted. u1
u2
u3
u4
C1
C2
C3
D1
D2
Y
38 / 72
Proof - from 3SAT to PIBNETD When the number of clauses increase so does the number of parents of the node Y . u1
u2 C1
u3 C2
u4 C3
Y
To avoid having a node with many parents, dummy variables functioning as AND-gates, can be inserted. u1
u2
u3
u4
C1
C2
C3
D1
D2
Y
The use of dummy variables keeps the indegree of any node in the network below four. 39 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po
40 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po The time complexity of constructing the Bayesian network is polynomial in the size of the 3SAT instance:
41 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po The time complexity of constructing the Bayesian network is polynomial in the size of the 3SAT instance: O(n) - the construction of the truth-setting component
42 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po The time complexity of constructing the Bayesian network is polynomial in the size of the 3SAT instance: O(n) - the construction of the truth-setting component O(m) - the construction of the clause-satisfaction-testing component
43 / 72
Proof - from 3SAT to PIBNETD To complete the construction of the Bayesian network BN, we set BN = (V , A, P), where V = Vt ∪ Vs ∪ Vo A = At ∪ As ∪ Ao P = Pt ∪ Ps ∪ Po The time complexity of constructing the Bayesian network is polynomial in the size of the 3SAT instance: O(n) - the construction of the truth-setting component O(m) - the construction of the clause-satisfaction-testing component O(m) - the construction of the overall-satisfaction-testing component 44 / 72
Proof - from 3SAT to PIBNETD
We now only need to show that: P(T = Y ) > 0 ⇐⇒ C is satisfiable.
45 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T .
46 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T . For example U5 represents the instantiation: u4 = F , u3 = T , u2 = F , and u1 = T , because the binary representation of 5 is 0101.
47 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T . For example U5 represents the instantiation: u4 = F , u3 = T , u2 = F , and u1 = T , because the binary representation of 5 is 0101. Let Cβ denote an instantiation of the variables in C similar to Uα .
48 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T . For example U5 represents the instantiation: u4 = F , u3 = T , u2 = F , and u1 = T , because the binary representation of 5 is 0101. Let Cβ denote an instantiation of the variables in C similar to Uα . Let Us denote a truth assignment to the variables in U that satisfies all the clauses in C .
49 / 72
Proof - from 3SAT to PIBNETD Let Uα denote an instantiation of the variables in U such that: if the ith digit (from the right) of the binary representation of α is 0, then ui = F . if the ith digit (from the right) of the binary representation of α is 1, then ui = T . For example U5 represents the instantiation: u4 = F , u3 = T , u2 = F , and u1 = T , because the binary representation of 5 is 0101. Let Cβ denote an instantiation of the variables in C similar to Uα . Let Us denote a truth assignment to the variables in U that satisfies all the clauses in C . Let Cs denote the set of instantiated variables that results from assigning the value T to each variable Ci . 50 / 72
Proof - from 3SAT to PIBNETD We can compute P(Y = T ) as: P(Y = T ) =
n −1 2m −1 2X X
P(Y = T |Cβ )P(Cβ |Uα )P(Uα )
α=0 β=0
51 / 72
Proof - from 3SAT to PIBNETD We can compute P(Y = T ) as: P(Y = T ) =
n −1 2m −1 2X X
P(Y = T |Cβ )P(Cβ |Uα )P(Uα )
α=0 β=0
Now we assume that C is satisfiable.
52 / 72
Proof - from 3SAT to PIBNETD We can compute P(Y = T ) as: P(Y = T ) =
n −1 2m −1 2X X
P(Y = T |Cβ )P(Cβ |Uα )P(Uα )
α=0 β=0
Now we assume that C is satisfiable. Since C is satisfiable, one term of the sum is P(Y = T |Cs )P(Cs |Us )P(Us ) and we have that: P(Y = T ) ≥ P(Y = T |Cs )P(Cs |Us )P(Us )
53 / 72
Proof - from 3SAT to PIBNETD We can compute P(Y = T ) as: P(Y = T ) =
n −1 2m −1 2X X
P(Y = T |Cβ )P(Cβ |Uα )P(Uα )
α=0 β=0
Now we assume that C is satisfiable. Since C is satisfiable, one term of the sum is P(Y = T |Cs )P(Cs |Us )P(Us ) and we have that: P(Y = T ) ≥ P(Y = T |Cs )P(Cs |Us )P(Us ) We know that P(Y = T |Cs ) = 1 and P(Us ) = ( 12 )n so we only need to show that P(Cs |Us ) > 0. 54 / 72
Proof - from 3SAT to PIBNETD P(Cs |Us ) can be expressed as: 1 , w 2 , w 3 ), P(Cs |Us ) = P(C1 = T |w11 , w12 , w13 ) · · · P(Cm = T |wm m m
where wi1 , wi2 , and wi3 denote the instantiation of the variables wi1 , wi2 , and wi3 as designated by Us .
55 / 72
Proof - from 3SAT to PIBNETD P(Cs |Us ) can be expressed as: 1 , w 2 , w 3 ), P(Cs |Us ) = P(C1 = T |w11 , w12 , w13 ) · · · P(Cm = T |wm m m
where wi1 , wi2 , and wi3 denote the instantiation of the variables wi1 , wi2 , and wi3 as designated by Us . Because Us represents a satisfying truth assignment for C , it also represent a satisfying truth assignment for each clause ci in C .
56 / 72
Proof - from 3SAT to PIBNETD P(Cs |Us ) can be expressed as: 1 , w 2 , w 3 ), P(Cs |Us ) = P(C1 = T |w11 , w12 , w13 ) · · · P(Cm = T |wm m m
where wi1 , wi2 , and wi3 denote the instantiation of the variables wi1 , wi2 , and wi3 as designated by Us . Because Us represents a satisfying truth assignment for C , it also represent a satisfying truth assignment for each clause ci in C . We therefore have: P(Ci = T |wi1 , wi2 , wi3 ) = 1 for 1 ≤ i ≤ m,
57 / 72
Proof - from 3SAT to PIBNETD P(Cs |Us ) can be expressed as: 1 , w 2 , w 3 ), P(Cs |Us ) = P(C1 = T |w11 , w12 , w13 ) · · · P(Cm = T |wm m m
where wi1 , wi2 , and wi3 denote the instantiation of the variables wi1 , wi2 , and wi3 as designated by Us . Because Us represents a satisfying truth assignment for C , it also represent a satisfying truth assignment for each clause ci in C . We therefore have: P(Ci = T |wi1 , wi2 , wi3 ) = 1 for 1 ≤ i ≤ m, and hence: P(Cs |Us ) = 1 which means P(Y = T ) > 0. 58 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable.
59 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0
60 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0 and hence: P(Cs |Uα ) = 0 for all Uα .
61 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0 and hence: P(Cs |Uα ) = 0 for all Uα . This implies that the only possible positive term in the sum must exist when Cβ 6= Cs . However, when Cβ 6= Cs , we have that P(Y = T |Cβ ) = 0,
62 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0 and hence: P(Cs |Uα ) = 0 for all Uα . This implies that the only possible positive term in the sum must exist when Cβ 6= Cs . However, when Cβ 6= Cs , we have that P(Y = T |Cβ ) = 0, and therefore that: P(Y = T ) = 0.
63 / 72
Proof - from 3SAT to PIBNETD Now we assume that C is not satisfiable. Since C is not satisfiable, for every instantiation of Uα there is at least one variable Ci for which P(Ci = T |wi1 , wi2 , wi3 ) = 0 and hence: P(Cs |Uα ) = 0 for all Uα . This implies that the only possible positive term in the sum must exist when Cβ 6= Cs . However, when Cβ 6= Cs , we have that P(Y = T |Cβ ) = 0, and therefore that: P(Y = T ) = 0. This implies that PIBNETD is N P-hard. 64 / 72
Proof - from PIBNETD to PIBNET
PIBNET is the probabilistic inference function problem of determining P(Y = T ).
65 / 72
Proof - from PIBNETD to PIBNET
PIBNET is the probabilistic inference function problem of determining P(Y = T ). The reduction from PIBNETD to PIBNET is very simple.
66 / 72
Proof - from PIBNETD to PIBNET
PIBNET is the probabilistic inference function problem of determining P(Y = T ). The reduction from PIBNETD to PIBNET is very simple. We simply let PIBNET determine P(Y = T ) and answer ”yes” if P(Y = T ) > 0 and ”no” if P(Y = T ) = 0.
67 / 72
Proof - from PIBNETD to PIBNET
PIBNET is the probabilistic inference function problem of determining P(Y = T ). The reduction from PIBNETD to PIBNET is very simple. We simply let PIBNET determine P(Y = T ) and answer ”yes” if P(Y = T ) > 0 and ”no” if P(Y = T ) = 0. Thus, we have proven that PIBNET is N P-hard.
68 / 72
Additional Complexity Results Together, the truth-setting component and the clause-satisfaction-testing component defines a bipartite Bayesian network, which can be used as one type of restricted diagnostic system, where each node ui corresponds to a disease and each node Ci corresponds to a symptom.
69 / 72
Additional Complexity Results Together, the truth-setting component and the clause-satisfaction-testing component defines a bipartite Bayesian network, which can be used as one type of restricted diagnostic system, where each node ui corresponds to a disease and each node Ci corresponds to a symptom. The probability of a disease ui given a set of evidence variables {C1 , C2 , · · · , Cm } can be calculated as: P(ui = T |C1 , C2 , · · · , Cm ) =
P(C1 ,C2 ,··· ,Cm |ui =T )P(ui =T ) P(C1 ,C2 ,··· ,Cm )
70 / 72
Additional Complexity Results Together, the truth-setting component and the clause-satisfaction-testing component defines a bipartite Bayesian network, which can be used as one type of restricted diagnostic system, where each node ui corresponds to a disease and each node Ci corresponds to a symptom. The probability of a disease ui given a set of evidence variables {C1 , C2 , · · · , Cm } can be calculated as: P(ui = T |C1 , C2 , · · · , Cm ) =
P(C1 ,C2 ,··· ,Cm |ui =T )P(ui =T ) P(C1 ,C2 ,··· ,Cm )
Recall that Cs is the set of instantiated variables that results from instantiating each Ci in {C1 , C2 , · · · , Cm } to the value T.
71 / 72
Additional Complexity Results Together, the truth-setting component and the clause-satisfaction-testing component defines a bipartite Bayesian network, which can be used as one type of restricted diagnostic system, where each node ui corresponds to a disease and each node Ci corresponds to a symptom. The probability of a disease ui given a set of evidence variables {C1 , C2 , · · · , Cm } can be calculated as: P(ui = T |C1 , C2 , · · · , Cm ) =
P(C1 ,C2 ,··· ,Cm |ui =T )P(ui =T ) P(C1 ,C2 ,··· ,Cm )
Recall that Cs is the set of instantiated variables that results from instantiating each Ci in {C1 , C2 , · · · , Cm } to the value T. Inferring the value of P(Cs ) using a bipartite Bayesian network answers the PIBNETD decision problem and therefore this inference task is N P-hard as well. 72 / 72
