Adversarial Classification - UCSD CSE
Adversarial Classification Nilesh Dalvi, Pedro Domingos, Mausam, Sumit Sanghai, Deepak Verma [KDD ’04, Seattle]
Presented by: Aditya Menon UCSD
April 25, 2008
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
1 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
2 / 71
Previous supervised learning research
Learner typically agnostic about data producer I
Assumes that data just arises “naturally”
But is this realistic? I I I
Spam: spammers try to fool Bayesian classifiers Intrusion detection: intruder tries to cover up footprint ...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
3 / 71
Previous supervised learning research
Learner typically agnostic about data producer I
Assumes that data just arises “naturally”
But is this realistic? I I I
Spam: spammers try to fool Bayesian classifiers Intrusion detection: intruder tries to cover up footprint ...
Reality: Producer very much aware data is being classified I
And can change data to fool learner
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
3 / 71
Example of clever producer Learner recognizes: Cheap high quality Rolex 75% off November only
Spammer switches to: This November, purchasing low cost Rolexes is simple
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
4 / 71
Consequences of producer intereference
What happens if we ignore this fact? I I I
Producer can generate “worst case” data for classifier Classifier accuracy degrades rapidly Have to change our classifier to keep up
Net result: keep reconstructing classifier I
Figure out how we were fooled, and fix it
But matter of time before malicious producer strikes again...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
5 / 71
Adversarial classification
Realistic model: treat data producer as an adversary I I
Producer knows the classifier being used Tweaks data to maximize misclassification
Question: If we know the data will be tampered, can we improve our classifier?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
6 / 71
It’s all a game
Learner and adversary are locked in a game I I I I
Learner makes a prediction Adversary deduces prediction technique, modifies data to breaks it Learner knows adversary’s strategy, changes classifier ...
Question: What is the best strategy for the Learner?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
7 / 71
This paper
Assumes naive Bayes classifier I
Important assumption, hence constrains applicability
Derives optimal adversary strategy Consequently, derives optimal classifier strategy Shows that this has a significant impact on classifier accuracy I
Spam detection application
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
8 / 71
Notions of optimality
How do we define the “best” strategy? In game theory? I
I
Typically seek a Nash equilibrium: neither player has an incentive to change his strategy Does not mean that either player’s payoff is maximized
In this paper? I I I
Simply a locally optimal strategy “Best response” to what the other player did Players constantly changing strategy based on what the other does
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
9 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
10 / 71
Standard naive Bayes classifier Suppose an instance has n features: x = (x1 , x2 , . . . , xn ) xi ∈ Xi Given an instance x, probability of it having class y is P (y) P (y|x) = P (x|y) = P (x)
"
n Y i=1
# P (xi |y)
P (y) P (x)
Conditional feature-independence is the naive Bayes assumption
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
11 / 71
Standard naive Bayes classifier
Suppose an instance has n features: x = (x1 , x2 , . . . , xn ) xi ∈ Xi Given an instance x, probability of it having class y is P (y) = P (y|x) = P (x|y) P (x) I
"
n Y i=1
# P (xi |y)
P (y) P (x)
Conditional feature-independence is the naive Bayes assumption
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
11 / 71
Cost-sensitive prediction
Paper uses the notion of the utility of a classification I
UC (y 0 , y) is the utility or benefit of predicting that something with true class y has class y 0
Then, just choose the y 0 that maximizes X U (y 0 |x) = P (y|x)UC (y 0 , y) y
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
12 / 71
Optimal cost-sensitive prediction
Paper considers problems with two classes I I
Malicious (+) e.g. spam Harmless (-) e.g. normal email
Optimal prediction is the y 0 that maximizes U (y 0 |x) = P (+|x)UC (y 0 , +) + P (−|x)UC (y 0 , −)
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
13 / 71
Example of a utility matrix
Reasonable utility choices? Actual Predicted
Presented by: Aditya Menon (UCSD)
UC + -
+ True positive False negative
False positive True negative
UC + -
+ 1 −1
−10 1
Adversarial Classification
April 25, 2008
14 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
15 / 71
The nature of the adversary
Now suppose there is an adversary that modifies the data Adversary’s goal: For each example, modify its feature values so that the probability they are classified as “harmless” increases Adversary will transform an instance x using a function A(x): A : X1 × . . . × Xn → X1 × . . . × Xn We need to know the nature of the optimal A(x)
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
16 / 71
Adversary’s limitations
Why can’t adversary just change every feature value? I
Naturally, there is some notion of the cost to the adversary
We suppose the adversary has a set of matrices Wi , where i runs over all the features I
Wi (xi , x0i ) = cost for the adversary to modify value of feature i from xi to x0i
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
17 / 71
Adversary’s utility matrix How can the adversary measure success? I
Has a matrix UA , like the UC matrix for the classifier
Reasonable utility choices for UA ?
Predicted
UA + UA + -
Presented by: Aditya Menon (UCSD)
Actual + Malicious, prevented Harmless, prevented Malicious, let through Harmless, let through + −1 20
Adversarial Classification
0 0
April 25, 2008
18 / 71
Cost-sensitive classification: adversary’s perspective Recall that the utility for the learner to classify x as class y is U (y|x) = P (+|x)UC (y, +) + P (−|x)UC (y, −) Learner classifies x as harmless (-) when U (+|x) ≤ U (−|x) P (+|x) UC (−, −) − UC (+, −) =⇒ ≤ P (−|x) UC (+, +) − UC (−, +) I
Call these the odds and the threshold
So, this is what the adversary wants to ensure
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
19 / 71
Adversary’s goal
Now define logarithmic equivalents: P (+|x) , “log odds” P (−|x) UC (−, −) − UC (+, −) T (UC ) = log , “log threshold” UC (+, +) − UC (−, +) L(x) = log
Prediction is “harmless” when the gap between the two is non-positive: gap(x) := L(x) − T (UC ) ≤ 0
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
20 / 71
Tricking the learner
Adversary’s goal: Make x classified as harmless (-), but not if it costs the adversary too much to do so Two questions I I
How to make classification of x harmless? How much cost is too much?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
21 / 71
How can adversary trick learner?
First problem: how to make x be classified as harmless? Transform x 7→ x0 , so that gap(x0 ) = L(x0 ) − T (UC ) ≤ 0
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
22 / 71
How can adversary trick learner? In a naive Bayes classifier, we know P (+|x) =
Y
P (xi |+)
P (+) P (x)
P (xi |−)
P (−) P (x)
i
P (−|x) =
Y i
Dividing and taking logs, log
P (+|x) P (+) X P (xi |+) = log + log P (−|x) P (−) P (xi |−) i
L(x) := log
P (+) X + logodds(xi ) P (−) i
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
23 / 71
How can adversary trick learner? Recall that we want L(x0 ) − T (UC ) ≤ 0 Rewrite in terms of x’s gap: L(x) + L(x0 ) − T (UC ) ≤ L(x) L(x) − T (UC ) ≤ L(x) − L(x0 ) X logodds(xi ) − logodds(x0i ) =⇒ gap(x) ≤ i
:=
X
Di (xi , x0i )
i
Di (xi , x0i ) measures the change in log-odds if we change the ith feature from xi to x0i
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
24 / 71
Formulating an integer program
We can find the optimal strategy by formulating an integer program I
Want to change a subset features to cause a misclassification
How to model cost of feature modification? Let δi,x0i be a binary denoting if we modify feature i into x0i Recall: Wi (xi , x0i ) is the cost to change ith feature from xi to x0i Then Cost =
XX i
Presented by: Aditya Menon (UCSD)
Wi (xi , x0i )δi,x0i
x0i
Adversarial Classification
April 25, 2008
25 / 71
Finding optimal strategy Optimal strategy can be found by solving a integer program Natural linear constraints with integer variables: I
Minimize cost of making changes (Our goal)
I
Changes cause the example to be classified as harmless
I
Feature i can only be changed once
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
26 / 71
Finding optimal strategy Optimal strategy can be found by solving a integer program Natural linear constraints with integer variables: I
Minimize cost of making changes (Our goal) X X min Wi (xi , x0i )δi,x0i 0 i
xi
I
Changes cause the example to be classified as harmless
I
Feature i can only be changed once
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
26 / 71
Finding optimal strategy Optimal strategy can be found by solving a integer program Natural linear constraints with integer variables: I
Minimize cost of making changes (Our goal) X X min Wi (xi , x0i )δi,x0i 0 i
I
Changes cause the example to be classified as harmless XX Di (xi , x0i )δi,x0i ≥ gap(x) i
I
xi
x0i
Feature i can only be changed once
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
26 / 71
Finding optimal strategy Optimal strategy can be found by solving a integer program Natural linear constraints with integer variables: I
Minimize cost of making changes (Our goal) X X min Wi (xi , x0i )δi,x0i 0 i
I
Changes cause the example to be classified as harmless XX Di (xi , x0i )δi,x0i ≥ gap(x) i
I
xi
x0i
Feature i can only be changed once X δi,x0i ≤ 1 x0i
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
26 / 71
Optimal strategy program Optimal program min
X X
XX i
i
Wi (xi , x0i )δi,x0i
x0i
Di (xi , x0i )δi,x0i ≥ gap(x)
x0i
X
δi,x0i ≤ 1
x0i
δi,x0i ∈ {0, 1} The solution to the program is denoted by M CC(x): the “minimum cost camouflage”
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
27 / 71
Adversary’s output
Assuming we find a solution, what do we do? Only use the solution if the cost is smaller than the benefit I
The benefit is the change in utility from misclassification, ∆UA = UA (−, +) − UA (+, +)
Given x, we output A(x), where ( M CC(x) N B(x) = +, W (x, M CC(x)) < ∆UA A(x) = x otherwise
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
28 / 71
But can we find the M CC?
But how easy is it to find M CC(x)? This is an integer program... I
=⇒ NP-hard!
How to get around this? I I
Show that P = NP... ...or use an approximation!
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
29 / 71
Breaking intractability of M CC
Solve the integer program by discretizing the problem space I
Use dynamic programming to solve the discretized version
Use two pruning rules to further simplify the results Skip to summary of adversarial algorithm
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
30 / 71
Breaking intractability of M CC
Discretize our problem space, so that we can use dynamic programming (xi |+) discrete Make logodds(xi ) = log PP (x i |−) I I
Minimum interval of δ, say Forces Di to be discrete too
Focus on a new problem...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
31 / 71
Splitting related problem into subproblems
Using only the first i features, what is the least-cost set of changes that decreases the log-odds by w? I
I
If we change the ith feature to x0i , then we can change the log-odds by Di (xi , x0i ) So, recursively find minimum cost needed to change the log-odds by w − Di (xi , x0i ), using the first (i − 1) features
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
32 / 71
Using dynamic programming for the M CC
Suppose F indM CC(i, w) finds the minimum cost needed to change the log-odds by w, using the first i features Consider gd ap(x) to be gap(x) in the discrete space Now run the algorithm F indM CC(n, gd ap(x)) I
The MCC requires us to change the log-odds by gap(x), using the first n features i.e. all features
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
33 / 71
The M CC algorithm
FindMCC(i, w): MinCost = ∞, MinList = [] for x0i ∈ Xi if Di (xi , x0i ) ≥ 0
Cost, List ← FindMCC(i − 1, w − Di (xi , x0i )) Cost + = Wi (xi , x0i ) List + = (i, x0i ) if Cost < MinCost MinCost = Cost MinList = List
return MinCost, MinList
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
34 / 71
Further improvements to tractability
Even after discretization, we might take a lot of time to solve the program I
Around O(d g ap(x)
P
i
|Xi |)
Can prune results with two further insights I I
Easily detect when we would require too much cost Discretized coarse metric
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
35 / 71
First pruning optimization
Can immediately strike out instances that are “too positive” Don’t need to spend time finding their minimum camouflage
Theorem If
max
Di (xi , x0i ) Wi (xi , x0i )
∆UA , so is the camouflage cost
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
37 / 71
Second pruning optimization
Can eliminate redundant checks Sort the (i, x0i ) tuples in increasing order of Wi (xi , x0i ) For identical values of Wi (xi , x0i ), only keep the one with largest Di (xi , x0i ) I I
Works because optimal solution is invariant under choice of Di (xi , x0i ) With coarse discretization, remove a lot of pairs from consideration
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
38 / 71
Summary thus far
Thus far, we have shown the following
Fact Given a naive Bayes classifier, it is possible for an adversary to efficiently compute a transformation A, which, given a malicious instance x, creates x0 = A(x) that is classified as harmless I
Since we can efficiently compute A, we cannot just ignore the adversarial presence
So now, the question is what the classifier can do...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
39 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
40 / 71
Learner strategy
So now assume that the adversary has applied A on the data How can the classifier try to foil his plan? I
Compensate for the fact that P (x|+) is now suspect
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
41 / 71
Learner strategy Brute force way to deal with adversary? I
Look at each instance x, and estimate the probability is was modified into the instance we see now
Denote the new estimate for the conditional probability PA , given an instance x0 : X PA (x0 |+) = P (x0 |x, +)P (x|+) x I
But P (x0 |x, +) is a 0 − 1 variable (i.e. not random!)
In terms of the adversarial function A, X PA (x0 |+) = x∈XA
I
P (x|+)
(x0 )
So, we consider all x which could camouflage to x0 , and sum up their probabilities
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
42 / 71
Learner algorithm
After adjusting the probability, Learner proceeds as normal Recall that the classification is whichever class y maximizes U (y|x) = P (+|x)UC (y, +) + P (−|x)UC (y, −) Classifier simply computes U (+|x), U (−|x) I
Estimates P ’s based on training set
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
43 / 71
Learner algorithm
Classify(x): P (−|x) = P (−)
Q
i P (Xi
= xi |−)
P (+|x) = P (+)PA (x|+) for y = {+, −} U (y|x) = P (y|x)UC (y, +) + P (−|x)UC (y, −) return argmax U (y|x) y
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
44 / 71
Simple as that?
Are we done? Estimating the P ’s can be done easily But computing PA requires summing over XA (x0 ) I I
Any x whose camouflage is x0 This is a very large set...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
45 / 71
Problem again?
Again, computing the set XA (x0 ) is intractable Trivial simplification: consider X 0 A (x0 ) = XA (x0 ) − x, and X P (x0 |+) = P (x|+) + δx0 P (x0 |+) x∈X 0 A (x0 )
I
Easy to check if x0 ∈ XA (x0 )
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
46 / 71
Estimating the set
How to estimate X 0 A (x0 )? Again, two pruning rules I I
First try and eliminate those x who cannot have x0 as a camouflage Then try and bound those x who must have x0 as a camouflage
Skip to summary of classifier algorithm
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
47 / 71
First pruning rule
Use the following theorem
Theorem If x is a malicious instance, and x0 = M CC(x), then for each i, xi 6= x0i =⇒ gap(x) + LO(xi ) − LO(x0i ) > 0 That is, those features on which x and x0 disagree have the above technical property for their gap
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
48 / 71
First pruning rule
What does theorem tell us? I
If x does not satisfy the above property, then x0 6= M CC(x)
Reduce the instances we need to check I I
Only consider those x that satisfy the theorem Rest cannot have x0 as their camouflage
Still could have exponentially large search space, though...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
49 / 71
Second pruning rule
One can prove that x0 6= M CC(x) says something about other x, too I
I
Being a camouflage of x means we are the camouflage for more restricted feature sets Cannot make x0 the camouflage of any instance formed by just changing more features
Uses the following theorem
Theorem Suppose x is a positive instance, and x0 = M CC(x). Let D be the features changed in x to produce x0 , and E ⊆ D. Let x00 be x modified only in the features E. Then, x0 = M CC(x00 ) also.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
50 / 71
Visualization
What does theorem tell us? I I I
Say x0 = M CC(x), with features D of x changed to get x0 Suppose y is like x, but with some of the features in D already changed Then x0 = M CC(y)
x01
x02
x03
x04
...
x0n−1
x0n
x1
x2
x3
x4
...
xn−1
xn
y1
y2
y3
y4
...
yn−1
yn
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
51 / 71
Combining the pruning rules Let F V = {(i, xi )} be those feature-value pairs we get from the first rule Let x[i→y] denote the instance x with the ith feature value changed to y Now, using the second rule, only consider GV = {(i, xi ) ∈ F V } so that x0[i→xi ] ∈ XA (x0 ) I
The second rule tells us that if x0 = M CC(x), then the changes from x → x0 must be contained in GV
This means that we only check if a subset of feature changes produces a MCC
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
52 / 71
Combining the pruning rules
Pruning rules tell us
Theorem X (i,xi )∈GV
P (x0[i→xi ] |+) ≤
X
P (x|+)
0 (x0 ) x∈XA
Use the lower bound as an estimation of the true value
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
53 / 71
Summary thus far
Thus far, we have shown the following
Fact If the learner knows that there is an adversary modifying the data according to A(x), it is possible to efficiently create a new classifier strategy that minimizes the chance of misprediction.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
54 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
55 / 71
Experiments
Experiments on spam filtering Two data sets I I
Ling spam: messages on a linguistics mailing list (16.6% spam) Email-data: collection of emails (55.1% spam)
Three data models I
I I
Add words: Spammer adds words to fool classifier, each word has unit cost Add length: Add words, except each character has unit cost Synonym: Spammer changes words in document to fool classifier
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
56 / 71
Add words
Add words: Spammer adds words to fool classifier, each word has unit cost
Original We offer cheap high quality watches.
Changed Bob meeting field prevaricate. We offer cheap high quality watches.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
57 / 71
Add length
Add length: Add words, except each character has unit cost
Original We offer cheap high quality watches.
Changed prevaricate We offer cheap high quality watches.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
58 / 71
Synonym
Synonym: Spammer changes words in document to fool classifier
Original We offer cheap high quality watches.
Changed We provide inexpensive high quality watches.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
59 / 71
Experimental setup
Naive Bayes classifier is run on the untampered data For each data model, adversary’s algorithm is run to compute A(x) On the modified data: I I
Run Naive Bayes Run optimal classifier strategy
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
60 / 71
Results Ling-spam with different misprediction costs (misclassify malicious as harmless)
Adversarial classifier significantly improves results (∼ 40%) Data model has little effect Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
61 / 71
Results Email-spam with different misprediction costs
Adversarial classifier significantly improves results (∼ 90%) Only Synonym is feasible for adversary-agnostic classifier Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
62 / 71
Runtime
Informal comparison of runtime Add Length takes the longest time for adversary to compute strategy (∼ 500 ms)
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
63 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
64 / 71
Critique on classifier front
Only works on Naive Bayes I
Good start, but other classifiers could also be studied
Assumes that all parameters are known to classifier and adversary I
Unrealistic, though they can probably be estimated
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
65 / 71
Critique on game theory front
Only for single round of classifier-adversary interaction I
I I
Does not tell us what happens when adversary responds to the improved classifier Also a good start, but a long-run optimal solution is also important Does not eliminate manual intervention
Nash equilbrium result seems inevitable I
Theoretical importance
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
66 / 71
Subsequent work
[2] removed assumption of adversary possessing perfect knowledge I
Studied how adversary could deduce good values for the parameters
Generally, something of a dead end I I
Hard to study theoretically Unrealistic in practise: naive Bayes assumption itself is a major limitation
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
67 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
68 / 71
Conclusion
In many classification problems, data generator is adversarial Classifier must try to minimize the damage caused by adversary I
Or risk performance degradation
Naive Bayes classifier can be made adversary aware Good performance for spam detection But can it be used in practise?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
69 / 71
Questions?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
70 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
71 / 71
Icons from http://www.iconaholic.com/downloads.html Nilesh Dalvi, Pedro Domingos, Mausam, Sumit Sanghai, and Deepak Verma. Adversarial classification. In KDD ’04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 99–108, New York, NY, USA, 2004. ACM. Daniel Lowd and Christopher Meek. Adversarial learning. In KDD ’05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pages 641–647, New York, NY, USA, 2005. ACM.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
71 / 71
Presented by: Aditya Menon UCSD
April 25, 2008
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
1 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
2 / 71
Previous supervised learning research
Learner typically agnostic about data producer I
Assumes that data just arises “naturally”
But is this realistic? I I I
Spam: spammers try to fool Bayesian classifiers Intrusion detection: intruder tries to cover up footprint ...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
3 / 71
Previous supervised learning research
Learner typically agnostic about data producer I
Assumes that data just arises “naturally”
But is this realistic? I I I
Spam: spammers try to fool Bayesian classifiers Intrusion detection: intruder tries to cover up footprint ...
Reality: Producer very much aware data is being classified I
And can change data to fool learner
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
3 / 71
Example of clever producer Learner recognizes: Cheap high quality Rolex 75% off November only
Spammer switches to: This November, purchasing low cost Rolexes is simple
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
4 / 71
Consequences of producer intereference
What happens if we ignore this fact? I I I
Producer can generate “worst case” data for classifier Classifier accuracy degrades rapidly Have to change our classifier to keep up
Net result: keep reconstructing classifier I
Figure out how we were fooled, and fix it
But matter of time before malicious producer strikes again...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
5 / 71
Adversarial classification
Realistic model: treat data producer as an adversary I I
Producer knows the classifier being used Tweaks data to maximize misclassification
Question: If we know the data will be tampered, can we improve our classifier?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
6 / 71
It’s all a game
Learner and adversary are locked in a game I I I I
Learner makes a prediction Adversary deduces prediction technique, modifies data to breaks it Learner knows adversary’s strategy, changes classifier ...
Question: What is the best strategy for the Learner?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
7 / 71
This paper
Assumes naive Bayes classifier I
Important assumption, hence constrains applicability
Derives optimal adversary strategy Consequently, derives optimal classifier strategy Shows that this has a significant impact on classifier accuracy I
Spam detection application
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
8 / 71
Notions of optimality
How do we define the “best” strategy? In game theory? I
I
Typically seek a Nash equilibrium: neither player has an incentive to change his strategy Does not mean that either player’s payoff is maximized
In this paper? I I I
Simply a locally optimal strategy “Best response” to what the other player did Players constantly changing strategy based on what the other does
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
9 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
10 / 71
Standard naive Bayes classifier Suppose an instance has n features: x = (x1 , x2 , . . . , xn ) xi ∈ Xi Given an instance x, probability of it having class y is P (y) P (y|x) = P (x|y) = P (x)
"
n Y i=1
# P (xi |y)
P (y) P (x)
Conditional feature-independence is the naive Bayes assumption
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
11 / 71
Standard naive Bayes classifier
Suppose an instance has n features: x = (x1 , x2 , . . . , xn ) xi ∈ Xi Given an instance x, probability of it having class y is P (y) = P (y|x) = P (x|y) P (x) I
"
n Y i=1
# P (xi |y)
P (y) P (x)
Conditional feature-independence is the naive Bayes assumption
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
11 / 71
Cost-sensitive prediction
Paper uses the notion of the utility of a classification I
UC (y 0 , y) is the utility or benefit of predicting that something with true class y has class y 0
Then, just choose the y 0 that maximizes X U (y 0 |x) = P (y|x)UC (y 0 , y) y
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
12 / 71
Optimal cost-sensitive prediction
Paper considers problems with two classes I I
Malicious (+) e.g. spam Harmless (-) e.g. normal email
Optimal prediction is the y 0 that maximizes U (y 0 |x) = P (+|x)UC (y 0 , +) + P (−|x)UC (y 0 , −)
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
13 / 71
Example of a utility matrix
Reasonable utility choices? Actual Predicted
Presented by: Aditya Menon (UCSD)
UC + -
+ True positive False negative
False positive True negative
UC + -
+ 1 −1
−10 1
Adversarial Classification
April 25, 2008
14 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
15 / 71
The nature of the adversary
Now suppose there is an adversary that modifies the data Adversary’s goal: For each example, modify its feature values so that the probability they are classified as “harmless” increases Adversary will transform an instance x using a function A(x): A : X1 × . . . × Xn → X1 × . . . × Xn We need to know the nature of the optimal A(x)
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
16 / 71
Adversary’s limitations
Why can’t adversary just change every feature value? I
Naturally, there is some notion of the cost to the adversary
We suppose the adversary has a set of matrices Wi , where i runs over all the features I
Wi (xi , x0i ) = cost for the adversary to modify value of feature i from xi to x0i
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
17 / 71
Adversary’s utility matrix How can the adversary measure success? I
Has a matrix UA , like the UC matrix for the classifier
Reasonable utility choices for UA ?
Predicted
UA + UA + -
Presented by: Aditya Menon (UCSD)
Actual + Malicious, prevented Harmless, prevented Malicious, let through Harmless, let through + −1 20
Adversarial Classification
0 0
April 25, 2008
18 / 71
Cost-sensitive classification: adversary’s perspective Recall that the utility for the learner to classify x as class y is U (y|x) = P (+|x)UC (y, +) + P (−|x)UC (y, −) Learner classifies x as harmless (-) when U (+|x) ≤ U (−|x) P (+|x) UC (−, −) − UC (+, −) =⇒ ≤ P (−|x) UC (+, +) − UC (−, +) I
Call these the odds and the threshold
So, this is what the adversary wants to ensure
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
19 / 71
Adversary’s goal
Now define logarithmic equivalents: P (+|x) , “log odds” P (−|x) UC (−, −) − UC (+, −) T (UC ) = log , “log threshold” UC (+, +) − UC (−, +) L(x) = log
Prediction is “harmless” when the gap between the two is non-positive: gap(x) := L(x) − T (UC ) ≤ 0
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
20 / 71
Tricking the learner
Adversary’s goal: Make x classified as harmless (-), but not if it costs the adversary too much to do so Two questions I I
How to make classification of x harmless? How much cost is too much?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
21 / 71
How can adversary trick learner?
First problem: how to make x be classified as harmless? Transform x 7→ x0 , so that gap(x0 ) = L(x0 ) − T (UC ) ≤ 0
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
22 / 71
How can adversary trick learner? In a naive Bayes classifier, we know P (+|x) =
Y
P (xi |+)
P (+) P (x)
P (xi |−)
P (−) P (x)
i
P (−|x) =
Y i
Dividing and taking logs, log
P (+|x) P (+) X P (xi |+) = log + log P (−|x) P (−) P (xi |−) i
L(x) := log
P (+) X + logodds(xi ) P (−) i
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
23 / 71
How can adversary trick learner? Recall that we want L(x0 ) − T (UC ) ≤ 0 Rewrite in terms of x’s gap: L(x) + L(x0 ) − T (UC ) ≤ L(x) L(x) − T (UC ) ≤ L(x) − L(x0 ) X logodds(xi ) − logodds(x0i ) =⇒ gap(x) ≤ i
:=
X
Di (xi , x0i )
i
Di (xi , x0i ) measures the change in log-odds if we change the ith feature from xi to x0i
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
24 / 71
Formulating an integer program
We can find the optimal strategy by formulating an integer program I
Want to change a subset features to cause a misclassification
How to model cost of feature modification? Let δi,x0i be a binary denoting if we modify feature i into x0i Recall: Wi (xi , x0i ) is the cost to change ith feature from xi to x0i Then Cost =
XX i
Presented by: Aditya Menon (UCSD)
Wi (xi , x0i )δi,x0i
x0i
Adversarial Classification
April 25, 2008
25 / 71
Finding optimal strategy Optimal strategy can be found by solving a integer program Natural linear constraints with integer variables: I
Minimize cost of making changes (Our goal)
I
Changes cause the example to be classified as harmless
I
Feature i can only be changed once
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
26 / 71
Finding optimal strategy Optimal strategy can be found by solving a integer program Natural linear constraints with integer variables: I
Minimize cost of making changes (Our goal) X X min Wi (xi , x0i )δi,x0i 0 i
xi
I
Changes cause the example to be classified as harmless
I
Feature i can only be changed once
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
26 / 71
Finding optimal strategy Optimal strategy can be found by solving a integer program Natural linear constraints with integer variables: I
Minimize cost of making changes (Our goal) X X min Wi (xi , x0i )δi,x0i 0 i
I
Changes cause the example to be classified as harmless XX Di (xi , x0i )δi,x0i ≥ gap(x) i
I
xi
x0i
Feature i can only be changed once
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
26 / 71
Finding optimal strategy Optimal strategy can be found by solving a integer program Natural linear constraints with integer variables: I
Minimize cost of making changes (Our goal) X X min Wi (xi , x0i )δi,x0i 0 i
I
Changes cause the example to be classified as harmless XX Di (xi , x0i )δi,x0i ≥ gap(x) i
I
xi
x0i
Feature i can only be changed once X δi,x0i ≤ 1 x0i
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
26 / 71
Optimal strategy program Optimal program min
X X
XX i
i
Wi (xi , x0i )δi,x0i
x0i
Di (xi , x0i )δi,x0i ≥ gap(x)
x0i
X
δi,x0i ≤ 1
x0i
δi,x0i ∈ {0, 1} The solution to the program is denoted by M CC(x): the “minimum cost camouflage”
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
27 / 71
Adversary’s output
Assuming we find a solution, what do we do? Only use the solution if the cost is smaller than the benefit I
The benefit is the change in utility from misclassification, ∆UA = UA (−, +) − UA (+, +)
Given x, we output A(x), where ( M CC(x) N B(x) = +, W (x, M CC(x)) < ∆UA A(x) = x otherwise
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
28 / 71
But can we find the M CC?
But how easy is it to find M CC(x)? This is an integer program... I
=⇒ NP-hard!
How to get around this? I I
Show that P = NP... ...or use an approximation!
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
29 / 71
Breaking intractability of M CC
Solve the integer program by discretizing the problem space I
Use dynamic programming to solve the discretized version
Use two pruning rules to further simplify the results Skip to summary of adversarial algorithm
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
30 / 71
Breaking intractability of M CC
Discretize our problem space, so that we can use dynamic programming (xi |+) discrete Make logodds(xi ) = log PP (x i |−) I I
Minimum interval of δ, say Forces Di to be discrete too
Focus on a new problem...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
31 / 71
Splitting related problem into subproblems
Using only the first i features, what is the least-cost set of changes that decreases the log-odds by w? I
I
If we change the ith feature to x0i , then we can change the log-odds by Di (xi , x0i ) So, recursively find minimum cost needed to change the log-odds by w − Di (xi , x0i ), using the first (i − 1) features
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
32 / 71
Using dynamic programming for the M CC
Suppose F indM CC(i, w) finds the minimum cost needed to change the log-odds by w, using the first i features Consider gd ap(x) to be gap(x) in the discrete space Now run the algorithm F indM CC(n, gd ap(x)) I
The MCC requires us to change the log-odds by gap(x), using the first n features i.e. all features
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
33 / 71
The M CC algorithm
FindMCC(i, w): MinCost = ∞, MinList = [] for x0i ∈ Xi if Di (xi , x0i ) ≥ 0
Cost, List ← FindMCC(i − 1, w − Di (xi , x0i )) Cost + = Wi (xi , x0i ) List + = (i, x0i ) if Cost < MinCost MinCost = Cost MinList = List
return MinCost, MinList
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
34 / 71
Further improvements to tractability
Even after discretization, we might take a lot of time to solve the program I
Around O(d g ap(x)
P
i
|Xi |)
Can prune results with two further insights I I
Easily detect when we would require too much cost Discretized coarse metric
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
35 / 71
First pruning optimization
Can immediately strike out instances that are “too positive” Don’t need to spend time finding their minimum camouflage
Theorem If
max
Di (xi , x0i ) Wi (xi , x0i )
∆UA , so is the camouflage cost
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
37 / 71
Second pruning optimization
Can eliminate redundant checks Sort the (i, x0i ) tuples in increasing order of Wi (xi , x0i ) For identical values of Wi (xi , x0i ), only keep the one with largest Di (xi , x0i ) I I
Works because optimal solution is invariant under choice of Di (xi , x0i ) With coarse discretization, remove a lot of pairs from consideration
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
38 / 71
Summary thus far
Thus far, we have shown the following
Fact Given a naive Bayes classifier, it is possible for an adversary to efficiently compute a transformation A, which, given a malicious instance x, creates x0 = A(x) that is classified as harmless I
Since we can efficiently compute A, we cannot just ignore the adversarial presence
So now, the question is what the classifier can do...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
39 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
40 / 71
Learner strategy
So now assume that the adversary has applied A on the data How can the classifier try to foil his plan? I
Compensate for the fact that P (x|+) is now suspect
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
41 / 71
Learner strategy Brute force way to deal with adversary? I
Look at each instance x, and estimate the probability is was modified into the instance we see now
Denote the new estimate for the conditional probability PA , given an instance x0 : X PA (x0 |+) = P (x0 |x, +)P (x|+) x I
But P (x0 |x, +) is a 0 − 1 variable (i.e. not random!)
In terms of the adversarial function A, X PA (x0 |+) = x∈XA
I
P (x|+)
(x0 )
So, we consider all x which could camouflage to x0 , and sum up their probabilities
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
42 / 71
Learner algorithm
After adjusting the probability, Learner proceeds as normal Recall that the classification is whichever class y maximizes U (y|x) = P (+|x)UC (y, +) + P (−|x)UC (y, −) Classifier simply computes U (+|x), U (−|x) I
Estimates P ’s based on training set
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
43 / 71
Learner algorithm
Classify(x): P (−|x) = P (−)
Q
i P (Xi
= xi |−)
P (+|x) = P (+)PA (x|+) for y = {+, −} U (y|x) = P (y|x)UC (y, +) + P (−|x)UC (y, −) return argmax U (y|x) y
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
44 / 71
Simple as that?
Are we done? Estimating the P ’s can be done easily But computing PA requires summing over XA (x0 ) I I
Any x whose camouflage is x0 This is a very large set...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
45 / 71
Problem again?
Again, computing the set XA (x0 ) is intractable Trivial simplification: consider X 0 A (x0 ) = XA (x0 ) − x, and X P (x0 |+) = P (x|+) + δx0 P (x0 |+) x∈X 0 A (x0 )
I
Easy to check if x0 ∈ XA (x0 )
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
46 / 71
Estimating the set
How to estimate X 0 A (x0 )? Again, two pruning rules I I
First try and eliminate those x who cannot have x0 as a camouflage Then try and bound those x who must have x0 as a camouflage
Skip to summary of classifier algorithm
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
47 / 71
First pruning rule
Use the following theorem
Theorem If x is a malicious instance, and x0 = M CC(x), then for each i, xi 6= x0i =⇒ gap(x) + LO(xi ) − LO(x0i ) > 0 That is, those features on which x and x0 disagree have the above technical property for their gap
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
48 / 71
First pruning rule
What does theorem tell us? I
If x does not satisfy the above property, then x0 6= M CC(x)
Reduce the instances we need to check I I
Only consider those x that satisfy the theorem Rest cannot have x0 as their camouflage
Still could have exponentially large search space, though...
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
49 / 71
Second pruning rule
One can prove that x0 6= M CC(x) says something about other x, too I
I
Being a camouflage of x means we are the camouflage for more restricted feature sets Cannot make x0 the camouflage of any instance formed by just changing more features
Uses the following theorem
Theorem Suppose x is a positive instance, and x0 = M CC(x). Let D be the features changed in x to produce x0 , and E ⊆ D. Let x00 be x modified only in the features E. Then, x0 = M CC(x00 ) also.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
50 / 71
Visualization
What does theorem tell us? I I I
Say x0 = M CC(x), with features D of x changed to get x0 Suppose y is like x, but with some of the features in D already changed Then x0 = M CC(y)
x01
x02
x03
x04
...
x0n−1
x0n
x1
x2
x3
x4
...
xn−1
xn
y1
y2
y3
y4
...
yn−1
yn
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
51 / 71
Combining the pruning rules Let F V = {(i, xi )} be those feature-value pairs we get from the first rule Let x[i→y] denote the instance x with the ith feature value changed to y Now, using the second rule, only consider GV = {(i, xi ) ∈ F V } so that x0[i→xi ] ∈ XA (x0 ) I
The second rule tells us that if x0 = M CC(x), then the changes from x → x0 must be contained in GV
This means that we only check if a subset of feature changes produces a MCC
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
52 / 71
Combining the pruning rules
Pruning rules tell us
Theorem X (i,xi )∈GV
P (x0[i→xi ] |+) ≤
X
P (x|+)
0 (x0 ) x∈XA
Use the lower bound as an estimation of the true value
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
53 / 71
Summary thus far
Thus far, we have shown the following
Fact If the learner knows that there is an adversary modifying the data according to A(x), it is possible to efficiently create a new classifier strategy that minimizes the chance of misprediction.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
54 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
55 / 71
Experiments
Experiments on spam filtering Two data sets I I
Ling spam: messages on a linguistics mailing list (16.6% spam) Email-data: collection of emails (55.1% spam)
Three data models I
I I
Add words: Spammer adds words to fool classifier, each word has unit cost Add length: Add words, except each character has unit cost Synonym: Spammer changes words in document to fool classifier
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
56 / 71
Add words
Add words: Spammer adds words to fool classifier, each word has unit cost
Original We offer cheap high quality watches.
Changed Bob meeting field prevaricate. We offer cheap high quality watches.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
57 / 71
Add length
Add length: Add words, except each character has unit cost
Original We offer cheap high quality watches.
Changed prevaricate We offer cheap high quality watches.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
58 / 71
Synonym
Synonym: Spammer changes words in document to fool classifier
Original We offer cheap high quality watches.
Changed We provide inexpensive high quality watches.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
59 / 71
Experimental setup
Naive Bayes classifier is run on the untampered data For each data model, adversary’s algorithm is run to compute A(x) On the modified data: I I
Run Naive Bayes Run optimal classifier strategy
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
60 / 71
Results Ling-spam with different misprediction costs (misclassify malicious as harmless)
Adversarial classifier significantly improves results (∼ 40%) Data model has little effect Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
61 / 71
Results Email-spam with different misprediction costs
Adversarial classifier significantly improves results (∼ 90%) Only Synonym is feasible for adversary-agnostic classifier Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
62 / 71
Runtime
Informal comparison of runtime Add Length takes the longest time for adversary to compute strategy (∼ 500 ms)
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
63 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
64 / 71
Critique on classifier front
Only works on Naive Bayes I
Good start, but other classifiers could also be studied
Assumes that all parameters are known to classifier and adversary I
Unrealistic, though they can probably be estimated
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
65 / 71
Critique on game theory front
Only for single round of classifier-adversary interaction I
I I
Does not tell us what happens when adversary responds to the improved classifier Also a good start, but a long-run optimal solution is also important Does not eliminate manual intervention
Nash equilbrium result seems inevitable I
Theoretical importance
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
66 / 71
Subsequent work
[2] removed assumption of adversary possessing perfect knowledge I
Studied how adversary could deduce good values for the parameters
Generally, something of a dead end I I
Hard to study theoretically Unrealistic in practise: naive Bayes assumption itself is a major limitation
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
67 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
68 / 71
Conclusion
In many classification problems, data generator is adversarial Classifier must try to minimize the damage caused by adversary I
Or risk performance degradation
Naive Bayes classifier can be made adversary aware Good performance for spam detection But can it be used in practise?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
69 / 71
Questions?
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
70 / 71
Outline 1
Previous supervised learning research
2
Background: naive Bayes classifier Standard naive Bayes Cost-sensitive classification
3
Adversary’s strategy Adversary’s goal Finding optimal strategy Pruning optimizations
4
Learner’s strategy Learner’s goal Pruning optimizations
5
Experiments
6
Critique and future work
7
Conclusion
8
References
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
71 / 71
Icons from http://www.iconaholic.com/downloads.html Nilesh Dalvi, Pedro Domingos, Mausam, Sumit Sanghai, and Deepak Verma. Adversarial classification. In KDD ’04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 99–108, New York, NY, USA, 2004. ACM. Daniel Lowd and Christopher Meek. Adversarial learning. In KDD ’05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pages 641–647, New York, NY, USA, 2005. ACM.
Presented by: Aditya Menon (UCSD)
Adversarial Classification
April 25, 2008
71 / 71
