Labelling Image Regions Using Wavelet Features and Spatial ...
Information Systems & Semantic Web University of Koblenz ▪ Landau, Germany
Labelling Image Regions Using Wavelet Features and Spatial Prototypes Carsten Saathoff, Marcin Grzegorzek and Steffen Staab SAMT 2008, Koblenz, Germany
Motivation Sea Sky
↯
Sky Sea
Sand
Local features often not sufficient for classification Exploit explicitly defined spatial knowledge to improve labelling e.g. Sky not allowed left or right of Sea Allow for efficient training of classifiers and spatial knowledge Good labelling performance with few training examples
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 2 of 26
Agenda
Analysis Framework Exploiting Spatial Context, 1st try Fuzzy Constraint Satisfaction (WIAMIS08) Contribution Low-Level Region Classification Training of statistical models Classification Exploiting Spatial Context, revisited Binary Integer Programming Evaluation Conclusions
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 3 of 26
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 4 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Exploiting Spatial Context Create appropriate optimization problem from Regions Spatial Relations Hypotheses sets Spatial background knowledge Approaches Fuzzy Constraint Satisfaction • WIAMIS08 Binary Integer Programming • later...
left-of
above
Carsten Saathoff [email protected]
above
Hypotheses Sand, 0.8 Sea, 0.7 Person,0.5 …
Background Knowledge above-of
ISWeb - Information Systems & Semantic Web
above
above
SAMT 2008 5 of 26
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
Region Labelling as a FCSP above-of
above-of above
above left-of
left-of above
above
above-of
above-of
Domain
Hypotheses
Sand, 0.8 Sea, 0.7 Person,0.5 …
Sand, 0.8 Sea, 0.7 Person,0.5 …
Background Knowledge above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 6 of 26
F-Measure (WIAMIS08) F-Measure for all concepts average F-Measure. 80
SVM FCSP
7% average gain
70
60
50
40
30
20
10
0 person
boat
ISWeb - Information Systems & Semantic Web
sand
building
road
mountain
Carsten Saathoff [email protected]
water
sky
SAMT 2008 7 of 26
plant
snow
macro avg
Contribution Region-Level classification based on Wavelet Features Statistical Classification Spatial Reasoning Training of Spatial Knowledge Formalization as Binary Integer Programming Comparison with alternative approach over different training set sizes
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 8 of 26
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 9 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Wavelet Features and Statistical Training
Mean value over wavelet coefficients of all neighborhoods
Mean pixel value Feature Vector:
m , b R , bG ,b B Compute mean and standard deviation
Statistical model for concept K
ISWeb - Information Systems & Semantic Web
K = m , R , G , B K = m , R , G , B
Carsten Saathoff [email protected]
SAMT 2008 10 of 26
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 11 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Statistical Classification Feature extraction:
c r = m , b R , bG , b B
Training Set Sky Sea
above
Input for spatial reasoning ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 12 of 26
Sky, 0.8 Sea,0.76 Sand, 0.68 Person, 0.67 Building, 0.54 …
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 13 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 14 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Constraint Acquisition Background Knowledge consists of Spatial Constraint Templates degree of satisfaction for spatial arrangements of concepts Simplest version: crisp degrees • e.g. above = (Sky, Sea):1.0, (Sea, Sky):0.0
Acquired by mining from a set of labelled examples (spatial prototypes) mining based on confidence (and support) Background Knowledge
ISWeb - Information Systems & Semantic Web
above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
left-of
(Sky, Sea) -> 0.0 (Sea, Sand) -> 1.0 (Sea, Sea) -> 1.0 …
Carsten Saathoff [email protected]
SAMT 2008 15 of 26
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 16 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Exploiting Spatial Context Create appropriate optimization problem from Regions Spatial Relations Hypotheses sets Spatial background knowledge Approaches Fuzzy Constraint Satisfaction • WIAMIS08 Binary Integer Programming
above
above left-of
above
above
Hypotheses Sand, 0.8 Sea, 0.7 Person,0.5 …
Background Knowledge above-of
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 17 of 26
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
Binary Integer Programming Problem of the form:
minimise Z =c T x st. A x=b x ∈{ 0,1}
Example: Assignment Problem • i=1...n workers • j=1..n machines • cij cost of assigning worker i to machine j • xij=1 assigns worker i to machine j
minimise n
n
∑ j =1 ∑i=1 c ij xij st. n
∑ j =1 x ij=1 for i =1,. .. , n n ∑i=1 xij =1 for j=1,. .. , n x ij ∈{0,1 }
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 18 of 26
Spatial Reasoning as a BIP
above
above left-of
Set of variables: ko
c itj =1
above
ko
c itj
si =l k , s j =l o
above
Only one assignment per relation Hypotheses Sand, 0.8 Sea, 0.7 Person,0.5 …
∀ r t = si , s j : ∑l
Background Knowledge above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 19 of 26
k
∑ l c koitj =1 o
Spatial Reasoning as a BIP (2)
above
above left-of
above
Link outgoing relations: Base relation:
above
rt
O
Link remaining outgoing relations to base relation:
∀ l k : ∑l c i t j −∑ l c it j ' =0 ko
o
k o'
O
o'
Hypotheses
Accordingly for incoming relations:
Sand, 0.8 Sea, 0.7 Person,0.5 …
∀ l k : ∑l c okj t i − ∑l c oj '' kti =0 o
E
o'
Background Knowledge above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
Link outgoing and incoming relations o' k ∀ l k : ∑l c ko i t j −∑ l c j ' t i =0 o
SAMT 2008 20 of 26
O
o'
E
Objective Function Set of variables:
above
above left-of
ko
c itj
Linking constraints Hypotheses i l k
above
above
Templates
T t l k , l o
Hypotheses Sand, 0.8 Sea, 0.7 Person,0.5 …
Objective Function:
Background Knowledge above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
ISWeb - Information Systems & Semantic Web
∑r ∑l ∑l min i l k , j l o∗T t l k ,l o ∗c t
Carsten Saathoff [email protected]
k
o
SAMT 2008 21 of 26
ko itj
Experimental Setup
Measure improvement achieved with FCSP Linear Programming over low-level classification with different training set sizes Data set with 923 natural and urban images Set of 10 concepts building,foliage,mountain,person,road,sailingboat,sand,sea,sky,snow 5690 labelled regions 568 labelled „unknown“ -> ignored Ground truth defined on automatic segmentation Regions labelled with dominant concept Ground truth created for this work ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 22 of 26
Overview of dataset
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 23 of 26
Classification Rate over Training Set Size
0.85 0.80 0.75
Low-level classification FCSP BIP
0.70 0.65 0.60 0.55 0.50 50
100
ISWeb - Information Systems & Semantic Web
150
200
250
Carsten Saathoff [email protected]
300
350
SAMT 2008 24 of 26
400
F-Measure per Concept Training Set with 300 images (best performing one for BIP) 1 0.9 0.8 0.7 0.6 Low-level FCSP BIP
0.5 0.4 0.3 0.2 0.1 0 snow
road
mountain foliage
ISWeb - Information Systems & Semantic Web
person
sand
Carsten Saathoff [email protected]
sky sailing-boat sea
SAMT 2008 25 of 26
building average
Conclusions
Combination of Wavelets based region classification Spatial context with explicit knowledge BIP outperforms FCSP Classification rate Efficiency (BIP avg ~1 sec, FCSP ~40secs) Validation of earlier observation Explicit knowledge provides good model for spatial context exploitation Low amount of training data required to acquire good performing models
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 26 of 26
Thanks!
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 27 of 26
Labelling Image Regions Using Wavelet Features and Spatial Prototypes Carsten Saathoff, Marcin Grzegorzek and Steffen Staab SAMT 2008, Koblenz, Germany
Motivation Sea Sky
↯
Sky Sea
Sand
Local features often not sufficient for classification Exploit explicitly defined spatial knowledge to improve labelling e.g. Sky not allowed left or right of Sea Allow for efficient training of classifiers and spatial knowledge Good labelling performance with few training examples
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 2 of 26
Agenda
Analysis Framework Exploiting Spatial Context, 1st try Fuzzy Constraint Satisfaction (WIAMIS08) Contribution Low-Level Region Classification Training of statistical models Classification Exploiting Spatial Context, revisited Binary Integer Programming Evaluation Conclusions
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 3 of 26
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 4 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Exploiting Spatial Context Create appropriate optimization problem from Regions Spatial Relations Hypotheses sets Spatial background knowledge Approaches Fuzzy Constraint Satisfaction • WIAMIS08 Binary Integer Programming • later...
left-of
above
Carsten Saathoff [email protected]
above
Hypotheses Sand, 0.8 Sea, 0.7 Person,0.5 …
Background Knowledge above-of
ISWeb - Information Systems & Semantic Web
above
above
SAMT 2008 5 of 26
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
Region Labelling as a FCSP above-of
above-of above
above left-of
left-of above
above
above-of
above-of
Domain
Hypotheses
Sand, 0.8 Sea, 0.7 Person,0.5 …
Sand, 0.8 Sea, 0.7 Person,0.5 …
Background Knowledge above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 6 of 26
F-Measure (WIAMIS08) F-Measure for all concepts average F-Measure. 80
SVM FCSP
7% average gain
70
60
50
40
30
20
10
0 person
boat
ISWeb - Information Systems & Semantic Web
sand
building
road
mountain
Carsten Saathoff [email protected]
water
sky
SAMT 2008 7 of 26
plant
snow
macro avg
Contribution Region-Level classification based on Wavelet Features Statistical Classification Spatial Reasoning Training of Spatial Knowledge Formalization as Binary Integer Programming Comparison with alternative approach over different training set sizes
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 8 of 26
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 9 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Wavelet Features and Statistical Training
Mean value over wavelet coefficients of all neighborhoods
Mean pixel value Feature Vector:
m , b R , bG ,b B Compute mean and standard deviation
Statistical model for concept K
ISWeb - Information Systems & Semantic Web
K = m , R , G , B K = m , R , G , B
Carsten Saathoff [email protected]
SAMT 2008 10 of 26
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 11 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Statistical Classification Feature extraction:
c r = m , b R , bG , b B
Training Set Sky Sea
above
Input for spatial reasoning ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 12 of 26
Sky, 0.8 Sea,0.76 Sand, 0.68 Person, 0.67 Building, 0.54 …
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 13 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 14 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Constraint Acquisition Background Knowledge consists of Spatial Constraint Templates degree of satisfaction for spatial arrangements of concepts Simplest version: crisp degrees • e.g. above = (Sky, Sea):1.0, (Sea, Sky):0.0
Acquired by mining from a set of labelled examples (spatial prototypes) mining based on confidence (and support) Background Knowledge
ISWeb - Information Systems & Semantic Web
above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
left-of
(Sky, Sea) -> 0.0 (Sea, Sand) -> 1.0 (Sea, Sea) -> 1.0 …
Carsten Saathoff [email protected]
SAMT 2008 15 of 26
Analysis Framework Training Training Examples
Sky Sea
above
Trained Spatial Background Knowledge Low Level Classifiers
(x,x,x,x) (x,x,x,x) (x,x,x,x) (x,x,x,x)
Hypotheses Generation
Spatial Relations Extraction
Analysis
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 16 of 26
Spatial Reasoning
Sky Sea
Sea Sand
Exploiting Spatial Context Create appropriate optimization problem from Regions Spatial Relations Hypotheses sets Spatial background knowledge Approaches Fuzzy Constraint Satisfaction • WIAMIS08 Binary Integer Programming
above
above left-of
above
above
Hypotheses Sand, 0.8 Sea, 0.7 Person,0.5 …
Background Knowledge above-of
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 17 of 26
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
Binary Integer Programming Problem of the form:
minimise Z =c T x st. A x=b x ∈{ 0,1}
Example: Assignment Problem • i=1...n workers • j=1..n machines • cij cost of assigning worker i to machine j • xij=1 assigns worker i to machine j
minimise n
n
∑ j =1 ∑i=1 c ij xij st. n
∑ j =1 x ij=1 for i =1,. .. , n n ∑i=1 xij =1 for j=1,. .. , n x ij ∈{0,1 }
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 18 of 26
Spatial Reasoning as a BIP
above
above left-of
Set of variables: ko
c itj =1
above
ko
c itj
si =l k , s j =l o
above
Only one assignment per relation Hypotheses Sand, 0.8 Sea, 0.7 Person,0.5 …
∀ r t = si , s j : ∑l
Background Knowledge above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 19 of 26
k
∑ l c koitj =1 o
Spatial Reasoning as a BIP (2)
above
above left-of
above
Link outgoing relations: Base relation:
above
rt
O
Link remaining outgoing relations to base relation:
∀ l k : ∑l c i t j −∑ l c it j ' =0 ko
o
k o'
O
o'
Hypotheses
Accordingly for incoming relations:
Sand, 0.8 Sea, 0.7 Person,0.5 …
∀ l k : ∑l c okj t i − ∑l c oj '' kti =0 o
E
o'
Background Knowledge above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
Link outgoing and incoming relations o' k ∀ l k : ∑l c ko i t j −∑ l c j ' t i =0 o
SAMT 2008 20 of 26
O
o'
E
Objective Function Set of variables:
above
above left-of
ko
c itj
Linking constraints Hypotheses i l k
above
above
Templates
T t l k , l o
Hypotheses Sand, 0.8 Sea, 0.7 Person,0.5 …
Objective Function:
Background Knowledge above-of
(Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 …
ISWeb - Information Systems & Semantic Web
∑r ∑l ∑l min i l k , j l o∗T t l k ,l o ∗c t
Carsten Saathoff [email protected]
k
o
SAMT 2008 21 of 26
ko itj
Experimental Setup
Measure improvement achieved with FCSP Linear Programming over low-level classification with different training set sizes Data set with 923 natural and urban images Set of 10 concepts building,foliage,mountain,person,road,sailingboat,sand,sea,sky,snow 5690 labelled regions 568 labelled „unknown“ -> ignored Ground truth defined on automatic segmentation Regions labelled with dominant concept Ground truth created for this work ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 22 of 26
Overview of dataset
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 23 of 26
Classification Rate over Training Set Size
0.85 0.80 0.75
Low-level classification FCSP BIP
0.70 0.65 0.60 0.55 0.50 50
100
ISWeb - Information Systems & Semantic Web
150
200
250
Carsten Saathoff [email protected]
300
350
SAMT 2008 24 of 26
400
F-Measure per Concept Training Set with 300 images (best performing one for BIP) 1 0.9 0.8 0.7 0.6 Low-level FCSP BIP
0.5 0.4 0.3 0.2 0.1 0 snow
road
mountain foliage
ISWeb - Information Systems & Semantic Web
person
sand
Carsten Saathoff [email protected]
sky sailing-boat sea
SAMT 2008 25 of 26
building average
Conclusions
Combination of Wavelets based region classification Spatial context with explicit knowledge BIP outperforms FCSP Classification rate Efficiency (BIP avg ~1 sec, FCSP ~40secs) Validation of earlier observation Explicit knowledge provides good model for spatial context exploitation Low amount of training data required to acquire good performing models
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 26 of 26
Thanks!
ISWeb - Information Systems & Semantic Web
Carsten Saathoff [email protected]
SAMT 2008 27 of 26
