arabic online
ELE 491 - Senior Design II
ARABIC ONLINE TEXT-‐INDEPENDENT WRITER RECOGNITION Adam Aboudan Hasnaa Rabbat Lara Kanbar Advised by: Dr. Khaled Assaleh Date: 24/05/2011 College of Engineering -‐ Department of Electrical Engineering
OUTLINE ! Purpose ! Objective ! Literature Review ! Methodology ! Results ! Future Work
2
PURPOSE ! Personal Recognition System ! Online Text-‐Independent
3
OBJECTIVES ! High Recognition Rate ! Real-‐time ! Market: § Banks § Criminal Justice § Fraud Prevention
4
LITERATURE REVIEW ! Current State of the Problem ! Drawbacks
5
PROPOSED APPROACH ! Novel ! Better Performance § Shorter processing time § Shorter data requirements
6
SYSTEM MODES ! Enrollment Mode: Data Acquisition
Preprocessing
! Recognition Mode: Data Acquisition
Preprocessing
Feature Extraction
Feature Extraction
Evaluating User Model
Classifier Training
Decision Logic
7
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Digitizer Interface Software
8
DATABASE
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! The database consists of 35 enrolled writers: § 2 repetitions of training data per user § 10 repetitions of each test sentence per user
9
DATABASE
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Training Data
10
DATABASE
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Testing Data
11
SYSTEM
Data Acquisition
Pre-Processing
Feature Extraction
Classifier Training
! Removes zero-‐ pressure data. ! Implements segmentation on the writing. ! Removes dots and other diacritics 12
SYSTEM
Data Acquisition
Pre-Processing
Feature Extraction
Classifier Training
! Segmentation Process
13
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Feature Extraction Process
14
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Statistical Based Features § Mean of Pressure § Mean of Altitude § Mean of Azimuth § Standard Deviation of Pressure § Standard Deviation of Azimuth
15
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Shape Based Features § Height § Number of Points § Accumulated Length § Accumulated Angle to Horizontal § Accumulated Angle to Vertical § Number of Movements to the Left 16
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Implemented ClassiWiers: § Discriminative ClassiWiers o Neural Networks (NN) o Polynomial Classifier
§ Non-‐Discriminative ClassiWiers o K-Nearest Neighbors (k-NN) o Naïve Bayesian Classifier o Gaussian Mixture Models (GMM)
17
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Neural Networks (NN)
18
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Neural Networks (NN) § The artiWicial neuron
19
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Neural Networks (NN) – ClassiWier Training
20
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! Neural Networks (NN)– Decision Metric
21
SYSTEM ! Neural Networks (NN) – Results (All Test Sentences) Users/ Hidden Nodes
10 users
20 users
Rate
Std.
Rate
Std.
5
58.83
5.44
34.08
5.26
11
89.10
3.14
58.38
1.21
22
93.08
1.38
71.32
1.20
33
92.35
1.23
73.33
0.74
44
92.90
0.91
Training Fails
30 users
35 users
Training Fails
22
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Polynomial ClassiWier – ClassiWier Training
23
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! Polynomial ClassiWier – Decision Metric
Score1 Score2 …. Score35
24
SYSTEM ! Polynomial ClassiWier – Results (All Test Sentences)
Users/ Order
10 users 20 users 30 users 35 users
1
73.75
40.12
32.33
30.92
2
87.75
72.62
64.17
60.50
3
94.00
77.87
74.75
70.71
25
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
§ Discriminative ClassiWiers o Neural Networks (NN) o Polynomial Classifier
§ Non-‐Discriminative ClassiWiers o K-Nearest Neighbors (k-NN) o Naïve Bayesian Classifier o Gaussian Mixture Models (GMM)
26
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! K-‐Nearest Neighbors (k-‐NN) – ClassiWier Training § Feature Vectors
User X
Feature Extraction
User X’s Model – Segment Feature Vectors SV1 1 SV1 2 ….
SV1 11
SV2 1 SV2 2 ….
SV2 11
• • •
SVn 1 SVn 2 ….
SVn 11
27
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! K-‐Nearest Neighbors (k-‐NN) – Decision Metric Test Point Class A
Class B
2 neighbors
1 neighbor
28
SYSTEM ! k-‐NN – Results (All Test Sentences)
Users/ Nearest Neighbors
10 users
20 users
30 users
35 users
1 NN
89.50
71.75
67.17
63.93
10 NN
91.00
71.88
68.33
65.5
50 NN
91.75
69.25
66.25
64.43
100 NN
91.75
68.13
64.33
61.93
150 NN
90.50
66.88
63.17
60.29
200 NN
90.50
64.75
62.75
59.71 29
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Naïve Bayesian ClassiWier – ClassiWier Training § Utilize Bayes Rule:
§ Priors
30
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! Naïve Bayesian ClassiWier – Decision Metric Segment Feature Vector User A SV1 1 SV1 2 …. SV2 1 SV2 2 ….
• • •
SVn 1 SVn 2 ….
User A
SV1 11 SV2 11
SVn 11
Naïve Bayesian Classifier
User A • • •
User B
User A
User A User A
31
SYSTEM ! Naïve Bayesian ClassiWier – Results (All Test Sentences) § Two different Distribution Densities: o Kernel o Gaussian
Distribution
10 users 20 users 30 users 35 users
Kernel
91.00
75.75
69.25
65.07
Gaussian
93.5
67.75
62.83
58.50
32
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! GMM – ClassiWier Training § A weighted sum of several component Gaussian densities:
where , M is the number of Gaussian components, x is a D-‐dimensional data vector (a set of features), wi = 1,..., M are the mixture weights, g(.) are the component densities.
33
SYSTEM
Data Acquisition
Preprocessing
Classifier Training
Feature Extraction
! GMM -‐ ClassiWier Training 4
3
3
2
2
1
1
0
0
y
4
-1
-1
-2
-2
-3
-3
-4 -4
-3
-2
-1
0
1
2
3
4
-4 -4
pdf(obj,[x,y])
-3
-2
-1
0 x
1
2
3
4
34
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! GMM – Decision Metric (IdentiWication) § Negative of the log-‐likelihood Negative of log-likelihood Score User A
Test Sentence User B
GMM Classifier
Score User B
• • •
User B
Score User C
35
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! GMM – Decision Metric (VeriWication) § Maximum Likelihood Estimation (MLE) § Cohort – based § Number of Cohorts: 5
36
SYSTEM ! GMM – Results (All Test Sentences) § For different number of components: 3 No. of Components Rate Std. Max
2
76.47
0.89
3
77.12
0.90
5
77.76
0.77
79.50
10
78.30
0.70
80.21
78.36 79.43
37
RESULTS ! Best IdentiWication Rates for all the systems for 35 users (Test Sentence 2) Classifier
Identification R a t e
GMM
KNN
NN
NB
Polynomial
Rate
Rate
Rate
Rate
Rate
96.00
80.57
NA
76.00
88.86
38
RESULTS
Identification Rates for All Classifiers
100 3 Component GMM 3rd Order Polynomial Kernel Naive Bayesian 10 KNN
95
Identification Rate
90
85
80
75
70
65 10
15
20 25 Number of Users
30
35
39
RESULTS
Identification Rate vs. Number of Test Segments
100
90
Identification Rate (%)
80
70
60
50
40
30
0
2
4
6 8 10 Number of Test Segments
12
14
40
RESULTS ! Optimal System: GMM Results for 35 users.
Test
Identification
Test Sentence/ Rate
1
2
3
4
All
Rate
63.11
93.05
77.59
74.75
77.12
Std.
1.52
1.03
1.45
1.39
0.9
Max.
67.43
96.00
81.14
79.71
79.43
41
RESULTS ! Optimal System: GMM Results for 35 users. Test Sentence/ Rate
1
2
3
4
2,3 & 4
FAR
2.47
2.16
2.37
2.05
6
FRR
9.71
1.33
6.10
6.67
15.05
42
FUTURE WORK ! Expand Writer Database ! Optimize Test Words ! Extract More Features ! Enhance ClassiWiers
43
REFERENCES [1] Anoop Namboodiri ; Sachin Gupta; , “Text Independent Writer IdentiWication from Online Handwriting,” International Institute of Information Technology, Gachibowli, Hyderabad, India [email protected], sachin [email protected] [2] Ming-‐Yen Tsai; Leu-‐Shing Lan; , "Online Writer IdentiWication Using The Point Distribution Model," Systems, Man and Cybernetics, 2005 IEEE International Conference on, vol.2, no., pp.1264-‐1268, 12-‐12 Oct. 2005 doi: 10.1109/ICSMC. 2005.1571320 URL: http://ieeexplore.ieee.org/stamp/stamp.jsp? tp=&arnumber=1571320&isnumber=33255 [3] Andreas Schlapbach, Marcus Liwicki, Horst Bunke, A writer identiWication system for on-‐line whiteboard data, Pattern Recognition, Volume 41, Issue 7, July 2008, Pages 2381-‐2397, ISSN 0031-‐3203, DOI: 10.1016/j.patcog.2008.01.006. ( http://www.sciencedirect.com/science/article/ B6V14-‐4RM7MT9-‐3/2/492d5c1c879d5546e922cbd5ed4a89bd)
44
THANK YOU!
45
APPENDIX
46
APPENDIX-‐CONTINUED
47
APPENDIX-‐CONTINUED
48
APPENDIX – CONTINUED ! Polynomial ClassiWiers: § The polynomial expansion of a D-‐dimensional feature vector x = [x1 x2 ... xD] to a K-‐dimensional vector involves developing a set of basis functions as shown below:
§ where ki is a positive integer and the sum of all the ki coefWicients is between 0 and Q:
49
APPENDIX – CONTINUED ! Polynomial ClassiWiers: § These basis functions are used to expand each class’s set of N feature vectors (NxD matrix) Xj = [xj,1 xj,2 … xj, N]T into an N x DQ matrix, where DQ is:
§ and is the number of distinct element subsets that can be made out of elements.
50
APPENDIX – CONTINUED ! Kernel Density Distribution:
§ Kernel Smoothing Function we used: o Gaussian
51
APPENDIX -‐ CONTINUED ! MAP (Maximum a posteriori) § Used in Naïve Bayes ClassiWiers
52
APPENDIX -‐ CONTINUED ! MLE (Maximum Likelihood Estimation) § One of the methods to estimate the parameters of the GMM model § ML estimation determines the model parameters that maximize the likelihood of the GMM based on the feature vectors provided. § The GMM likelihood for a set of T independent training vectors X = {x1, ... , xT }, is:
53
APPENDIX -‐ CONTINUED ! MLE § Likelihood: § Log-‐likelihood
54
.
APPENDIX -‐ CONTINUED ! Expectation Maximization (EM) Algorithm § Utilizing a special case of the Expectation Maximization (EM) algorithm iteratively can give estimates of the ML parameters. The EM algorithm initializes a model using a form of binary VQ estimation and then uses it to calculate new model such that : § This new will become the initial model for the next iteration and the process continues until some form of convergence occurs. 55
APPENDIX -‐ CONTINUED ! Vector Quantization (VQ) § A quantization technique which allows the modelling of probability density functions by the distribution of prototype vectors. It divides a large set of points (vectors) into groups having approximately the same number of points closest to them. Each of these groups is represented by its centroid point, as is the case in k-‐ means and some other clustering algorithms.
56
APPENDIX -‐ CONTINUED ! GUI:
57
ARABIC ONLINE TEXT-‐INDEPENDENT WRITER RECOGNITION Adam Aboudan Hasnaa Rabbat Lara Kanbar Advised by: Dr. Khaled Assaleh Date: 24/05/2011 College of Engineering -‐ Department of Electrical Engineering
OUTLINE ! Purpose ! Objective ! Literature Review ! Methodology ! Results ! Future Work
2
PURPOSE ! Personal Recognition System ! Online Text-‐Independent
3
OBJECTIVES ! High Recognition Rate ! Real-‐time ! Market: § Banks § Criminal Justice § Fraud Prevention
4
LITERATURE REVIEW ! Current State of the Problem ! Drawbacks
5
PROPOSED APPROACH ! Novel ! Better Performance § Shorter processing time § Shorter data requirements
6
SYSTEM MODES ! Enrollment Mode: Data Acquisition
Preprocessing
! Recognition Mode: Data Acquisition
Preprocessing
Feature Extraction
Feature Extraction
Evaluating User Model
Classifier Training
Decision Logic
7
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Digitizer Interface Software
8
DATABASE
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! The database consists of 35 enrolled writers: § 2 repetitions of training data per user § 10 repetitions of each test sentence per user
9
DATABASE
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Training Data
10
DATABASE
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Testing Data
11
SYSTEM
Data Acquisition
Pre-Processing
Feature Extraction
Classifier Training
! Removes zero-‐ pressure data. ! Implements segmentation on the writing. ! Removes dots and other diacritics 12
SYSTEM
Data Acquisition
Pre-Processing
Feature Extraction
Classifier Training
! Segmentation Process
13
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Feature Extraction Process
14
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Statistical Based Features § Mean of Pressure § Mean of Altitude § Mean of Azimuth § Standard Deviation of Pressure § Standard Deviation of Azimuth
15
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Shape Based Features § Height § Number of Points § Accumulated Length § Accumulated Angle to Horizontal § Accumulated Angle to Vertical § Number of Movements to the Left 16
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Implemented ClassiWiers: § Discriminative ClassiWiers o Neural Networks (NN) o Polynomial Classifier
§ Non-‐Discriminative ClassiWiers o K-Nearest Neighbors (k-NN) o Naïve Bayesian Classifier o Gaussian Mixture Models (GMM)
17
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Neural Networks (NN)
18
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Neural Networks (NN) § The artiWicial neuron
19
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Neural Networks (NN) – ClassiWier Training
20
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! Neural Networks (NN)– Decision Metric
21
SYSTEM ! Neural Networks (NN) – Results (All Test Sentences) Users/ Hidden Nodes
10 users
20 users
Rate
Std.
Rate
Std.
5
58.83
5.44
34.08
5.26
11
89.10
3.14
58.38
1.21
22
93.08
1.38
71.32
1.20
33
92.35
1.23
73.33
0.74
44
92.90
0.91
Training Fails
30 users
35 users
Training Fails
22
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Polynomial ClassiWier – ClassiWier Training
23
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! Polynomial ClassiWier – Decision Metric
Score1 Score2 …. Score35
24
SYSTEM ! Polynomial ClassiWier – Results (All Test Sentences)
Users/ Order
10 users 20 users 30 users 35 users
1
73.75
40.12
32.33
30.92
2
87.75
72.62
64.17
60.50
3
94.00
77.87
74.75
70.71
25
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
§ Discriminative ClassiWiers o Neural Networks (NN) o Polynomial Classifier
§ Non-‐Discriminative ClassiWiers o K-Nearest Neighbors (k-NN) o Naïve Bayesian Classifier o Gaussian Mixture Models (GMM)
26
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! K-‐Nearest Neighbors (k-‐NN) – ClassiWier Training § Feature Vectors
User X
Feature Extraction
User X’s Model – Segment Feature Vectors SV1 1 SV1 2 ….
SV1 11
SV2 1 SV2 2 ….
SV2 11
• • •
SVn 1 SVn 2 ….
SVn 11
27
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! K-‐Nearest Neighbors (k-‐NN) – Decision Metric Test Point Class A
Class B
2 neighbors
1 neighbor
28
SYSTEM ! k-‐NN – Results (All Test Sentences)
Users/ Nearest Neighbors
10 users
20 users
30 users
35 users
1 NN
89.50
71.75
67.17
63.93
10 NN
91.00
71.88
68.33
65.5
50 NN
91.75
69.25
66.25
64.43
100 NN
91.75
68.13
64.33
61.93
150 NN
90.50
66.88
63.17
60.29
200 NN
90.50
64.75
62.75
59.71 29
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! Naïve Bayesian ClassiWier – ClassiWier Training § Utilize Bayes Rule:
§ Priors
30
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! Naïve Bayesian ClassiWier – Decision Metric Segment Feature Vector User A SV1 1 SV1 2 …. SV2 1 SV2 2 ….
• • •
SVn 1 SVn 2 ….
User A
SV1 11 SV2 11
SVn 11
Naïve Bayesian Classifier
User A • • •
User B
User A
User A User A
31
SYSTEM ! Naïve Bayesian ClassiWier – Results (All Test Sentences) § Two different Distribution Densities: o Kernel o Gaussian
Distribution
10 users 20 users 30 users 35 users
Kernel
91.00
75.75
69.25
65.07
Gaussian
93.5
67.75
62.83
58.50
32
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Classifier Training
! GMM – ClassiWier Training § A weighted sum of several component Gaussian densities:
where , M is the number of Gaussian components, x is a D-‐dimensional data vector (a set of features), wi = 1,..., M are the mixture weights, g(.) are the component densities.
33
SYSTEM
Data Acquisition
Preprocessing
Classifier Training
Feature Extraction
! GMM -‐ ClassiWier Training 4
3
3
2
2
1
1
0
0
y
4
-1
-1
-2
-2
-3
-3
-4 -4
-3
-2
-1
0
1
2
3
4
-4 -4
pdf(obj,[x,y])
-3
-2
-1
0 x
1
2
3
4
34
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! GMM – Decision Metric (IdentiWication) § Negative of the log-‐likelihood Negative of log-likelihood Score User A
Test Sentence User B
GMM Classifier
Score User B
• • •
User B
Score User C
35
SYSTEM
Data Acquisition
Preprocessing
Feature Extraction
Evaluating User Model
Decision Logic
! GMM – Decision Metric (VeriWication) § Maximum Likelihood Estimation (MLE) § Cohort – based § Number of Cohorts: 5
36
SYSTEM ! GMM – Results (All Test Sentences) § For different number of components: 3 No. of Components Rate Std. Max
2
76.47
0.89
3
77.12
0.90
5
77.76
0.77
79.50
10
78.30
0.70
80.21
78.36 79.43
37
RESULTS ! Best IdentiWication Rates for all the systems for 35 users (Test Sentence 2) Classifier
Identification R a t e
GMM
KNN
NN
NB
Polynomial
Rate
Rate
Rate
Rate
Rate
96.00
80.57
NA
76.00
88.86
38
RESULTS
Identification Rates for All Classifiers
100 3 Component GMM 3rd Order Polynomial Kernel Naive Bayesian 10 KNN
95
Identification Rate
90
85
80
75
70
65 10
15
20 25 Number of Users
30
35
39
RESULTS
Identification Rate vs. Number of Test Segments
100
90
Identification Rate (%)
80
70
60
50
40
30
0
2
4
6 8 10 Number of Test Segments
12
14
40
RESULTS ! Optimal System: GMM Results for 35 users.
Test
Identification
Test Sentence/ Rate
1
2
3
4
All
Rate
63.11
93.05
77.59
74.75
77.12
Std.
1.52
1.03
1.45
1.39
0.9
Max.
67.43
96.00
81.14
79.71
79.43
41
RESULTS ! Optimal System: GMM Results for 35 users. Test Sentence/ Rate
1
2
3
4
2,3 & 4
FAR
2.47
2.16
2.37
2.05
6
FRR
9.71
1.33
6.10
6.67
15.05
42
FUTURE WORK ! Expand Writer Database ! Optimize Test Words ! Extract More Features ! Enhance ClassiWiers
43
REFERENCES [1] Anoop Namboodiri ; Sachin Gupta; , “Text Independent Writer IdentiWication from Online Handwriting,” International Institute of Information Technology, Gachibowli, Hyderabad, India [email protected], sachin [email protected] [2] Ming-‐Yen Tsai; Leu-‐Shing Lan; , "Online Writer IdentiWication Using The Point Distribution Model," Systems, Man and Cybernetics, 2005 IEEE International Conference on, vol.2, no., pp.1264-‐1268, 12-‐12 Oct. 2005 doi: 10.1109/ICSMC. 2005.1571320 URL: http://ieeexplore.ieee.org/stamp/stamp.jsp? tp=&arnumber=1571320&isnumber=33255 [3] Andreas Schlapbach, Marcus Liwicki, Horst Bunke, A writer identiWication system for on-‐line whiteboard data, Pattern Recognition, Volume 41, Issue 7, July 2008, Pages 2381-‐2397, ISSN 0031-‐3203, DOI: 10.1016/j.patcog.2008.01.006. ( http://www.sciencedirect.com/science/article/ B6V14-‐4RM7MT9-‐3/2/492d5c1c879d5546e922cbd5ed4a89bd)
44
THANK YOU!
45
APPENDIX
46
APPENDIX-‐CONTINUED
47
APPENDIX-‐CONTINUED
48
APPENDIX – CONTINUED ! Polynomial ClassiWiers: § The polynomial expansion of a D-‐dimensional feature vector x = [x1 x2 ... xD] to a K-‐dimensional vector involves developing a set of basis functions as shown below:
§ where ki is a positive integer and the sum of all the ki coefWicients is between 0 and Q:
49
APPENDIX – CONTINUED ! Polynomial ClassiWiers: § These basis functions are used to expand each class’s set of N feature vectors (NxD matrix) Xj = [xj,1 xj,2 … xj, N]T into an N x DQ matrix, where DQ is:
§ and is the number of distinct element subsets that can be made out of elements.
50
APPENDIX – CONTINUED ! Kernel Density Distribution:
§ Kernel Smoothing Function we used: o Gaussian
51
APPENDIX -‐ CONTINUED ! MAP (Maximum a posteriori) § Used in Naïve Bayes ClassiWiers
52
APPENDIX -‐ CONTINUED ! MLE (Maximum Likelihood Estimation) § One of the methods to estimate the parameters of the GMM model § ML estimation determines the model parameters that maximize the likelihood of the GMM based on the feature vectors provided. § The GMM likelihood for a set of T independent training vectors X = {x1, ... , xT }, is:
53
APPENDIX -‐ CONTINUED ! MLE § Likelihood: § Log-‐likelihood
54
.
APPENDIX -‐ CONTINUED ! Expectation Maximization (EM) Algorithm § Utilizing a special case of the Expectation Maximization (EM) algorithm iteratively can give estimates of the ML parameters. The EM algorithm initializes a model using a form of binary VQ estimation and then uses it to calculate new model such that : § This new will become the initial model for the next iteration and the process continues until some form of convergence occurs. 55
APPENDIX -‐ CONTINUED ! Vector Quantization (VQ) § A quantization technique which allows the modelling of probability density functions by the distribution of prototype vectors. It divides a large set of points (vectors) into groups having approximately the same number of points closest to them. Each of these groups is represented by its centroid point, as is the case in k-‐ means and some other clustering algorithms.
56
APPENDIX -‐ CONTINUED ! GUI:
57
