EMi P i EM in Practice

Probabilistic Graphical Models

Learning

Incomplete Data

EM i P EM in Practice i Daphne Koller

Parameterr value

Train LL/ /instance

EM Convergence in Practice

Iteration G. Elidan

Iteration Daphne Koller

Train LL L/instance

Test LL L/instance

Overfitting

Iteration • •

Iteration

Early stopping using cross validation Use MAP with parameter priors rather than MLE

G. Elidan

Daphne Koller

Local Optima # distinct loc cal optima

hidden var

G. Elidan

50% missing

25% missing

sample size M

Daphne Koller

log-likelihoo od value

Significance of Local Optima

G. Elidan

% of runs achieving log-likelihood value

Daphne Koller

Initialization is Critical • Multiple random restarts • From prior knowledge • From F the th output t t of f a simpler i l algorithm l ith

Daphne Koller

Summary • Convergence of likelihood ≠ convergence of parameters • Running to convergence can lead to overfitting • Local optima are unavoidable, and increase with the amount of missing data • Local optima can be very different • Initialization is critical

Daphne Koller