INTRODUCTION TO MACHINE LEARNING
INTRODUCTION TO MACHINE LEARNING
Regression: Simple and Linear
Introduction to Machine Learning
Regression Principle
PREDICTORS
REGRESSION
RESPONSE
Introduction to Machine Learning
Example Shop Data: sales, competition, district size, ... Data Analyst
Relationship?
●
Predictors: competition, advertisement, …
●
Response: sales
Shopkeeper
Predictions!
Introduction to Machine Learning
Simple Linear Regression ●
Simple: one predictor to model the response
●
Linear: approximately linear relationship
Linearity Plausible?
Sca!erplot!
Introduction to Machine Learning
Example ●
Relationship: advertisement
●
Expectation: positively correlated
sales
Introduction to Machine Learning
Example Observation: upwards linear trend
●
First Step: simple linear regression
300 200 100 0
sales
400
500
●
5
10
advertisement
15
Introduction to Machine Learning
Model Fi!ing a line
●
Predictor:
●
Intercept:
●
Response:
●
Slope:
●
Statistical Error:
Introduction to Machine Learning
Estimating Coefficients
Minimize!
300 300 300 200 200 200 100 100 100
#Observations
0 00
sales sales
400 400 400
Fi!ed Response Sales
True Response
500 500 500
Residuals
555
10 10 10
advertisement advertisement
Advertisement
15 15 15
Introduction to Machine Learning
Predictor
Returns coefficients
0
> my_lm$coefficients
100
200
300
> my_lm y_new summary(my_lm)$r.squared
good fit! Example: 0.84
INTRODUCTION TO MACHINE LEARNING
Let’s practice!
INTRODUCTION TO MACHINE LEARNING
Multivariable Linear Regression
Introduction to Machine Learning
Example
Loss of information!
300 0
100
> lm(sales ~ comp, data = shop_data)
200
> lm(sales ~ ads, data = shop_data)
sales
400
500
Simple Linear Regression:
0
5
5
10
10
nearby competition
15
15
Introduction to Machine Learning
Multi-Linear Model Solution: combine in multi linear model! ●
Higher predictive power
●
Higher accuracy
Individual Effect
Introduction to Machine Learning
Multi-Linear Regression Model
●
Predictors:
●
Response:
●
Statistical Error:
●
Coefficients:
Introduction to Machine Learning
Estimating Coefficients Residuals True Response
Fi!ed Response
#Observations
Minimize!
Introduction to Machine Learning
Extending! More predictors: total inventory, district size, … Extend methodology to p predictors:
Response
Predictors
> my_lm summary(my_lm)$adj.r.squared
In Example:
0.819
0.906
Introduction to Machine Learning
Influence of predictors ●
p-value: indicator influence of parameter
●
p-value low — more likely parameter has significant influence > summary(my_lm) Call: lm(formula = sales ~ ads + comp, data = shop_data) Residuals: Min 1Q -131.920 -23.009
Median -4.448
3Q 33.978
Max 146.486
Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 228.740 80.592 2.838 0.009084 ** ads 25.521 5.900 4.325 0.000231 *** comp -19.234 4.549 -4.228 0.000296 ***
P-Values
Introduction to Machine Learning
Example ●
Want 95% confidence — p-value plot(lm_shop$fitted.values, > qqnorm(lm_shop$residuals)
lm_shop$residuals)
Draws normal Q-Q plot
1 500
600 2
Introduction to Machine Learning
Verfiying Assumptions Normal Q−Q Plot
50 0 −100
−50
0 −100
−50
Residuals
50
Residual Quantiles
100
100
150
150
Residual Plot
0
100
200
300
400
500
600
Estimated Sales
●
Important to avoid mistakes!
●
Alternative tests exist
−2
−1
0
Theoretical Quantiles
1
2
INTRODUCTION TO MACHINE LEARNING
Let’s practice!
INTRODUCTION TO MACHINE LEARNING
k-Nearest Neighbors and Generalization
Introduction to Machine Learning
Non-Parametric Regression
50 48 46 44
y
52
54
56
Problem: Visible pa!ern, but not linear
2
3
4
x
5
6
Introduction to Machine Learning
Non-Parametric Regression Problem: Visible pa!ern, but not linear Solutions: ●
Transformation
Tedious
●
Multi-linear Regression
Advanced
●
non-Parametric Regression
Doable
Introduction to Machine Learning
Non-Parametric Regression Problem: Visible pa!ern, but not linear Techniques: ●
k-Nearest Neighbors
●
Kernel Regression
●
Regression Trees
●
…
No parameter estimations required!
Introduction to Machine Learning
k-NN: Algorithm
47.0 46.5 46.0
New observation
45.5
y
47.5
48.0
48.5
Given a training set and a new observation:
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
k-NN: Algorithm Given a training set and a new observation:
47.0 46.5 46.0 45.5
y
47.5
48.0
48.5
1. Calculate the distance in the predictors
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
k-NN: Algorithm Given a training set and a new observation:
47.0 46.5 46.0 45.5
k=4
y
47.5
48.0
48.5
2. Select the k nearest
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
k-NN: Algorithm Given a training set and a new observation:
48.0
48.5
3. Aggregate the response of the k nearest
47.0 46.5 46.0 45.5
y
47.5
Mean of 4 responses
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
k-NN: Algorithm Given a training set and a new observation:
48.0
48.5
4. The outcome is your prediction
47.0 46.5 46.0 45.5
y
47.5
Prediction
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
Choosing k ●
k = 1: Perfect fit on training set but poor predictions
●
k = #obs in training set: Mean, also poor predictions
Bias - Variance trade off! Reasonable: k = 20% of #obs in training set
Introduction to Machine Learning
Generalization in Regression ●
Built your own regression model
●
Worked on training set
●
Does it generalize well?!
●
Two techniques ●
Hold Out: simply split the dataset
●
K-fold cross-validation
Introduction to Machine Learning
Hold Out Method for Regression Training set
Test set
Build regression model on training set
Predict the outcome of the test set
Calculate RMSE within training set
Calculate the RMSE within test set
Compare Test RMSE and Training RMSE
Introduction to Machine Learning
Under and Overfi!ing
2
3
4
5
6
7
8
10 8 −2
0
2
y
4
6
8 6 4 −2
0
2
y −2
0
2
y
4
6
8
10
Overfit
10
Underfit
2
3
x
4
5
6
7
8
2
3
x
4
5
6
7
x
●
Fit:
✘
●
Fit:
✔
●
Fit:
✔
●
Generalize:
✔
●
Generalize:
✔
●
Generalize:
✘
●
Prediction:
✘
●
Prediction:
✔
●
Prediction:
✘
8
INTRODUCTION TO MACHINE LEARNING
Let’s practice!
Regression: Simple and Linear
Introduction to Machine Learning
Regression Principle
PREDICTORS
REGRESSION
RESPONSE
Introduction to Machine Learning
Example Shop Data: sales, competition, district size, ... Data Analyst
Relationship?
●
Predictors: competition, advertisement, …
●
Response: sales
Shopkeeper
Predictions!
Introduction to Machine Learning
Simple Linear Regression ●
Simple: one predictor to model the response
●
Linear: approximately linear relationship
Linearity Plausible?
Sca!erplot!
Introduction to Machine Learning
Example ●
Relationship: advertisement
●
Expectation: positively correlated
sales
Introduction to Machine Learning
Example Observation: upwards linear trend
●
First Step: simple linear regression
300 200 100 0
sales
400
500
●
5
10
advertisement
15
Introduction to Machine Learning
Model Fi!ing a line
●
Predictor:
●
Intercept:
●
Response:
●
Slope:
●
Statistical Error:
Introduction to Machine Learning
Estimating Coefficients
Minimize!
300 300 300 200 200 200 100 100 100
#Observations
0 00
sales sales
400 400 400
Fi!ed Response Sales
True Response
500 500 500
Residuals
555
10 10 10
advertisement advertisement
Advertisement
15 15 15
Introduction to Machine Learning
Predictor
Returns coefficients
0
> my_lm$coefficients
100
200
300
> my_lm y_new summary(my_lm)$r.squared
good fit! Example: 0.84
INTRODUCTION TO MACHINE LEARNING
Let’s practice!
INTRODUCTION TO MACHINE LEARNING
Multivariable Linear Regression
Introduction to Machine Learning
Example
Loss of information!
300 0
100
> lm(sales ~ comp, data = shop_data)
200
> lm(sales ~ ads, data = shop_data)
sales
400
500
Simple Linear Regression:
0
5
5
10
10
nearby competition
15
15
Introduction to Machine Learning
Multi-Linear Model Solution: combine in multi linear model! ●
Higher predictive power
●
Higher accuracy
Individual Effect
Introduction to Machine Learning
Multi-Linear Regression Model
●
Predictors:
●
Response:
●
Statistical Error:
●
Coefficients:
Introduction to Machine Learning
Estimating Coefficients Residuals True Response
Fi!ed Response
#Observations
Minimize!
Introduction to Machine Learning
Extending! More predictors: total inventory, district size, … Extend methodology to p predictors:
Response
Predictors
> my_lm summary(my_lm)$adj.r.squared
In Example:
0.819
0.906
Introduction to Machine Learning
Influence of predictors ●
p-value: indicator influence of parameter
●
p-value low — more likely parameter has significant influence > summary(my_lm) Call: lm(formula = sales ~ ads + comp, data = shop_data) Residuals: Min 1Q -131.920 -23.009
Median -4.448
3Q 33.978
Max 146.486
Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 228.740 80.592 2.838 0.009084 ** ads 25.521 5.900 4.325 0.000231 *** comp -19.234 4.549 -4.228 0.000296 ***
P-Values
Introduction to Machine Learning
Example ●
Want 95% confidence — p-value plot(lm_shop$fitted.values, > qqnorm(lm_shop$residuals)
lm_shop$residuals)
Draws normal Q-Q plot
1 500
600 2
Introduction to Machine Learning
Verfiying Assumptions Normal Q−Q Plot
50 0 −100
−50
0 −100
−50
Residuals
50
Residual Quantiles
100
100
150
150
Residual Plot
0
100
200
300
400
500
600
Estimated Sales
●
Important to avoid mistakes!
●
Alternative tests exist
−2
−1
0
Theoretical Quantiles
1
2
INTRODUCTION TO MACHINE LEARNING
Let’s practice!
INTRODUCTION TO MACHINE LEARNING
k-Nearest Neighbors and Generalization
Introduction to Machine Learning
Non-Parametric Regression
50 48 46 44
y
52
54
56
Problem: Visible pa!ern, but not linear
2
3
4
x
5
6
Introduction to Machine Learning
Non-Parametric Regression Problem: Visible pa!ern, but not linear Solutions: ●
Transformation
Tedious
●
Multi-linear Regression
Advanced
●
non-Parametric Regression
Doable
Introduction to Machine Learning
Non-Parametric Regression Problem: Visible pa!ern, but not linear Techniques: ●
k-Nearest Neighbors
●
Kernel Regression
●
Regression Trees
●
…
No parameter estimations required!
Introduction to Machine Learning
k-NN: Algorithm
47.0 46.5 46.0
New observation
45.5
y
47.5
48.0
48.5
Given a training set and a new observation:
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
k-NN: Algorithm Given a training set and a new observation:
47.0 46.5 46.0 45.5
y
47.5
48.0
48.5
1. Calculate the distance in the predictors
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
k-NN: Algorithm Given a training set and a new observation:
47.0 46.5 46.0 45.5
k=4
y
47.5
48.0
48.5
2. Select the k nearest
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
k-NN: Algorithm Given a training set and a new observation:
48.0
48.5
3. Aggregate the response of the k nearest
47.0 46.5 46.0 45.5
y
47.5
Mean of 4 responses
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
k-NN: Algorithm Given a training set and a new observation:
48.0
48.5
4. The outcome is your prediction
47.0 46.5 46.0 45.5
y
47.5
Prediction
4.5
4.6
4.7
4.8 x
4.9
5.0
Introduction to Machine Learning
Choosing k ●
k = 1: Perfect fit on training set but poor predictions
●
k = #obs in training set: Mean, also poor predictions
Bias - Variance trade off! Reasonable: k = 20% of #obs in training set
Introduction to Machine Learning
Generalization in Regression ●
Built your own regression model
●
Worked on training set
●
Does it generalize well?!
●
Two techniques ●
Hold Out: simply split the dataset
●
K-fold cross-validation
Introduction to Machine Learning
Hold Out Method for Regression Training set
Test set
Build regression model on training set
Predict the outcome of the test set
Calculate RMSE within training set
Calculate the RMSE within test set
Compare Test RMSE and Training RMSE
Introduction to Machine Learning
Under and Overfi!ing
2
3
4
5
6
7
8
10 8 −2
0
2
y
4
6
8 6 4 −2
0
2
y −2
0
2
y
4
6
8
10
Overfit
10
Underfit
2
3
x
4
5
6
7
8
2
3
x
4
5
6
7
x
●
Fit:
✘
●
Fit:
✔
●
Fit:
✔
●
Generalize:
✔
●
Generalize:
✔
●
Generalize:
✘
●
Prediction:
✘
●
Prediction:
✔
●
Prediction:
✘
8
INTRODUCTION TO MACHINE LEARNING
Let’s practice!
