Exploration of Bagging Ensembles Comprising ... - Semantic Scholar
Exploration of Bagging Ensembles Comprising Genetic Fuzzy Models to Assist with Real Estate Appraisals Tadeusz Lasota1, Zbigniew Telec2, Bogdan Trawiński2, Krzysztof Trawiński3, 1
Wrocław University of Environmental and Life Sciences, Dept. of Spatial Management Ul. Norwida 25/27, 50-375 Wroclaw, Poland 2 Wrocław University of Technology, Institute of Informatics, Wybrzeże Wyspiańskiego 27, 50-370 Wrocław, Poland 3 European Centre for Soft Computing, Edificio Científico-Tecnológico, 3ª Planta, C. Gonzalo Gutiérrez Quirós S/N, 33600 Mieres, Asturias, Spain [email protected], {zbigniew.telec, bogdan.trawinski}@pwr.wroc.pl, [email protected]
Abstract. The study reported was devoted to investigate to what extent bagging approach could lead to the improvement of the accuracy machine learning regression models. Four algorithms implemented in the KEEL tool, including two evolutionary fuzzy systems, decision trees for regression, and neural network, were used in the experiments. The results showed that some bagging ensembles ensured higher prediction accuracy than single models.
Keywords: bagging ensembles, genetic fuzzy systems, regression models, real estate appraisal, KEEL
1 Introduction Ensemble learning systems combine the output of machine learning algorithms, called “weak learners”, in order to get smaller prediction errors (in regression) or lower error rates (in classification). The individual estimator must provide different patterns of generalization, thus in the training process diversity is employed. Otherwise, the ensemble would be composed of the same predictors and would provide as good accuracy as the single one. It has been proved that the ensemble performs better when each individual machine learning system is accurate and makes errors on different examples [2], [10]. Although, there are many taxonomies for that, there is one recognized group the so-called data resampling, which generates different training sets to obtain unique regressor or classificator. To this group we may include bagging [3], boosting [18], and stacking [21]. The most popular method - bagging, which stands for bootstrap aggregating, is one of the most intuitive and simplest ensemble algorithms providing a good performance. Diversity of regressors is obtained by using bootstrapped replicas of the training data. That is, different training data subsets are randomly drawn with replacement from the original training set. So obtained training data subsets, called also bags, are used then
to train different regression models. Finally, individual regressors are combined through an algebraic expression, such as minimum, maximum, sum, mean, product, median, etc.[16]. Although the ensemble learning systems are more common in classification, they are also applied to regression problems. Bagging has been employed among others to regression trees [4], Gaussian process [7], SVM [22], and neural networks [9]. You can find also some works on applying ensemble fuzzy systems to solve classification [5], [12] and prediction problems [11], but they are not too numerous. In our previous works [13], [14] we tested different machine learning algorithms, among others genetic fuzzy systems trying to select the most appropriate ones to build data driven models for real estate appraisal using MATLAB and KEEL. Since ensemble learning can be used to improve the performance of a model and reduce the likelihood of an unfortunate selection of a poor one, in this paper we focused our efforts into investigating to what extent the bagging approach could lead to the improvement of the accuracy machine learning regression models devoted to assist with real estate appraisals. The concept of a data driven models for premises valuation, presented in the paper, was developed based on the sales comparison method. The architecture of the proposed system is shown in Fig. 1. The appraiser accesses the system through the internet and input the values of the attributes of the premises being evaluated into the system, which calculates the output using a given model. The final result, that is a suggested value of the property, is sent back to the appraiser.
Fig. 1. Information systems to assist with real estate appraisals
Actual data used to generate and learn appraisal models came from the cadastral system and the registry of real estate transactions referring to residential premises sold in one of big Polish cities at market prices within two years 2001 and 2002. They constituted original data set of 1098 cases of sales/purchase transactions. Four attributes were pointed out as price drivers: usable area of premises, floor on which premises were located, year of building construction, number of storeys in the building, in turn, price of premises was the output variable.
2 Plan of Experiments The main goal of our study was to explore the performance of the bagged ensembles of evolutionary fuzzy models to be employed in an internet system assisting property valuation. For this purpose KEEL, a non-commercial Java software tool designed to assess evolutionary algorithms for data mining problems, was used [1]. Two fuzzy rule based systems for regression, namely COR algorithm and Wang-Mendel one combined with genetic tuning were applied to create bagging models. For comparison two other popular algorithms i.e. a multilayer perceptron neural network and a model tree M5 were utilized (see Table 1). Table 1. KEEL algorithms used in study Code COR WMG MLP M5T
KEEL name Regr-COR_GA
Description Genetic fuzzy rule learning using cooperative rules technique [6] Regr-Fuzzy-WM + Post-G-G- Wang-Mendel algorithm with global genetic tuning of the Tuning-FRBSs fuzzy partition [8], [20] Regr-MLPerceptronConj-Grad Multilayer perceptron for modeling [15] Regr-M5 Model tree M5 for regression [17], [19]
Schema of the experiments is depicted in Fig. 2. On the basis of the original data set 28 bootstrap replicates were created. These replicates were used to generate models employing COR, WMG, MLP, and M5T regression algorithms implemented in KEEL. Normalization of data was performed using the min-max approach. As fitness function the mean square error (MSE) was applied and 10-fold cross validation (10cv) was accomplished. As aggregation functions simple averages were used.
Fig. 2. Schema of bagging ensemble model development
The values of three common performance measures were used as the arguments of the aggregation function. They are mean square error (MSE – Formula 1), which strongly penalizes bigger errors, mean absolute percentage error (MAPE – 2), which is a relative measure and easy to interpret and compare different models, and mean absolute error (MAE – 3), which gives values in the same dimension as output variables. The values of the measures were calculated using actual and predicted prices, obtained for testing sets. These measures are expressed in the form of
following formulas below, where yi denotes actual price and 𝑦i – predicted price of ith case, and N – number of cases in the testing set. 𝑁 1 (1) 𝑀𝑆𝐸 = 𝑦𝑖 − 𝑦𝑖 2 𝑁 𝑖=1 𝑀𝐴𝑃𝐸 =
1 𝑁
𝑀𝐴𝐸 =
𝑁 𝑖=1
1 𝑁
𝑦𝑖 − 𝑦𝑖 ∗ 100% 𝑦𝑖 𝑁 𝑖=1
𝑦𝑖 − 𝑦𝑖
(2) (3)
3 Results of Experiments The performance of bagging ensembles comprising from 1 to 28 subsequently drawn bootstrap models is presented in Fig. 3 and 4 for MSE, 5 and 6 for MAPE, and finally 7 and 8 for MAE. It can be observed that, except for some first ensembles, which should be passed over in order to avoid the effect of favourable drawing, the bagging models including 24 bootstrap replicates achieved the best prediction performance. Similar considerations presented Breiman [3], who stated that 25 bootstrap replicates could be reasonable to find the ensemble with the lowest prediction error. Substantial differences between COR and WMG ensembles can be observed. Using WMG brings benefit for any number of bootstrap replicates whereas COR provides gain for ensembles composed of above 18 bags.
Fig. 3. Performance of bagged ensembles for MSE compared with original COR model
Fig. 4. Performance of bagged ensembles for MSE compared with original WMG model
Fig. 5. Performance of bagged ensembles for MAPE compared with original COR model
Fig. 6. Performance of bagged ensembles for MAPE compared with original WMG model
Fig. 7 Performance of bagged ensembles for MAE compared with original COR model
Fig. 8. Performance of bagged ensembles for MAE compared with original WMG model
Table 2. Percentage error reduction of bagged ensembles compared to single models No. of bags 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
COR 11.0% 0.6% 7.4% 5.4% 3.9% 4.2% 4.5% 3.1% 1.8% 1.6% 0.9% 0.5% 0.5% 0.3% 0.7% 0.5% 1.3% 1.3% 2.0% 1.8% 2.4% 2.5% 2.5% 2.5% 2.3% 2.0% 1.9% 1.7%
MAPE WMG 22.5% 11.7% 13.1% 10.5% 9.6% 8.0% 7.8% 6.1% 5.8% 5.1% 4.7% 5.1% 4.8% 4.4% 4.7% 4.3% 4.9% 5.0% 5.5% 5.7% 5.8% 5.9% 6.1% 6.4% 6.3% 6.1% 6.0% 5.9%
MLP 3.1% -3.0% -0.4% -0.9% -0.6% -1.8% -0.5% -1.3% -2.2% -2.9% -3.6% -3.7% -4.3% -4.0% -3.5% -3.5% -2.9% -3.0% -2.5% -2.4% -2.3% -2.2% -1.8% -1.6% -1.4% -1.8% -1.8% -2.1%
M5T 8.3% 8.5% 9.8% 7.0% 5.4% 5.7% 5.7% 4.7% 4.2% 4.0% 3.7% 4.0% 4.2% 3.7% 3.8% 4.2% 4.7% 4.7% 5.0% 5.3% 5.6% 5.4% 5.3% 5.5% 5.5% 5.0% 4.8% 4.4%
COR 15.2% -2.0% 6.8% 4.3% 2.7% 2.6% 2.4% 1.1% -0.4% -0.9% -1.6% -1.5% -1.4% -1.3% -0.8% -1.1% 0.0% 0.0% 1.0% 1.0% 1.8% 1.8% 1.6% 1.8% 1.6% 1.0% 0.7% 0.5%
MAE WMG 24.1% 12.1% 14.7% 11.8% 11.3% 8.3% 8.3% 7.2% 6.7% 6.0% 6.0% 6.6% 6.0% 5.6% 5.9% 5.2% 5.9% 6.2% 6.8% 7.1% 7.4% 7.6% 7.7% 7.9% 7.8% 7.4% 7.3% 7.0%
MLP 4.9% -3.2% 1.6% 0.5% 1.0% -0.8% 0.6% 0.3% -0.7% -1.6% -2.1% -1.9% -2.7% -2.5% -2.0% -2.3% -1.5% -1.5% -0.8% -0.6% -0.5% -0.3% -0.1% 0.2% 0.4% -0.1% -0.3% -0.6%
M5T 10.6% 8.8% 11.6% 7.9% 6.8% 6.7% 6.6% 6.1% 5.5% 4.9% 4.8% 5.1% 5.6% 4.9% 5.0% 5.0% 5.7% 5.8% 6.3% 6.7% 7.1% 7.0% 6.8% 6.9% 6.9% 6.2% 6.0% 5.5%
Fig. 9. Comparison of MSE provided by single models built using original data sets
Fig. 10. MSE comparison of bagging ensembles of COR, WMG, MLP, and M5T
Fig. 11. Comparison of MAPE provided by single models built using original data sets
Fig. 12. MAPE comparison of bagging ensembles of COR, WMG, MLP, and M5T
Fig. 13 Comparison of MAE provided by single models built using original data sets
Fig. 14. MAE comparison of bagging ensembles of COR, WMG, MLP, and M5T
The comparison of fuzzy models with MLP and M5T ones is illustrated in Fig. 914. In turn, Table 2 contains percentage reduction of MAPE and MAE values of respective ensembles compared to MAPE and MAE provided by the original models. It can be noticed that the MLP and M5T algorithms revealed better performance than the evolutionary fuzzy ones. The most substantial benefit provided WMG and M5T multi-models. In turn, MLP ensembles did not bring any advantage almost in all cases.
4 Conclusions and Future Work The goal of the research reported was to investigate to what extent bagging approach leads to the improvement of the accuracy machine learning regression models devoted to assist with real estate appraisals. Four algorithms implemented in KEEL, including genetic fuzzy rule learning algorithm using cooperative rules technique, WangMendel algorithm with global genetic tuning of the fuzzy partition were applied to create bagged models, and two other popular algorithms i.e. a neural network of multilayer perceptron type and a model tree M5 were used in the experiments. 112 bootstrap models and 112 bagging ensembles combining the former by means of unweighted averages were created and evaluated in respect of prediction accuracy. Three performance measures MSE, MAPE, and MAE were applied. The best performance ensured bagging ensembles encompassing 24 bootstrap models. However, in our opinion the problem of optimal selection of bootstrap replicates to a composite model is still open and should be the subject of further research. The largest percentage reduction of prediction error achieved the ensemble of tuned Wang-Mendel models and decision trees M5 for regression. There was substantial difference between COR and WMG ensembles, in that using WMG brought benefit for any number of bootstrap replicates. The MLP and M5T algorithms revealed better performance than the evolutionary fuzzy ones. However, the application of fuzzy systems to the real estate appraisal has its merits, because fuzzy models are much more interpretable than those obtained using neural networks or decision trees. Moreover we plan to extend the scope of property attributes to include features derived from cadastral map and subjective assessments provided by
appraisals in result of on-site inspections. Then the use of evolutionary fuzzy systems will be still more justified, since they deal with imprecise linguistic information.
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.
Alcalá-Fdez, J. et al: KEEL: A software tool to assess evolutionary algorithms for data mining problems, Soft Computing 13:3, pp. 307--318 (2009) Banfield, R. E., et al.: A comparison of decision tree ensemble creation techniques, IEEE Transactions on Pattern Analysis and Machine Intelligence 29:1, pp. 173--180 (2007) Breiman, L.: Bagging Predictors, Machine Learning 24:2, pp. 123--140 (1996) Büchlmann, P., Yu, B.: Analyzing bagging, Annals of Statistics, vol. 30, 927--961 (2002) Canul-Reich, J., Shoemaker, L., Hall, L.O.: Ensembles of Fuzzy Classifiers, IEEE International Conference on Fuzzy Systems, FUZZ-IEEE 2007, pp. 1--6 (2007) Casillas, J., Cordón, O., and Herrera, F.: COR: A methodology to improve ad hoc datadriven linguistic rule learning methods by inducing cooperation among rules. IEEE Trans. on System, Man and Cybernetics, Part B: Cybernetics 32:4, pp. 526--537 (2002) Chen T., Ren J.: Bagging for Gaussian Process Regression, Neurocomputing 72:7-9, pp. 1605--1610 (2009) Cordón, O., Herrera, F.: A three-stage evolutionary process for learning descriptive and approximate fuzzy logic controller knowledge bases from examples. International Journal of Approximate Reasoning 17:4, pp. 369--407 (1997). Islam M.M., et. al.: Bagging and Boosting Negatively Correlated Neural Networks, IEEE Trans. On Systems, Man, And Cybernetics, Part B: Cybernetics 38:3, pp. 771--784 (2008) Kégl, B.: Robust regression by boosting the median, in Proceedings of the 16th Conference on Computational Learning Theory, pp. 258--272 (2003) Kim, D.: Improving the fuzzy system performance by fuzzy system ensemble, Fuzzy Sets and Systems 98:1, pp. 43--56 (1998) Korytkowski, M., Rutkowski, L., Scherer R.: From Ensemble of Fuzzy Classifiers to Single Fuzzy Rule Base Classifier, ICAISC 2008, LNCS, vol. 5097, pp. 265--272. Springer, Heidelberg (2008) Król, D., Lasota, T., Trawiński, B., Trawiński, K.: Investigation of evolutionary optimization methods of TSK fuzzy model for real estate appraisal. International Journal of Hybrid Intelligent Systems 5:3, pp. 111--128 (2008) Lasota, T., Mazurkiewicz, J., Trawiński, B., Trawiński, K.: Comparison of Data Driven Models for the Validation of Residential Premises using KEEL, International Journal of Hybrid Intelligent Systems, in press (2009) Moller, F.: A scaled conjugate gradient algorithm for fast supervised learning, Neural Networks 6, pp. 525--533 (1990) Polikar R.: Ensemble learning, Scholarpedia 4:1, pp. 2776 (2009) Quinlan, J.R.: Learning with Continuous Classes, 5th Australian Joint Conference on Artificial Intelligence (AI92), Singapore, pp. 343--348 (1992) Schapire, R. E.: The strength of weak learnability, Mach. Learning 5:2, 197--227 (1990) Wang, I., Witten, I.H.: Induction of model trees for predicting continuous classes. 9th European Conference on Machine Learning, Prague, pp. 128--137 (1997) Wang, L.X., and Mendel, J.M.: Generating Fuzzy Rules by Learning from Examples, IEEE Trans. on Systems, Man and Cybernetics 22:6, pp. 1414--1427 (1992) Wolpert, D.H.: Stacked Generalization. Neural Networks 5:2, pp. 241--259 (1992) Yan, J.: Ensemble SVM Regression Based Multi-View Face Detection System, IEEE Workshop on Machine Learning for Signal Processing, pp. 163--169 (2007)
Wrocław University of Environmental and Life Sciences, Dept. of Spatial Management Ul. Norwida 25/27, 50-375 Wroclaw, Poland 2 Wrocław University of Technology, Institute of Informatics, Wybrzeże Wyspiańskiego 27, 50-370 Wrocław, Poland 3 European Centre for Soft Computing, Edificio Científico-Tecnológico, 3ª Planta, C. Gonzalo Gutiérrez Quirós S/N, 33600 Mieres, Asturias, Spain [email protected], {zbigniew.telec, bogdan.trawinski}@pwr.wroc.pl, [email protected]
Abstract. The study reported was devoted to investigate to what extent bagging approach could lead to the improvement of the accuracy machine learning regression models. Four algorithms implemented in the KEEL tool, including two evolutionary fuzzy systems, decision trees for regression, and neural network, were used in the experiments. The results showed that some bagging ensembles ensured higher prediction accuracy than single models.
Keywords: bagging ensembles, genetic fuzzy systems, regression models, real estate appraisal, KEEL
1 Introduction Ensemble learning systems combine the output of machine learning algorithms, called “weak learners”, in order to get smaller prediction errors (in regression) or lower error rates (in classification). The individual estimator must provide different patterns of generalization, thus in the training process diversity is employed. Otherwise, the ensemble would be composed of the same predictors and would provide as good accuracy as the single one. It has been proved that the ensemble performs better when each individual machine learning system is accurate and makes errors on different examples [2], [10]. Although, there are many taxonomies for that, there is one recognized group the so-called data resampling, which generates different training sets to obtain unique regressor or classificator. To this group we may include bagging [3], boosting [18], and stacking [21]. The most popular method - bagging, which stands for bootstrap aggregating, is one of the most intuitive and simplest ensemble algorithms providing a good performance. Diversity of regressors is obtained by using bootstrapped replicas of the training data. That is, different training data subsets are randomly drawn with replacement from the original training set. So obtained training data subsets, called also bags, are used then
to train different regression models. Finally, individual regressors are combined through an algebraic expression, such as minimum, maximum, sum, mean, product, median, etc.[16]. Although the ensemble learning systems are more common in classification, they are also applied to regression problems. Bagging has been employed among others to regression trees [4], Gaussian process [7], SVM [22], and neural networks [9]. You can find also some works on applying ensemble fuzzy systems to solve classification [5], [12] and prediction problems [11], but they are not too numerous. In our previous works [13], [14] we tested different machine learning algorithms, among others genetic fuzzy systems trying to select the most appropriate ones to build data driven models for real estate appraisal using MATLAB and KEEL. Since ensemble learning can be used to improve the performance of a model and reduce the likelihood of an unfortunate selection of a poor one, in this paper we focused our efforts into investigating to what extent the bagging approach could lead to the improvement of the accuracy machine learning regression models devoted to assist with real estate appraisals. The concept of a data driven models for premises valuation, presented in the paper, was developed based on the sales comparison method. The architecture of the proposed system is shown in Fig. 1. The appraiser accesses the system through the internet and input the values of the attributes of the premises being evaluated into the system, which calculates the output using a given model. The final result, that is a suggested value of the property, is sent back to the appraiser.
Fig. 1. Information systems to assist with real estate appraisals
Actual data used to generate and learn appraisal models came from the cadastral system and the registry of real estate transactions referring to residential premises sold in one of big Polish cities at market prices within two years 2001 and 2002. They constituted original data set of 1098 cases of sales/purchase transactions. Four attributes were pointed out as price drivers: usable area of premises, floor on which premises were located, year of building construction, number of storeys in the building, in turn, price of premises was the output variable.
2 Plan of Experiments The main goal of our study was to explore the performance of the bagged ensembles of evolutionary fuzzy models to be employed in an internet system assisting property valuation. For this purpose KEEL, a non-commercial Java software tool designed to assess evolutionary algorithms for data mining problems, was used [1]. Two fuzzy rule based systems for regression, namely COR algorithm and Wang-Mendel one combined with genetic tuning were applied to create bagging models. For comparison two other popular algorithms i.e. a multilayer perceptron neural network and a model tree M5 were utilized (see Table 1). Table 1. KEEL algorithms used in study Code COR WMG MLP M5T
KEEL name Regr-COR_GA
Description Genetic fuzzy rule learning using cooperative rules technique [6] Regr-Fuzzy-WM + Post-G-G- Wang-Mendel algorithm with global genetic tuning of the Tuning-FRBSs fuzzy partition [8], [20] Regr-MLPerceptronConj-Grad Multilayer perceptron for modeling [15] Regr-M5 Model tree M5 for regression [17], [19]
Schema of the experiments is depicted in Fig. 2. On the basis of the original data set 28 bootstrap replicates were created. These replicates were used to generate models employing COR, WMG, MLP, and M5T regression algorithms implemented in KEEL. Normalization of data was performed using the min-max approach. As fitness function the mean square error (MSE) was applied and 10-fold cross validation (10cv) was accomplished. As aggregation functions simple averages were used.
Fig. 2. Schema of bagging ensemble model development
The values of three common performance measures were used as the arguments of the aggregation function. They are mean square error (MSE – Formula 1), which strongly penalizes bigger errors, mean absolute percentage error (MAPE – 2), which is a relative measure and easy to interpret and compare different models, and mean absolute error (MAE – 3), which gives values in the same dimension as output variables. The values of the measures were calculated using actual and predicted prices, obtained for testing sets. These measures are expressed in the form of
following formulas below, where yi denotes actual price and 𝑦i – predicted price of ith case, and N – number of cases in the testing set. 𝑁 1 (1) 𝑀𝑆𝐸 = 𝑦𝑖 − 𝑦𝑖 2 𝑁 𝑖=1 𝑀𝐴𝑃𝐸 =
1 𝑁
𝑀𝐴𝐸 =
𝑁 𝑖=1
1 𝑁
𝑦𝑖 − 𝑦𝑖 ∗ 100% 𝑦𝑖 𝑁 𝑖=1
𝑦𝑖 − 𝑦𝑖
(2) (3)
3 Results of Experiments The performance of bagging ensembles comprising from 1 to 28 subsequently drawn bootstrap models is presented in Fig. 3 and 4 for MSE, 5 and 6 for MAPE, and finally 7 and 8 for MAE. It can be observed that, except for some first ensembles, which should be passed over in order to avoid the effect of favourable drawing, the bagging models including 24 bootstrap replicates achieved the best prediction performance. Similar considerations presented Breiman [3], who stated that 25 bootstrap replicates could be reasonable to find the ensemble with the lowest prediction error. Substantial differences between COR and WMG ensembles can be observed. Using WMG brings benefit for any number of bootstrap replicates whereas COR provides gain for ensembles composed of above 18 bags.
Fig. 3. Performance of bagged ensembles for MSE compared with original COR model
Fig. 4. Performance of bagged ensembles for MSE compared with original WMG model
Fig. 5. Performance of bagged ensembles for MAPE compared with original COR model
Fig. 6. Performance of bagged ensembles for MAPE compared with original WMG model
Fig. 7 Performance of bagged ensembles for MAE compared with original COR model
Fig. 8. Performance of bagged ensembles for MAE compared with original WMG model
Table 2. Percentage error reduction of bagged ensembles compared to single models No. of bags 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
COR 11.0% 0.6% 7.4% 5.4% 3.9% 4.2% 4.5% 3.1% 1.8% 1.6% 0.9% 0.5% 0.5% 0.3% 0.7% 0.5% 1.3% 1.3% 2.0% 1.8% 2.4% 2.5% 2.5% 2.5% 2.3% 2.0% 1.9% 1.7%
MAPE WMG 22.5% 11.7% 13.1% 10.5% 9.6% 8.0% 7.8% 6.1% 5.8% 5.1% 4.7% 5.1% 4.8% 4.4% 4.7% 4.3% 4.9% 5.0% 5.5% 5.7% 5.8% 5.9% 6.1% 6.4% 6.3% 6.1% 6.0% 5.9%
MLP 3.1% -3.0% -0.4% -0.9% -0.6% -1.8% -0.5% -1.3% -2.2% -2.9% -3.6% -3.7% -4.3% -4.0% -3.5% -3.5% -2.9% -3.0% -2.5% -2.4% -2.3% -2.2% -1.8% -1.6% -1.4% -1.8% -1.8% -2.1%
M5T 8.3% 8.5% 9.8% 7.0% 5.4% 5.7% 5.7% 4.7% 4.2% 4.0% 3.7% 4.0% 4.2% 3.7% 3.8% 4.2% 4.7% 4.7% 5.0% 5.3% 5.6% 5.4% 5.3% 5.5% 5.5% 5.0% 4.8% 4.4%
COR 15.2% -2.0% 6.8% 4.3% 2.7% 2.6% 2.4% 1.1% -0.4% -0.9% -1.6% -1.5% -1.4% -1.3% -0.8% -1.1% 0.0% 0.0% 1.0% 1.0% 1.8% 1.8% 1.6% 1.8% 1.6% 1.0% 0.7% 0.5%
MAE WMG 24.1% 12.1% 14.7% 11.8% 11.3% 8.3% 8.3% 7.2% 6.7% 6.0% 6.0% 6.6% 6.0% 5.6% 5.9% 5.2% 5.9% 6.2% 6.8% 7.1% 7.4% 7.6% 7.7% 7.9% 7.8% 7.4% 7.3% 7.0%
MLP 4.9% -3.2% 1.6% 0.5% 1.0% -0.8% 0.6% 0.3% -0.7% -1.6% -2.1% -1.9% -2.7% -2.5% -2.0% -2.3% -1.5% -1.5% -0.8% -0.6% -0.5% -0.3% -0.1% 0.2% 0.4% -0.1% -0.3% -0.6%
M5T 10.6% 8.8% 11.6% 7.9% 6.8% 6.7% 6.6% 6.1% 5.5% 4.9% 4.8% 5.1% 5.6% 4.9% 5.0% 5.0% 5.7% 5.8% 6.3% 6.7% 7.1% 7.0% 6.8% 6.9% 6.9% 6.2% 6.0% 5.5%
Fig. 9. Comparison of MSE provided by single models built using original data sets
Fig. 10. MSE comparison of bagging ensembles of COR, WMG, MLP, and M5T
Fig. 11. Comparison of MAPE provided by single models built using original data sets
Fig. 12. MAPE comparison of bagging ensembles of COR, WMG, MLP, and M5T
Fig. 13 Comparison of MAE provided by single models built using original data sets
Fig. 14. MAE comparison of bagging ensembles of COR, WMG, MLP, and M5T
The comparison of fuzzy models with MLP and M5T ones is illustrated in Fig. 914. In turn, Table 2 contains percentage reduction of MAPE and MAE values of respective ensembles compared to MAPE and MAE provided by the original models. It can be noticed that the MLP and M5T algorithms revealed better performance than the evolutionary fuzzy ones. The most substantial benefit provided WMG and M5T multi-models. In turn, MLP ensembles did not bring any advantage almost in all cases.
4 Conclusions and Future Work The goal of the research reported was to investigate to what extent bagging approach leads to the improvement of the accuracy machine learning regression models devoted to assist with real estate appraisals. Four algorithms implemented in KEEL, including genetic fuzzy rule learning algorithm using cooperative rules technique, WangMendel algorithm with global genetic tuning of the fuzzy partition were applied to create bagged models, and two other popular algorithms i.e. a neural network of multilayer perceptron type and a model tree M5 were used in the experiments. 112 bootstrap models and 112 bagging ensembles combining the former by means of unweighted averages were created and evaluated in respect of prediction accuracy. Three performance measures MSE, MAPE, and MAE were applied. The best performance ensured bagging ensembles encompassing 24 bootstrap models. However, in our opinion the problem of optimal selection of bootstrap replicates to a composite model is still open and should be the subject of further research. The largest percentage reduction of prediction error achieved the ensemble of tuned Wang-Mendel models and decision trees M5 for regression. There was substantial difference between COR and WMG ensembles, in that using WMG brought benefit for any number of bootstrap replicates. The MLP and M5T algorithms revealed better performance than the evolutionary fuzzy ones. However, the application of fuzzy systems to the real estate appraisal has its merits, because fuzzy models are much more interpretable than those obtained using neural networks or decision trees. Moreover we plan to extend the scope of property attributes to include features derived from cadastral map and subjective assessments provided by
appraisals in result of on-site inspections. Then the use of evolutionary fuzzy systems will be still more justified, since they deal with imprecise linguistic information.
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.
Alcalá-Fdez, J. et al: KEEL: A software tool to assess evolutionary algorithms for data mining problems, Soft Computing 13:3, pp. 307--318 (2009) Banfield, R. E., et al.: A comparison of decision tree ensemble creation techniques, IEEE Transactions on Pattern Analysis and Machine Intelligence 29:1, pp. 173--180 (2007) Breiman, L.: Bagging Predictors, Machine Learning 24:2, pp. 123--140 (1996) Büchlmann, P., Yu, B.: Analyzing bagging, Annals of Statistics, vol. 30, 927--961 (2002) Canul-Reich, J., Shoemaker, L., Hall, L.O.: Ensembles of Fuzzy Classifiers, IEEE International Conference on Fuzzy Systems, FUZZ-IEEE 2007, pp. 1--6 (2007) Casillas, J., Cordón, O., and Herrera, F.: COR: A methodology to improve ad hoc datadriven linguistic rule learning methods by inducing cooperation among rules. IEEE Trans. on System, Man and Cybernetics, Part B: Cybernetics 32:4, pp. 526--537 (2002) Chen T., Ren J.: Bagging for Gaussian Process Regression, Neurocomputing 72:7-9, pp. 1605--1610 (2009) Cordón, O., Herrera, F.: A three-stage evolutionary process for learning descriptive and approximate fuzzy logic controller knowledge bases from examples. International Journal of Approximate Reasoning 17:4, pp. 369--407 (1997). Islam M.M., et. al.: Bagging and Boosting Negatively Correlated Neural Networks, IEEE Trans. On Systems, Man, And Cybernetics, Part B: Cybernetics 38:3, pp. 771--784 (2008) Kégl, B.: Robust regression by boosting the median, in Proceedings of the 16th Conference on Computational Learning Theory, pp. 258--272 (2003) Kim, D.: Improving the fuzzy system performance by fuzzy system ensemble, Fuzzy Sets and Systems 98:1, pp. 43--56 (1998) Korytkowski, M., Rutkowski, L., Scherer R.: From Ensemble of Fuzzy Classifiers to Single Fuzzy Rule Base Classifier, ICAISC 2008, LNCS, vol. 5097, pp. 265--272. Springer, Heidelberg (2008) Król, D., Lasota, T., Trawiński, B., Trawiński, K.: Investigation of evolutionary optimization methods of TSK fuzzy model for real estate appraisal. International Journal of Hybrid Intelligent Systems 5:3, pp. 111--128 (2008) Lasota, T., Mazurkiewicz, J., Trawiński, B., Trawiński, K.: Comparison of Data Driven Models for the Validation of Residential Premises using KEEL, International Journal of Hybrid Intelligent Systems, in press (2009) Moller, F.: A scaled conjugate gradient algorithm for fast supervised learning, Neural Networks 6, pp. 525--533 (1990) Polikar R.: Ensemble learning, Scholarpedia 4:1, pp. 2776 (2009) Quinlan, J.R.: Learning with Continuous Classes, 5th Australian Joint Conference on Artificial Intelligence (AI92), Singapore, pp. 343--348 (1992) Schapire, R. E.: The strength of weak learnability, Mach. Learning 5:2, 197--227 (1990) Wang, I., Witten, I.H.: Induction of model trees for predicting continuous classes. 9th European Conference on Machine Learning, Prague, pp. 128--137 (1997) Wang, L.X., and Mendel, J.M.: Generating Fuzzy Rules by Learning from Examples, IEEE Trans. on Systems, Man and Cybernetics 22:6, pp. 1414--1427 (1992) Wolpert, D.H.: Stacked Generalization. Neural Networks 5:2, pp. 241--259 (1992) Yan, J.: Ensemble SVM Regression Based Multi-View Face Detection System, IEEE Workshop on Machine Learning for Signal Processing, pp. 163--169 (2007)
