Multiple criteria for evaluating machine learning algorithms for land cover classification from satellite data

Operational monitoring of land cover from satellite data will require autom ated procedures for analyzing large volumes of data. We propose multiple cr iteria for assessing algorithms for this task. In addition to standard clas sification accuracy measures, we propose criteria to account for computatio nal resources requires by the algorithms, stability of the algorithms, and robustness to noise in the training data. We also propose that classificati on accuracy take account, through estimation of misclassification costs, of unequal consequences to the user depending on which cover types are confus ed. In this article, we apply these criteria to three variants of decision tree classifiers, a standard decision tree implemented in C5.0 and two tech niques recently proposed in the machine learning literature known as "baggi ng" and "boosting." Each of these algorithms are applied to two data sets, a global land cover classification from 8 km AVHRR data and a Landsat Thema tic Mapper scene in Peru. Results indicate comparable accuracy of the three variants of the decision tree algorithms on the two data sets, with boosti ng providing marginally higher accuracies. The bagging and boosting algorit hms, however, are both substantially more stable and more robust to noise i n the training data compared with the standard C5.0 decision tree. The bagg ing algorithm is most costly in terms of computational resources while the standard decision tree is least costly. The results illustrate that the cho ice of the most suitable algorithm requires consideration of a suite of cri teria in additions to the traditional accuracy measures and that there are likely to be trade-offs between algorithm performance and required computat ional resources. (C) Elsevier Science Inc., 2000.