192
divided into three datasets, considering (a) all species with an abundance larger than
10, (b) frequent species with an abundance larger than 30 and (c) infrequent species
with abundances between 30 and 10 individuals (Table 2). These species subsets
were all tested in combination with all parameter subsets, leading to a final number
of 24 single datasets.
For each single dataset, a classifier was produced and its accuracy was verified
using the test dataset. For accuracy assessment, confusion matrices were generated
and the following accuracy measures were derived: Overall accuracy (OA), confidence limits for OA based on cross-validation, and Cohen´s Kappa which takes
class imbalance into account (Kuhn and Johnson 2013). As a null-model for the
overall accuracy, we calculated the No-Information Rate (Kuhn and Johnson 2013),
which is simply defined as the proportion of the largest class expressed as a percentage. A one-sided test of equal proportions was then conducted to provide a p-value
for the null-model.
The relevance of the single predictors was assessed by calculating their variable
importance. Variable importance describes the relationship between each parameter
and the outcome of the classification or regression procedure. It is measured as the
loss in performance when the respective parameter is not considered. Variable
importance was measured for all parameters in the three species subsets in order to
identify consistently important predictors across all predictors considered.
Random Forests were run with 5000 trials. The parameter mtry was set to 1/3 of
the number of variables considered. The parameter mtry describes the number of
parameters that are included in each single decision tree. In addition, a repeated
cross validation was implemented using a tenfold cross validation with five repetitions to be able to achieve standard errors and confidence intervals for the overall
accuracy. Classification was performed in the free and open source software R (R
Core Team 2016) using the packages caret (Kuhn et al. 2016), randomForest (Liaw
and Wiener 2002) and e1071 (Meyer et al. 2015).
Results
Only frequent species constantly exhibited significant p-values (Table 3, Fig. 3)
meaning that OA was higher than the respective null-model. When using all species
or only infrequent species, this was not the case. The species subsets “All” and
“Infrequent” had always low Kappa and OA values except for infrequent species
with the ALL and ALL+CHM (Table 3).
The highest OA and Kappa values were obtained for the combined RGB and
CHM dataset for the frequent species, with an OA value of 0.77 and a Kappa value
of 0.63. Globally, the “ALL” model was ranked second by Kappa for frequent species. However, “ALL” is much more complex (42 parameters) than RGB+CHM (16
parameters). Thus, the simpler RGB solution can be regarded as much more informative and easier to reproduce as fewer parameters have to be derived from the
imagery (Table 3). The lower quality of the infrequent species dataset was also eviJ. Oldeland et al.
divided into three datasets, considering (a) all species with an abundance larger than
10, (b) frequent species with an abundance larger than 30 and (c) infrequent species
with abundances between 30 and 10 individuals (Table 2). These species subsets
were all tested in combination with all parameter subsets, leading to a final number
of 24 single datasets.
For each single dataset, a classifier was produced and its accuracy was verified
using the test dataset. For accuracy assessment, confusion matrices were generated
and the following accuracy measures were derived: Overall accuracy (OA), confidence limits for OA based on cross-validation, and Cohen´s Kappa which takes
class imbalance into account (Kuhn and Johnson 2013). As a null-model for the
overall accuracy, we calculated the No-Information Rate (Kuhn and Johnson 2013),
which is simply defined as the proportion of the largest class expressed as a percentage. A one-sided test of equal proportions was then conducted to provide a p-value
for the null-model.
The relevance of the single predictors was assessed by calculating their variable
importance. Variable importance describes the relationship between each parameter
and the outcome of the classification or regression procedure. It is measured as the
loss in performance when the respective parameter is not considered. Variable
importance was measured for all parameters in the three species subsets in order to
identify consistently important predictors across all predictors considered.
Random Forests were run with 5000 trials. The parameter mtry was set to 1/3 of
the number of variables considered. The parameter mtry describes the number of
parameters that are included in each single decision tree. In addition, a repeated
cross validation was implemented using a tenfold cross validation with five repetitions to be able to achieve standard errors and confidence intervals for the overall
accuracy. Classification was performed in the free and open source software R (R
Core Team 2016) using the packages caret (Kuhn et al. 2016), randomForest (Liaw
and Wiener 2002) and e1071 (Meyer et al. 2015).
Results
Only frequent species constantly exhibited significant p-values (Table 3, Fig. 3)
meaning that OA was higher than the respective null-model. When using all species
or only infrequent species, this was not the case. The species subsets “All” and
“Infrequent” had always low Kappa and OA values except for infrequent species
with the ALL and ALL+CHM (Table 3).
The highest OA and Kappa values were obtained for the combined RGB and
CHM dataset for the frequent species, with an OA value of 0.77 and a Kappa value
of 0.63. Globally, the “ALL” model was ranked second by Kappa for frequent species. However, “ALL” is much more complex (42 parameters) than RGB+CHM (16
parameters). Thus, the simpler RGB solution can be regarded as much more informative and easier to reproduce as fewer parameters have to be derived from the
imagery (Table 3). The lower quality of the infrequent species dataset was also eviJ. Oldeland et al.