The time and cost of collecting calibration and validation samples is often the
limiting factor in developing comprehensive accuracy assessments (Green et al.
2000; Stehman 2001; Andréfouët et al. 2008). These costs can be reduced by:
• combining field efforts through collaboration of different monitoring agencies
(Roelfsema et al. 2009);
• applying a field sampling approach that can easily be split into independent data
sets for calibration and validation (Roelfsema and Phinn 2010; Andréfouët
2008);
• utilizing cross validation methods, where a majority subset of samples are used
for calibration and the remainder for validation, applied in an iterative fashion
such that calibration/validation is repeated using different subsets of samples
(Andréfouët et al. 2004).
Studies with insufficient resources to collect independent calibration and validation set can alternatively use all of the samples for both calibration and validation, resulting in a ‘pseudo’ accuracy assessment (Schweizer et al. 2005;
Roelfsema et al. 2006). The pseudo accuracy is higher than the normal accuracy,
since it describes how well the training samples are classified using the same
sample set. Although this approach is generally not recommended, it can provide
some measure of accuracy when no independent validation data is available
(Congalton and Green 1999).
14.3.3 Accuracy Measures
The 80 studies reviewed were categorized according to whether their main
application purpose was related to: (1) management or (2) research/science
(Table 14.5). In terms of management applications, habitat maps were created
either for reporting the extent and composition of resources (e.g., Lauer and
Aswani 2008) or changes in resource extent and composition (e.g., Palandro et al.
2008). The most common mapping applications were for assessment of resource
extent and composition (27 papers) and for assessing images for coral reef extent
and composition (16 papers). The science applications tended to focus on whether
different types of remotely sensed images and image processing techniques could
be used to effectively map a specific property of coral reefs (e.g., algal biomass;
Andréfouët et al. 2004). Science applications also focused on improving remote
sensing approaches by studying changes in mapping accuracy produced by variations in image type (Phinn et al. 2008), processing method (Purkis 2005; Benfield
et al. 2007), and inclusion of additional field data and acoustic based image data
sets (Pasqualini et al. 2000).
Of the 80 publications only 38 reported mapped area size, number of categories
mapped, map accuracy and sensor type. Table 14.5 shows that the most common
map accuracy measures reported were overall accuracy (39 papers), Kappa (15
papers) and Tau (3 papers). The user and producer accuracies were provided in 25
388
C. M. Roelfsema and S. R. Phinn
limiting factor in developing comprehensive accuracy assessments (Green et al.
2000; Stehman 2001; Andréfouët et al. 2008). These costs can be reduced by:
• combining field efforts through collaboration of different monitoring agencies
(Roelfsema et al. 2009);
• applying a field sampling approach that can easily be split into independent data
sets for calibration and validation (Roelfsema and Phinn 2010; Andréfouët
2008);
• utilizing cross validation methods, where a majority subset of samples are used
for calibration and the remainder for validation, applied in an iterative fashion
such that calibration/validation is repeated using different subsets of samples
(Andréfouët et al. 2004).
Studies with insufficient resources to collect independent calibration and validation set can alternatively use all of the samples for both calibration and validation, resulting in a ‘pseudo’ accuracy assessment (Schweizer et al. 2005;
Roelfsema et al. 2006). The pseudo accuracy is higher than the normal accuracy,
since it describes how well the training samples are classified using the same
sample set. Although this approach is generally not recommended, it can provide
some measure of accuracy when no independent validation data is available
(Congalton and Green 1999).
14.3.3 Accuracy Measures
The 80 studies reviewed were categorized according to whether their main
application purpose was related to: (1) management or (2) research/science
(Table 14.5). In terms of management applications, habitat maps were created
either for reporting the extent and composition of resources (e.g., Lauer and
Aswani 2008) or changes in resource extent and composition (e.g., Palandro et al.
2008). The most common mapping applications were for assessment of resource
extent and composition (27 papers) and for assessing images for coral reef extent
and composition (16 papers). The science applications tended to focus on whether
different types of remotely sensed images and image processing techniques could
be used to effectively map a specific property of coral reefs (e.g., algal biomass;
Andréfouët et al. 2004). Science applications also focused on improving remote
sensing approaches by studying changes in mapping accuracy produced by variations in image type (Phinn et al. 2008), processing method (Purkis 2005; Benfield
et al. 2007), and inclusion of additional field data and acoustic based image data
sets (Pasqualini et al. 2000).
Of the 80 publications only 38 reported mapped area size, number of categories
mapped, map accuracy and sensor type. Table 14.5 shows that the most common
map accuracy measures reported were overall accuracy (39 papers), Kappa (15
papers) and Tau (3 papers). The user and producer accuracies were provided in 25
388
C. M. Roelfsema and S. R. Phinn