14.2 Sampling Design and Accuracy Measures
To determine an unbiased accuracy measure of a map, independent reference data
is collected and compared with the mapped data. Before collecting the reference
data, however, a sampling design needs to be defined, which answers the questions
of how, what and where the reference data is to be collected. The comparison of
map data with this reference data is then used for calculating discrete and continuous mapping accuracies.
14.2.1 Sampling Design
A carefully developed sampling design, in terms of the number, location and
distribution of samples among mapped thematic classes or continuous values, is
needed for collecting reference data, as it will determine the validity, statistical
power and overall cost of the accuracy assessment (Stehman and Czaplewski
1998; Congalton and Green 1999). The sampling design is described by sample
unit size, number of samples, spatial distribution of samples, and data collection
process at each sample unit (Stehman and Czaplewski 1998; Congalton and Green
1999). An additional consideration is the number of validation samples per
mapped class in relation to the area covered by each class (Green et al. 2000).
Sampling units, which can be points or areas (e.g., pixels or cluster of pixels),
define the spatial extent of the reference data used to calibrate and validate a map
product and its map classes (Stehman and Czaplewski 1998). In terms of the
number of samples required for each map class or continuous map value to produce
a statistically valid analysis, a balance is required between what is statistically
sound in terms of probability sampling, and what is practically achievable
(Congalton and Green 1999). Ideally, the number of samples can be determined
based on the expected accuracy, the allowable error, and the expected confidence
level (Fitzpatrick-Lins 1981). Though the most rigorous in design, this idealized
approach of probability sampling is not commonly used, as it requires large sample
numbers and high spatial sample distribution, which typically cannot be sampled
with available resources, particularly given the logistical difficulties of fieldwork in
coral reef environments (Stehman 2001). These logistical challenges include:
limited resources (e.g., boats, skilled people and equipment); access to the sample
areas (e.g., water depth, water clarity, currents, tides, dangerous marine animals);
and/or extent and remoteness of sampling area. As a compromise, Congalton (1991)
suggests a ‘rule of thumb’ of a minimum of 50 validation samples per discrete
mapping category. This study further advised the minimum sample number should
increase when the study area is larger than 4,000 km
2 or when more than 12
categories are mapped. This approach has been adopted as the default sampling
design in the majority of satellite image-based land-cover mapping applications
published to date (Foody 2002; Jensen 2005; Lillesand et al. 2008) and is
378
C. M. Roelfsema and S. R. Phinn
To determine an unbiased accuracy measure of a map, independent reference data
is collected and compared with the mapped data. Before collecting the reference
data, however, a sampling design needs to be defined, which answers the questions
of how, what and where the reference data is to be collected. The comparison of
map data with this reference data is then used for calculating discrete and continuous mapping accuracies.
14.2.1 Sampling Design
A carefully developed sampling design, in terms of the number, location and
distribution of samples among mapped thematic classes or continuous values, is
needed for collecting reference data, as it will determine the validity, statistical
power and overall cost of the accuracy assessment (Stehman and Czaplewski
1998; Congalton and Green 1999). The sampling design is described by sample
unit size, number of samples, spatial distribution of samples, and data collection
process at each sample unit (Stehman and Czaplewski 1998; Congalton and Green
1999). An additional consideration is the number of validation samples per
mapped class in relation to the area covered by each class (Green et al. 2000).
Sampling units, which can be points or areas (e.g., pixels or cluster of pixels),
define the spatial extent of the reference data used to calibrate and validate a map
product and its map classes (Stehman and Czaplewski 1998). In terms of the
number of samples required for each map class or continuous map value to produce
a statistically valid analysis, a balance is required between what is statistically
sound in terms of probability sampling, and what is practically achievable
(Congalton and Green 1999). Ideally, the number of samples can be determined
based on the expected accuracy, the allowable error, and the expected confidence
level (Fitzpatrick-Lins 1981). Though the most rigorous in design, this idealized
approach of probability sampling is not commonly used, as it requires large sample
numbers and high spatial sample distribution, which typically cannot be sampled
with available resources, particularly given the logistical difficulties of fieldwork in
coral reef environments (Stehman 2001). These logistical challenges include:
limited resources (e.g., boats, skilled people and equipment); access to the sample
areas (e.g., water depth, water clarity, currents, tides, dangerous marine animals);
and/or extent and remoteness of sampling area. As a compromise, Congalton (1991)
suggests a ‘rule of thumb’ of a minimum of 50 validation samples per discrete
mapping category. This study further advised the minimum sample number should
increase when the study area is larger than 4,000 km
2 or when more than 12
categories are mapped. This approach has been adopted as the default sampling
design in the majority of satellite image-based land-cover mapping applications
published to date (Foody 2002; Jensen 2005; Lillesand et al. 2008) and is
378
C. M. Roelfsema and S. R. Phinn