Chapter 52
Is a Model’s Scatter Really “Very Small”
or Is Model A Really “Performing
Better” Than Model B?
Steven Hanna and Joseph Chang
Abstract Many papers are published in which a dispersion model’s predictions are
compared with field observations and/or with other models’ predictions. Standard
model performance measures are used such as Fractional Bias (FB). Many times,
subjective statements are made such as “The model has very small scatter” or “Model
A is performing better than Model B”. About 30 years ago, we developed the BOOT
model evaluation software, which has two main components: 1. Calculation of model
performance measures such as FB; and 2. Calculation of confidence limits (e.g.,
95%) on performance measures and on the difference in a performance measure
between two models. Bootstrap or Jackknife resampling methods are employed. We
briefly review the methodology in BOOT’s Component 2, which is seldom used by
researchers. We present an example from a project where several urban puff models’
predictions are compared with JU2003 field data, and where assessments are carried
out regarding whether, for example, it can be concluded, with 95% confidence, that
the difference in FB for two models is not significantly different from zero.
52.1 Introduction
Dispersion models are often used in decision-making regarding pollutant impacts on
the public and on the environment. Among other considerations, the decision-maker
is interested in the uncertainties in the dispersion model outputs, and whether one
model is better than another. The purpose of this paper is to describe a quantitative
method to assess these uncertainties and determine whether a performance measure
or a difference in performance measures between models are statistically significant.
S. Hanna (B)
Hanna Consultants, 7 Crescent Ave, Kennebunkport, ME 04046-7235, USA
e-mail: hannaconsult@roadrunner.com
J. Chang
RAND Corporation, Arlington, VA 22202-5050, USA
e-mail: jchang@rand.org
© Springer Nature Switzerland AG 2020
C. Mensink et al. (eds.), Air Pollution Modeling and its Application XXVI,
Springer Proceedings in Complexity,
https://doi.org/10.1007/978-3-030-22055-6_52
329
Is a Model’s Scatter Really “Very Small”
or Is Model A Really “Performing
Better” Than Model B?
Steven Hanna and Joseph Chang
Abstract Many papers are published in which a dispersion model’s predictions are
compared with field observations and/or with other models’ predictions. Standard
model performance measures are used such as Fractional Bias (FB). Many times,
subjective statements are made such as “The model has very small scatter” or “Model
A is performing better than Model B”. About 30 years ago, we developed the BOOT
model evaluation software, which has two main components: 1. Calculation of model
performance measures such as FB; and 2. Calculation of confidence limits (e.g.,
95%) on performance measures and on the difference in a performance measure
between two models. Bootstrap or Jackknife resampling methods are employed. We
briefly review the methodology in BOOT’s Component 2, which is seldom used by
researchers. We present an example from a project where several urban puff models’
predictions are compared with JU2003 field data, and where assessments are carried
out regarding whether, for example, it can be concluded, with 95% confidence, that
the difference in FB for two models is not significantly different from zero.
52.1 Introduction
Dispersion models are often used in decision-making regarding pollutant impacts on
the public and on the environment. Among other considerations, the decision-maker
is interested in the uncertainties in the dispersion model outputs, and whether one
model is better than another. The purpose of this paper is to describe a quantitative
method to assess these uncertainties and determine whether a performance measure
or a difference in performance measures between models are statistically significant.
S. Hanna (B)
Hanna Consultants, 7 Crescent Ave, Kennebunkport, ME 04046-7235, USA
e-mail: hannaconsult@roadrunner.com
J. Chang
RAND Corporation, Arlington, VA 22202-5050, USA
e-mail: jchang@rand.org
© Springer Nature Switzerland AG 2020
C. Mensink et al. (eds.), Air Pollution Modeling and its Application XXVI,
Springer Proceedings in Complexity,
https://doi.org/10.1007/978-3-030-22055-6_52
329