Consequently, complete verification of a model can only be done with regard to the
consistency, or logical accuracy, of its internal structure.
Generally, there are two ways to test a model. First, one can withhold some of
the basic data that was used to set up the model to determine the model parameters,
data that represent the real world to the extent that it can be measured. Then the
model can be used to predict the used data. For example, one might develop a model
to predict the future population of the U.S. based on actual data from, say, 1900 to
1960 and then use the resulting model to predict the (known) population data from
1970 to 1990. If the “prediction” is a success, the later data can be incorporated into
the model, and prediction for the next 20 or 30 years can be made with some
reasonable degree of confidence. Another way to test the “goodness” of a model is
to predict the condition in some heretofore unmeasured arena and then proceed to
measure these variables in the field. For example, suppose that a model is being
devised to predict the location of an endangered species. The model is built and
calibrated on the known habitats and then applied to the rest of the likely geography
to qualify these places as likely locations.
If it is not possible, by definition, to verify a model by comparing its results to the
performance of the real system, how can we know that we really captured the
essentials of that system in the model? We know that our model is not unique—
there are always other ways to construct a model. The guide for selection: always
first try to choose the simplest form. We may compare the model results to reality,
not to verify but to confirm, and if we are unable to reproduce at least the trends
observed in the real system, we know something is missing or wrong and we are
forced to revise our model or check the accuracy of the data that went into
specifying the model parameters and initial conditions.
Ironically, things can also become more problematic if the model results coincide well with our observations of the real system. The problem, for example, lies in
the possibility that errors in the model cancel each other. Such misspecifications are
difficult to detect and this is why we will start in many chapters of this book with a
theory of model behavior rather than with observations of real systems. Combining
theory, observations and, indeed intuition, in a disciplined way in dynamic models
is especially important when we make use of easy-to-learn and easy-to-use software
packages. These devices allow us to develop models that can get ahead of ourselves. At each point in the model construction process is it important to be able to
justify the assumptions that are made.
Once the model is built on a theoretical base and observations, or “reasonable”
initial conditions and parameter values, we let it yield the consequences of the
forces built into the model. The choice of observations versus “reasonable” values
is basically a choice of providing a predictive or descriptive model. For a description of the role of feedback mechanisms for system development it is frequently
sufficient to concentrate on those forces and the appropriate parameter range rather
than precise numerical values.
To confirm our model results we may compare them to appropriate data. The
greater the number of instances in which model results and reality coincide under a
variety of different scenarios, the more probable it is that the model captures the
1.8 Model Confirmation
25
consistency, or logical accuracy, of its internal structure.
Generally, there are two ways to test a model. First, one can withhold some of
the basic data that was used to set up the model to determine the model parameters,
data that represent the real world to the extent that it can be measured. Then the
model can be used to predict the used data. For example, one might develop a model
to predict the future population of the U.S. based on actual data from, say, 1900 to
1960 and then use the resulting model to predict the (known) population data from
1970 to 1990. If the “prediction” is a success, the later data can be incorporated into
the model, and prediction for the next 20 or 30 years can be made with some
reasonable degree of confidence. Another way to test the “goodness” of a model is
to predict the condition in some heretofore unmeasured arena and then proceed to
measure these variables in the field. For example, suppose that a model is being
devised to predict the location of an endangered species. The model is built and
calibrated on the known habitats and then applied to the rest of the likely geography
to qualify these places as likely locations.
If it is not possible, by definition, to verify a model by comparing its results to the
performance of the real system, how can we know that we really captured the
essentials of that system in the model? We know that our model is not unique—
there are always other ways to construct a model. The guide for selection: always
first try to choose the simplest form. We may compare the model results to reality,
not to verify but to confirm, and if we are unable to reproduce at least the trends
observed in the real system, we know something is missing or wrong and we are
forced to revise our model or check the accuracy of the data that went into
specifying the model parameters and initial conditions.
Ironically, things can also become more problematic if the model results coincide well with our observations of the real system. The problem, for example, lies in
the possibility that errors in the model cancel each other. Such misspecifications are
difficult to detect and this is why we will start in many chapters of this book with a
theory of model behavior rather than with observations of real systems. Combining
theory, observations and, indeed intuition, in a disciplined way in dynamic models
is especially important when we make use of easy-to-learn and easy-to-use software
packages. These devices allow us to develop models that can get ahead of ourselves. At each point in the model construction process is it important to be able to
justify the assumptions that are made.
Once the model is built on a theoretical base and observations, or “reasonable”
initial conditions and parameter values, we let it yield the consequences of the
forces built into the model. The choice of observations versus “reasonable” values
is basically a choice of providing a predictive or descriptive model. For a description of the role of feedback mechanisms for system development it is frequently
sufficient to concentrate on those forces and the appropriate parameter range rather
than precise numerical values.
To confirm our model results we may compare them to appropriate data. The
greater the number of instances in which model results and reality coincide under a
variety of different scenarios, the more probable it is that the model captures the
1.8 Model Confirmation
25