228
trometer). In this case there is no feedback in the process, and therefore there are
no problems in objectivity resulting from the one-way interaction.
Second case (non-oriented, random feedback): These are measurements in which
the interaction with the measuring instrument induces a random change in the
measurand, for example due to an uncontrolled transfer of energy between the
instrument and the object under measurement: in this case a non-oriented feedback loop is present in the process; in usual conditions, the measurement trueness is not affected by such loop (in other words, no systematic errors arise from
the interaction), and some problems in objectivity may arise due to insufficient
measurement precision, revealed by large random errors.
Third case (oriented, nonrandom feedback): These are measurements in which the
interaction with the measuring instrument induces a nonrandom change in the
measurand (which is called a “loading effect” in the context of electrotechnology, for example). Whenever such a change is identified and modeled, typically
as a systematic error (or as a bias, i.e., the estimate of systematic error; JCGM,
2012: 2.18), its effects may be experimentally minimized or mathematically corrected. As in the previous case, this situation of oriented feedback generally
results in problems in objectivity. In the human sciences, this is the context in
which the so-called Hawthorne effect (Landsberger, 1958) arises, in which an
individual alters his or her behavior due to being aware of being observed. An
undesired consequence of this effect is summarized in what is sometimes called
Goodhart’s law: “Any observed statistical regularity will tend to collapse once
pressure is placed upon it for control purposes” (Goodhart, 1981).
24
A related
phenomenon in educational measurement is sometimes called “teaching to the
test”, when the awareness of educators and their students of being tested (and the
consequences thereof) alters the nature of teaching and learning. A peculiar consequence is that, at least in the short term, measurement-like activities may be
exploited as managerial tools, leading individuals to change their behaviors only
because they are informed that some measurements will be performed on them
(even if the measurements are ultimately not utilized or even performed, in which
case the feedback loop does not actually include the measuring instrument).
24 Several cases of this phenomenon are proposed by Jerzy Muller in his book so explicitly titled
The Tyranny of Metrics. An example: “In England, in an attempt to reduce wait times in emergency
wards, the Department of Health adopted a policy that penalized hospitals with wait times longer
than 4 h. The program succeeded—at least on the surface. In fact, some hospitals responded by
keeping incoming patients in queues of ambulances, beyond the doors of the hospital, until the
staff was confident that the patient could be seen within the allotted 4 h of being admitted” (2018:
p. 5).
7 Modeling measurement and its quality
trometer). In this case there is no feedback in the process, and therefore there are
no problems in objectivity resulting from the one-way interaction.
Second case (non-oriented, random feedback): These are measurements in which
the interaction with the measuring instrument induces a random change in the
measurand, for example due to an uncontrolled transfer of energy between the
instrument and the object under measurement: in this case a non-oriented feedback loop is present in the process; in usual conditions, the measurement trueness is not affected by such loop (in other words, no systematic errors arise from
the interaction), and some problems in objectivity may arise due to insufficient
measurement precision, revealed by large random errors.
Third case (oriented, nonrandom feedback): These are measurements in which the
interaction with the measuring instrument induces a nonrandom change in the
measurand (which is called a “loading effect” in the context of electrotechnology, for example). Whenever such a change is identified and modeled, typically
as a systematic error (or as a bias, i.e., the estimate of systematic error; JCGM,
2012: 2.18), its effects may be experimentally minimized or mathematically corrected. As in the previous case, this situation of oriented feedback generally
results in problems in objectivity. In the human sciences, this is the context in
which the so-called Hawthorne effect (Landsberger, 1958) arises, in which an
individual alters his or her behavior due to being aware of being observed. An
undesired consequence of this effect is summarized in what is sometimes called
Goodhart’s law: “Any observed statistical regularity will tend to collapse once
pressure is placed upon it for control purposes” (Goodhart, 1981).
24
A related
phenomenon in educational measurement is sometimes called “teaching to the
test”, when the awareness of educators and their students of being tested (and the
consequences thereof) alters the nature of teaching and learning. A peculiar consequence is that, at least in the short term, measurement-like activities may be
exploited as managerial tools, leading individuals to change their behaviors only
because they are informed that some measurements will be performed on them
(even if the measurements are ultimately not utilized or even performed, in which
case the feedback loop does not actually include the measuring instrument).
24 Several cases of this phenomenon are proposed by Jerzy Muller in his book so explicitly titled
The Tyranny of Metrics. An example: “In England, in an attempt to reduce wait times in emergency
wards, the Department of Health adopted a policy that penalized hospitals with wait times longer
than 4 h. The program succeeded—at least on the surface. In fact, some hospitals responded by
keeping incoming patients in queues of ambulances, beyond the doors of the hospital, until the
staff was confident that the patient could be seen within the allotted 4 h of being admitted” (2018:
p. 5).
7 Modeling measurement and its quality