225
dents. A variant of this, called “grade-equivalent” norms (Nitko, 1983: Chapter 14),
is used in educational settings, where the different groups are successive grades
(i.e., year grades in school), and the numbers are set using the grade labels (4 for
Grade 4, 5 for Grade 5, etc.).
These norm-referenced approaches contrast with the second, criterion-referenced, approach. Again, there are several ways in which this has been done, but we
exemplify here with reference to recent work utilizing a Rasch approach (in particular, see Wilson, 2005). This approach starts with a qualitative description of the
underlying property, called a construct map, which consists of an ordered set (from
low to high) of qualitatively distinct descriptions of RCAs. These are derived using
the design features of the set of items, or the ordered categories of item responses,
or a combination of both, carefully describing them for interpretative purposes. An
example of a construct map is shown in Fig. 7.19 (Dray, Brown, Diakow, Lee, &
Wilson, 2019), where a RCA at the lowest level is described by the category at the
bottom of the construct map, with successively more advanced readers described by
successively higher categories on the construct map. In this approach, these categories then serve as descriptions of the public references s j
*
of the public reference
properties Θ[s j
*
]. The estimates of the RCA of readers and the reading difficulties of
items on a common scale can then be placed on a Wright map (see Fig. 7.20; Wright
& Masters, 1981), which allows one to describe “bands” (segments) along the
Rasch RCA scale, θ j (as described in Sect. 6.3.7) corresponding to Θ[s j
*
].
An example of empirical bands developed for this RCA construct (Dray et al.,
2019) is shown in Fig. 7.20. The Rasch RCA scale is shown on the left-hand side (in
logits going from low at the bottom to high at the top), a histogram of the reader
ability estimates is shown to the right of that, and the thresholds of item responses
are shown to the right. The “bands” are then delineated by the darker horizontal
lines (which are set by a combination of empirical information and judgment), starting with the first level of the construct map shown in Fig. 7.19 at the bottom, the
second (construct level 2) next, and the third (construct level 3) at the top—higher
levels of the construct were not captured by this test (and, looking at the histogram
of reader abilities, none were evident among this sample of readers). Then, with
additional interpretative materials such as rich textual descriptions, and examples of
items with responses, one can then associate specific locations on the RCA scale
with theory-based statements about expected performances (see Masters & Forster,
1997, for examples of these in the RCA field). In addition, the tactics mentioned
above regarding norm-referenced approaches are still available, so that one can, for
example, describe the range of performances expected in a typical fifth-grade class
(including proportions of such students). This approach also allows the application
of the Θ[s j
*
] construct references to RCA tests using other sets of items via statistical linkage.
Thus, the RCA Θ[a] under measurement can now be associated with public values θ j via the Basic Evaluation Equation using the scale based on, and interpreted
through, the public references Θ[s j
*
], corresponding to the levels in the construct
map in Fig. 7.19.
7.3 A structural model of direct measurement
dents. A variant of this, called “grade-equivalent” norms (Nitko, 1983: Chapter 14),
is used in educational settings, where the different groups are successive grades
(i.e., year grades in school), and the numbers are set using the grade labels (4 for
Grade 4, 5 for Grade 5, etc.).
These norm-referenced approaches contrast with the second, criterion-referenced, approach. Again, there are several ways in which this has been done, but we
exemplify here with reference to recent work utilizing a Rasch approach (in particular, see Wilson, 2005). This approach starts with a qualitative description of the
underlying property, called a construct map, which consists of an ordered set (from
low to high) of qualitatively distinct descriptions of RCAs. These are derived using
the design features of the set of items, or the ordered categories of item responses,
or a combination of both, carefully describing them for interpretative purposes. An
example of a construct map is shown in Fig. 7.19 (Dray, Brown, Diakow, Lee, &
Wilson, 2019), where a RCA at the lowest level is described by the category at the
bottom of the construct map, with successively more advanced readers described by
successively higher categories on the construct map. In this approach, these categories then serve as descriptions of the public references s j
*
of the public reference
properties Θ[s j
*
]. The estimates of the RCA of readers and the reading difficulties of
items on a common scale can then be placed on a Wright map (see Fig. 7.20; Wright
& Masters, 1981), which allows one to describe “bands” (segments) along the
Rasch RCA scale, θ j (as described in Sect. 6.3.7) corresponding to Θ[s j
*
].
An example of empirical bands developed for this RCA construct (Dray et al.,
2019) is shown in Fig. 7.20. The Rasch RCA scale is shown on the left-hand side (in
logits going from low at the bottom to high at the top), a histogram of the reader
ability estimates is shown to the right of that, and the thresholds of item responses
are shown to the right. The “bands” are then delineated by the darker horizontal
lines (which are set by a combination of empirical information and judgment), starting with the first level of the construct map shown in Fig. 7.19 at the bottom, the
second (construct level 2) next, and the third (construct level 3) at the top—higher
levels of the construct were not captured by this test (and, looking at the histogram
of reader abilities, none were evident among this sample of readers). Then, with
additional interpretative materials such as rich textual descriptions, and examples of
items with responses, one can then associate specific locations on the RCA scale
with theory-based statements about expected performances (see Masters & Forster,
1997, for examples of these in the RCA field). In addition, the tactics mentioned
above regarding norm-referenced approaches are still available, so that one can, for
example, describe the range of performances expected in a typical fifth-grade class
(including proportions of such students). This approach also allows the application
of the Θ[s j
*
] construct references to RCA tests using other sets of items via statistical linkage.
Thus, the RCA Θ[a] under measurement can now be associated with public values θ j via the Basic Evaluation Equation using the scale based on, and interpreted
through, the public references Θ[s j
*
], corresponding to the levels in the construct
map in Fig. 7.19.
7.3 A structural model of direct measurement