important information on expected “progress toward impact” (van den Berg 2005)
and whether the conditions have been set to enable longer term impact. The last
column presents ex post evaluations 5–8 years after the project has ended. These are
almost invariable historical evaluations, using a historical approach to trace
whether the results of the project have contributed to observed changes in trends,
markets, societies, economies and the environment. These evaluations tend to
advertise themselves as “theory of change” oriented and using mixed methods
and triangulation of evidence to come to conclusions on impact. They have less
of a problem to move beyond countries to regions and the global level, not because
the evidence is stronger after 5–8 years, but because they are more flexible in
approach and are more pragmatic and adaptable in using data sources and linking
data where possible. This sounds opportunistic, but there are many scientifically
sound methods and tools that can be combined and triangulated, as amply demonstrated by Stern et al. (2012) and Garcia and Zazueta (2015).
3.6 Using Time and Scale to Identify Approaches
Another cut-through of the three-dimensional matrix of time, space and scale would
be to combine time and scale. Figure 3.2 presents this matrix. The time dimension is
of course the same as in the time-space matrix, but has been simplified a bit, for
example presenting one row for ex-post rather than two. The scale dimension
provides various perspectives of scale. From interventions focused on one causal
mechanism, such as a project focusing on changing customer behaviour on the
energy market through price setting, to multiple interventions within one project, of
for example public-private partnerships, social change movements, capacity development efforts, to a perspective on an enabling environment that through rules and
regulations, taxation, knowledge dissemination and other incentives tries to redirect
a market or change behaviour, to market change and transformation, the interventions become more complex and challenging to evaluate. At the far right I have
included climate change, and again this environmental scale overlaps with others,
posing a special problem that two evaluends need to be recognized in an evaluation
that includes environmental objectives (see Rowe 2012).
Again we see randomized controlled trials and quasi-experimental approaches
focusing mostly on one intervention, as to control for combinations of interventions
will become very costly. Ex ante research will deliver counterfactual assessments of
how different scenarios will perform at all scales. A relatively new method such as
Qualitative Comparative Analysis (QCA) is currently often used for case studies of
more complex interventions and the enabling environment. Markets are of course
the subject of economic research and for evaluations especially market research to
assess whether a new product or approach has a chance on the market dominates in
the market columns and ideally before the new intervention starts. At the
programme and policy levels, ex post impact evaluations may look at triangulation
46
R.D. van den Berg
and whether the conditions have been set to enable longer term impact. The last
column presents ex post evaluations 5–8 years after the project has ended. These are
almost invariable historical evaluations, using a historical approach to trace
whether the results of the project have contributed to observed changes in trends,
markets, societies, economies and the environment. These evaluations tend to
advertise themselves as “theory of change” oriented and using mixed methods
and triangulation of evidence to come to conclusions on impact. They have less
of a problem to move beyond countries to regions and the global level, not because
the evidence is stronger after 5–8 years, but because they are more flexible in
approach and are more pragmatic and adaptable in using data sources and linking
data where possible. This sounds opportunistic, but there are many scientifically
sound methods and tools that can be combined and triangulated, as amply demonstrated by Stern et al. (2012) and Garcia and Zazueta (2015).
3.6 Using Time and Scale to Identify Approaches
Another cut-through of the three-dimensional matrix of time, space and scale would
be to combine time and scale. Figure 3.2 presents this matrix. The time dimension is
of course the same as in the time-space matrix, but has been simplified a bit, for
example presenting one row for ex-post rather than two. The scale dimension
provides various perspectives of scale. From interventions focused on one causal
mechanism, such as a project focusing on changing customer behaviour on the
energy market through price setting, to multiple interventions within one project, of
for example public-private partnerships, social change movements, capacity development efforts, to a perspective on an enabling environment that through rules and
regulations, taxation, knowledge dissemination and other incentives tries to redirect
a market or change behaviour, to market change and transformation, the interventions become more complex and challenging to evaluate. At the far right I have
included climate change, and again this environmental scale overlaps with others,
posing a special problem that two evaluends need to be recognized in an evaluation
that includes environmental objectives (see Rowe 2012).
Again we see randomized controlled trials and quasi-experimental approaches
focusing mostly on one intervention, as to control for combinations of interventions
will become very costly. Ex ante research will deliver counterfactual assessments of
how different scenarios will perform at all scales. A relatively new method such as
Qualitative Comparative Analysis (QCA) is currently often used for case studies of
more complex interventions and the enabling environment. Markets are of course
the subject of economic research and for evaluations especially market research to
assess whether a new product or approach has a chance on the market dominates in
the market columns and ideally before the new intervention starts. At the
programme and policy levels, ex post impact evaluations may look at triangulation
46
R.D. van den Berg