3.1 Re-instating an Older Impact Tradition?
The debate on what constitutes impact in evaluation continues, with many in the
evidence based movement
1 focusing on “rigorous” experiments to measure and
identify what works and what doesn’t, versus participatory and democratic
approaches enabling beneficiaries to state what would be relevant for them. It is
important to note that both approaches, and many others, tend to focus on the here
and now: what is relevant now, what works now and what doesn’t. However, there
is another tradition in impact evaluation which is often overlooked or ignored,
which is the historical approach. Every once in a while a historical evaluation is
done (Jerve et al. 1999), and every once in a while somebody asks attention for this
approach (van den Berg 2005), but it cannot be said to have been a strong tradition,
nor a tradition that made a big impression. Complaints have been that historical
evaluation studies are very expensive, are perhaps more research than evaluation,
take a lot of time and are not impressive as regards learning, because lessons from
years ago may not be relevant to the present circumstances, let alone the future (see
for example the controversies surrounding the Dutch historical evaluations of long
term relationships with several countries in van Beurden and Gewald 2004,
pp. 63–67). So it is with some enthusiasm that the development community turned
to experimental impact evaluation, preferably integrated into the design of projects
and executed during their lifetime, and hoped that this would turn up relevant
evidence of what works that would provide lessons for the immediate future.
However, what if the evidence of what works and what doesn’t only reveals itself
over time? What if the time horizon is in decades? What else are we to do but
integrate historical approaches with other tools and methods?
Many problems in development are longer term in nature: to reduce absolute
poverty, to reduce child-birth related death rates, to improve nutritional status, to
integrate countries into the global economy, and so on – these are measured over
decades and changes tend to happen relatively slowly. The Millennium Development Goals in general addressed global trends and impacts at higher scales. At these
levels impact evidence can no longer be generated directly through experiments and
other analytical tools such as meta-analysis, statistical analysis and modelling tend
to take over. The Millennium Development Goals were monitored through statistical data. As 2014 report on the achievements of the Millennium Development
Goals states: “reliable and robust data are critical for devising appropriate policies
and interventions for the achievement of the MDGs and for holding governments
and the international community accountable” (UN 2014, p. 6). However, especially when complex programmes and policies need to be improved, evaluations
and research have to play their role, as they can provide answers to questions why a
certain trend is occurring. For this reason the 2030 agenda for sustainable
1 A movement that has its roots in evidence based medicine (see https://en.wikipedia.org/wiki/
Evidence-based_medicine) and has spread to education, international development and other
areas, where its characteristics may differ in some aspects.
38
R.D. van den Berg
The debate on what constitutes impact in evaluation continues, with many in the
evidence based movement
1 focusing on “rigorous” experiments to measure and
identify what works and what doesn’t, versus participatory and democratic
approaches enabling beneficiaries to state what would be relevant for them. It is
important to note that both approaches, and many others, tend to focus on the here
and now: what is relevant now, what works now and what doesn’t. However, there
is another tradition in impact evaluation which is often overlooked or ignored,
which is the historical approach. Every once in a while a historical evaluation is
done (Jerve et al. 1999), and every once in a while somebody asks attention for this
approach (van den Berg 2005), but it cannot be said to have been a strong tradition,
nor a tradition that made a big impression. Complaints have been that historical
evaluation studies are very expensive, are perhaps more research than evaluation,
take a lot of time and are not impressive as regards learning, because lessons from
years ago may not be relevant to the present circumstances, let alone the future (see
for example the controversies surrounding the Dutch historical evaluations of long
term relationships with several countries in van Beurden and Gewald 2004,
pp. 63–67). So it is with some enthusiasm that the development community turned
to experimental impact evaluation, preferably integrated into the design of projects
and executed during their lifetime, and hoped that this would turn up relevant
evidence of what works that would provide lessons for the immediate future.
However, what if the evidence of what works and what doesn’t only reveals itself
over time? What if the time horizon is in decades? What else are we to do but
integrate historical approaches with other tools and methods?
Many problems in development are longer term in nature: to reduce absolute
poverty, to reduce child-birth related death rates, to improve nutritional status, to
integrate countries into the global economy, and so on – these are measured over
decades and changes tend to happen relatively slowly. The Millennium Development Goals in general addressed global trends and impacts at higher scales. At these
levels impact evidence can no longer be generated directly through experiments and
other analytical tools such as meta-analysis, statistical analysis and modelling tend
to take over. The Millennium Development Goals were monitored through statistical data. As 2014 report on the achievements of the Millennium Development
Goals states: “reliable and robust data are critical for devising appropriate policies
and interventions for the achievement of the MDGs and for holding governments
and the international community accountable” (UN 2014, p. 6). However, especially when complex programmes and policies need to be improved, evaluations
and research have to play their role, as they can provide answers to questions why a
certain trend is occurring. For this reason the 2030 agenda for sustainable
1 A movement that has its roots in evidence based medicine (see https://en.wikipedia.org/wiki/
Evidence-based_medicine) and has spread to education, international development and other
areas, where its characteristics may differ in some aspects.
38
R.D. van den Berg