earnings, and desperately wants to keep this secret from his wife Sybil. But the
Spanish waiter in the hotel, Manuel, discovers what goes on. Basil asks Manuel to
deny, if Sybil would question him, that he has any knowledge of this. When Basil is
discovered by Sybil in suspicious circumstances with a lot of money, he needs to
proof that he came by this money through legal means, and he asks Manuel to vouch
for him. Manuel looks at Basil, grins, and in a proud performance exclaims: “I
know nothing”. After a few seconds he repeats, with added emphasis: “I know
nothing”, thus sealing Basil’s fate. The evidence based movement came to the
foreground and argued for randomized controlled trials and counterfactual impact
evaluations by claiming that old fashioned evaluations could be thrown in the
wastepaper basket, and that there was a serious gap in evidence that needed to be
filled. On international cooperation the evidence on what works and what doesn’t
was, to adapt Manuel’s phrase: “we know nothing”. However, an analysis of the
dimensions of time, space and scale demonstrate that randomized controlled trials
are particularly good at covering a few of them, and that in many cases evaluators
will need to explore other methods and tools to provide evidence on impact. As a
result of the narrow scope of evidence that is accepted by the evidence movement,
they will have difficulty in explaining to policymakers, boards and parliaments that
what they want to see evidence on cannot be provided through randomized controlled trials.
The three dimensional matrix of time, space and scale provides a systemic
ordering of demand for impact evidence, and inspiration for how this can be
uncovered through various evaluation techniques. It underscores the wide range
of scientific tools and approaches as discussed in the Stern report (2012). Further
analysis is needed. No doubt more scientific tools exist and can be placed in the
matrix. It could be developed as a heuristic tool to identify key evaluation questions
and approaches. It also demonstrates that impact evidence is available throughout
the cycle of projects, programmes and policies and that demand for impact evidence
can be throughout the lifetime of a project and will get to higher levels and scales
after the project has ended.
In the case of climate change mitigation, the matrix provides a better understanding why impact is visible at project level and in markets directly influenced
(and hopefully changed) by the project, but that impact at the global level is illusive,
not visible, and has not led to the desired change in trends. Especially where goals
are formulated at the highest level the matrix may be useful in providing a
systematic understanding why impact cannot (yet) be demonstrated at that level.
My suggestion is to further develop the matrix as an analytical tool to:
1. Better identify the demand for impact evidence: is it on whether a specific causal
mechanism works, or is it whether the problem that needs to be addressed is
becoming solved, or whether global, regional or national trends are moving in
the right direction, and if so, how that is linked to the intervention.
2. When the demand is identified, how would this translate to key evaluation
questions that focus on the right moment in time, at the right location and at
the appropriate scale?
3 Mainstreaming Impact Evidence in Climate Change and Sustainable Development
49
Spanish waiter in the hotel, Manuel, discovers what goes on. Basil asks Manuel to
deny, if Sybil would question him, that he has any knowledge of this. When Basil is
discovered by Sybil in suspicious circumstances with a lot of money, he needs to
proof that he came by this money through legal means, and he asks Manuel to vouch
for him. Manuel looks at Basil, grins, and in a proud performance exclaims: “I
know nothing”. After a few seconds he repeats, with added emphasis: “I know
nothing”, thus sealing Basil’s fate. The evidence based movement came to the
foreground and argued for randomized controlled trials and counterfactual impact
evaluations by claiming that old fashioned evaluations could be thrown in the
wastepaper basket, and that there was a serious gap in evidence that needed to be
filled. On international cooperation the evidence on what works and what doesn’t
was, to adapt Manuel’s phrase: “we know nothing”. However, an analysis of the
dimensions of time, space and scale demonstrate that randomized controlled trials
are particularly good at covering a few of them, and that in many cases evaluators
will need to explore other methods and tools to provide evidence on impact. As a
result of the narrow scope of evidence that is accepted by the evidence movement,
they will have difficulty in explaining to policymakers, boards and parliaments that
what they want to see evidence on cannot be provided through randomized controlled trials.
The three dimensional matrix of time, space and scale provides a systemic
ordering of demand for impact evidence, and inspiration for how this can be
uncovered through various evaluation techniques. It underscores the wide range
of scientific tools and approaches as discussed in the Stern report (2012). Further
analysis is needed. No doubt more scientific tools exist and can be placed in the
matrix. It could be developed as a heuristic tool to identify key evaluation questions
and approaches. It also demonstrates that impact evidence is available throughout
the cycle of projects, programmes and policies and that demand for impact evidence
can be throughout the lifetime of a project and will get to higher levels and scales
after the project has ended.
In the case of climate change mitigation, the matrix provides a better understanding why impact is visible at project level and in markets directly influenced
(and hopefully changed) by the project, but that impact at the global level is illusive,
not visible, and has not led to the desired change in trends. Especially where goals
are formulated at the highest level the matrix may be useful in providing a
systematic understanding why impact cannot (yet) be demonstrated at that level.
My suggestion is to further develop the matrix as an analytical tool to:
1. Better identify the demand for impact evidence: is it on whether a specific causal
mechanism works, or is it whether the problem that needs to be addressed is
becoming solved, or whether global, regional or national trends are moving in
the right direction, and if so, how that is linked to the intervention.
2. When the demand is identified, how would this translate to key evaluation
questions that focus on the right moment in time, at the right location and at
the appropriate scale?
3 Mainstreaming Impact Evidence in Climate Change and Sustainable Development
49