mixed methods evaluations may see it as a waste of money to focus on one causal
mechanism only.
Potentially meta-evaluations and meta-analysis could go a long way towards
covering some of the gaps, as has been advocated by the evidence movement
through so-called systematic reviews. However, there are methodological problems
with these reviews. They tend to focus on a specific question and go through a huge
number of studies and evaluations to see whether they provide evidence on that
specific question. Many studies turn out not to have evidence for that question and
thus are not used. Another issue is that these systematic reviews tend to not accept
evidence that is gathered outside the narrow range of methods that are considered
by the evidence movement to be sufficiently rigorous.
4 More recently realist
perspectives have started to become more fashionable in meta-evaluations, which
broadens the range of evidence that is accepted. An example can be found in
Chap. 13, ‘What do evaluations tell us about climate change adaptation’ of
this book.
3.8 Conclusions
There is a famous scene in the British comedy Fawlty Towers which provides a
good metaphor of how impact evidence may be treated by a narrow interpretation of
evidence based politics. In this particular episode of Fawlty Towers the hotel
manager Basil Fawlty puts some money on a horse in the hope of substantial
Fig. 3.3 The space and scale dimensions of demand for impact evidence (Source: Author)
4 See for example http://www.whatworksgrowth.org/resources/the-scientific-maryland-scale/. See
also the discussion in https://en.wikipedia.org/wiki/Hierarchy_of_evidence.
48
R.D. van den Berg
mechanism only.
Potentially meta-evaluations and meta-analysis could go a long way towards
covering some of the gaps, as has been advocated by the evidence movement
through so-called systematic reviews. However, there are methodological problems
with these reviews. They tend to focus on a specific question and go through a huge
number of studies and evaluations to see whether they provide evidence on that
specific question. Many studies turn out not to have evidence for that question and
thus are not used. Another issue is that these systematic reviews tend to not accept
evidence that is gathered outside the narrow range of methods that are considered
by the evidence movement to be sufficiently rigorous.
4 More recently realist
perspectives have started to become more fashionable in meta-evaluations, which
broadens the range of evidence that is accepted. An example can be found in
Chap. 13, ‘What do evaluations tell us about climate change adaptation’ of
this book.
3.8 Conclusions
There is a famous scene in the British comedy Fawlty Towers which provides a
good metaphor of how impact evidence may be treated by a narrow interpretation of
evidence based politics. In this particular episode of Fawlty Towers the hotel
manager Basil Fawlty puts some money on a horse in the hope of substantial
Fig. 3.3 The space and scale dimensions of demand for impact evidence (Source: Author)
4 See for example http://www.whatworksgrowth.org/resources/the-scientific-maryland-scale/. See
also the discussion in https://en.wikipedia.org/wiki/Hierarchy_of_evidence.
48
R.D. van den Berg