The study material used was the evaluation reports of those CCA programmes
that: (1) have been implemented by UNDP and other United Nations agencies;
(2) have finished initial implementation cycles; and (3) have been subject to
terminal evaluations. One of the unique aspects of these identified CCA
programmes is that they represent the first evaluation results of the completed
CCA programmes within the UNDP system (as of November 2014). Out of a
total of 11, nine CCA programmes were selected based on the criterion that the
quality of the evaluation reports was rated to be moderately satisfactory or higher
by the UNDP Independent Evaluation Office.
1 The authors conducted a metaanalysis of those ex-post evaluations by closely examining and comparing the
contents of the evaluations by applying the philosophical lens of critical realism.
The nine programmes included were implemented in the following nine countries: Armenia, Egypt, Malawi, Mozambique, Namibia, the Philippines, Tanzania,
Turkey and Zimbabwe (see Table 13.1 for summary). As the table shows, within the
context of UN programming, these programmes vary in many aspects: the funding
source (such as Global Environment Facility, Millennium Development Goals
Achievement Fund, and United Nations internal resources); types of beneficiaries,
target audiences and geographic regions (ranging from local vulnerable communities to inter-ministerial mainstreaming at the government level); and implementation modalities (including UNDP stand-alone, United Nations interagency joint
programming and Delivering as One
2 ).
This paper presents the findings of the meta-analysis conducted of the nine
evaluation reports. Although the programmes evaluated vary from one another in
many aspects, what is common is the structure of the evaluation reports. Each report
consists of four major sections, each of which covers a specific evaluation criterion:
relevance, efficiency, effectiveness and sustainability.
The evaluators who conducted the nine CCA programme evaluations all utilised
the definitions of each criterion in Table 13.2, which are based on the OECD
evaluation criteria adapted by UNDP and its partners (OECD 2002).
3
1 It was done through UNDP IEO’s quality assurance exercise. It is concerned with the quality of
how evaluation report is written by checking whether the structure of evaluation reports includes
the necessary sections and a proper evaluation framework has been put in place. Thus “moderately
satisfactory” or above rated evaluation reports do not necessarily mean high quality of project
activity results themselves.
2 Although there is no unified definition of Delivering as One modality (UN 2012), it should entail
“Four Ones”, i.e. one leader, one programme, one budget and one office amongst different
agencies of the UN system. Joint Programming, is often contrasted with Joint Programmes,
where the latter implies a set of discrete but related programmes by UN agencies and the former
implies joint efforts even from the stage of planning and designing of a programme, which is also
to be implemented together.
3 The authors are aware of criticism pertaining to the rather narrow application of the criteria
internally towards interventions (for instance, relevance could include whether the intervention is
contributing to positive change and the achievement of impact; and sustainability should include
not only the continued benefits from the intervention but whether the intervention contributes to
broader sustainable development). However, as these criteria are widely used in the evaluations in
the narrow sense, this understanding is appropriate for our analysis.
13 What Do Evaluations Tell Us About Climate Change Adaptation? Meta-analysis. . .
237
that: (1) have been implemented by UNDP and other United Nations agencies;
(2) have finished initial implementation cycles; and (3) have been subject to
terminal evaluations. One of the unique aspects of these identified CCA
programmes is that they represent the first evaluation results of the completed
CCA programmes within the UNDP system (as of November 2014). Out of a
total of 11, nine CCA programmes were selected based on the criterion that the
quality of the evaluation reports was rated to be moderately satisfactory or higher
by the UNDP Independent Evaluation Office.
1 The authors conducted a metaanalysis of those ex-post evaluations by closely examining and comparing the
contents of the evaluations by applying the philosophical lens of critical realism.
The nine programmes included were implemented in the following nine countries: Armenia, Egypt, Malawi, Mozambique, Namibia, the Philippines, Tanzania,
Turkey and Zimbabwe (see Table 13.1 for summary). As the table shows, within the
context of UN programming, these programmes vary in many aspects: the funding
source (such as Global Environment Facility, Millennium Development Goals
Achievement Fund, and United Nations internal resources); types of beneficiaries,
target audiences and geographic regions (ranging from local vulnerable communities to inter-ministerial mainstreaming at the government level); and implementation modalities (including UNDP stand-alone, United Nations interagency joint
programming and Delivering as One
2 ).
This paper presents the findings of the meta-analysis conducted of the nine
evaluation reports. Although the programmes evaluated vary from one another in
many aspects, what is common is the structure of the evaluation reports. Each report
consists of four major sections, each of which covers a specific evaluation criterion:
relevance, efficiency, effectiveness and sustainability.
The evaluators who conducted the nine CCA programme evaluations all utilised
the definitions of each criterion in Table 13.2, which are based on the OECD
evaluation criteria adapted by UNDP and its partners (OECD 2002).
3
1 It was done through UNDP IEO’s quality assurance exercise. It is concerned with the quality of
how evaluation report is written by checking whether the structure of evaluation reports includes
the necessary sections and a proper evaluation framework has been put in place. Thus “moderately
satisfactory” or above rated evaluation reports do not necessarily mean high quality of project
activity results themselves.
2 Although there is no unified definition of Delivering as One modality (UN 2012), it should entail
“Four Ones”, i.e. one leader, one programme, one budget and one office amongst different
agencies of the UN system. Joint Programming, is often contrasted with Joint Programmes,
where the latter implies a set of discrete but related programmes by UN agencies and the former
implies joint efforts even from the stage of planning and designing of a programme, which is also
to be implemented together.
3 The authors are aware of criticism pertaining to the rather narrow application of the criteria
internally towards interventions (for instance, relevance could include whether the intervention is
contributing to positive change and the achievement of impact; and sustainability should include
not only the continued benefits from the intervention but whether the intervention contributes to
broader sustainable development). However, as these criteria are widely used in the evaluations in
the narrow sense, this understanding is appropriate for our analysis.
13 What Do Evaluations Tell Us About Climate Change Adaptation? Meta-analysis. . .
237