The lowest unit of analysis was the individual project. This was the most
appropriate level to unveil factors affecting performance, but as the resources for
the evaluation were limited only a sample of projects could be looked at in
sufficient depth. The evaluation team prepared rapid reviews of 19 projects –
about one third of the entire portfolio. Projects were selected on the basis of four
criteria: thematic area (adaptation, mitigation or REDD), project size (based on
estimated cost), project scope (global, regional or national) and maturity.
The evaluation made use of a Theory of Change (ToC) approach to address
several evaluation questions. A ToC depicts the logical sequence of desired changes
(also called “causal pathways” or “results chains”) to which an intervention,
programme, strategy etc. is expected to contribute. It shows the causal linkages
between changes at different results levels (outputs, outcomes, intermediate states
and impact), and the actors and factors influencing those changes. Initially inspired
by guidance provided by the Global Environment Fund
6 the UNEP Evaluation
Office has been systematically using a ToC approach in project and sub-programme
evaluations since 2009.
The ToC for each component of the CCSP, and then for the CCSP as a whole,
was reconstructed based on a review of strategic documents and UNEP staff
interviews, and using best practice in determining correct results levels. Figure 6.3
presents the overall reconstructed ToC for the CCSP. The reconstructed ToC helped
identify the expected outcomes of UNEP’s work on Climate Change and the
intermediary changes between outcomes and desired impact. Thus, it allowed to
cluster outputs and define summary direct outcome statements cutting across
components, which would prove very useful to frame data collection and synthesize
findings on sub-programme effectiveness.
The reconstruction of the ToC also helped to determine the key external factors
affecting the achievement of outcomes, intermediary states and impact, namely the
drivers that UNEP could influence through awareness raising, partnerships etc., and
the assumptions that were outside UNEP’s control. As these were key determinants
of the likelihood of impact, upscaling and sustainability of the sub-programme, it
was important to identify them early on so that adequate information on their status
could be collected in the course of the data collection phase.
The reconstructed ToC was also used to assess the internal logic and coherence
of the formal results framework of the sub-programme. Therefore, the formal
results framework comprised of the Sub-programme objective, Expected Accomplishments and Programme of Work Outputs was compared with the reconstructed
ToC, and differences between the two were pointed out. For instance, in the formal
results framework the results levels at which Expected Accomplishments and
Programme of Work Outputs had been set were inconsistent between and within
6 GEF Evaluation Office 2009, Fourth overall performance study of the GEF: The ROtI Handbook:
Towards Enhancing the Impacts of Environmental Projects, Methodological Paper #2. Web link:
https://www.thegef.org/gef/sites/thegef.org/files/documents/M2_ROtI%20Handbook.pdf
104
M. Carbon
appropriate level to unveil factors affecting performance, but as the resources for
the evaluation were limited only a sample of projects could be looked at in
sufficient depth. The evaluation team prepared rapid reviews of 19 projects –
about one third of the entire portfolio. Projects were selected on the basis of four
criteria: thematic area (adaptation, mitigation or REDD), project size (based on
estimated cost), project scope (global, regional or national) and maturity.
The evaluation made use of a Theory of Change (ToC) approach to address
several evaluation questions. A ToC depicts the logical sequence of desired changes
(also called “causal pathways” or “results chains”) to which an intervention,
programme, strategy etc. is expected to contribute. It shows the causal linkages
between changes at different results levels (outputs, outcomes, intermediate states
and impact), and the actors and factors influencing those changes. Initially inspired
by guidance provided by the Global Environment Fund
6 the UNEP Evaluation
Office has been systematically using a ToC approach in project and sub-programme
evaluations since 2009.
The ToC for each component of the CCSP, and then for the CCSP as a whole,
was reconstructed based on a review of strategic documents and UNEP staff
interviews, and using best practice in determining correct results levels. Figure 6.3
presents the overall reconstructed ToC for the CCSP. The reconstructed ToC helped
identify the expected outcomes of UNEP’s work on Climate Change and the
intermediary changes between outcomes and desired impact. Thus, it allowed to
cluster outputs and define summary direct outcome statements cutting across
components, which would prove very useful to frame data collection and synthesize
findings on sub-programme effectiveness.
The reconstruction of the ToC also helped to determine the key external factors
affecting the achievement of outcomes, intermediary states and impact, namely the
drivers that UNEP could influence through awareness raising, partnerships etc., and
the assumptions that were outside UNEP’s control. As these were key determinants
of the likelihood of impact, upscaling and sustainability of the sub-programme, it
was important to identify them early on so that adequate information on their status
could be collected in the course of the data collection phase.
The reconstructed ToC was also used to assess the internal logic and coherence
of the formal results framework of the sub-programme. Therefore, the formal
results framework comprised of the Sub-programme objective, Expected Accomplishments and Programme of Work Outputs was compared with the reconstructed
ToC, and differences between the two were pointed out. For instance, in the formal
results framework the results levels at which Expected Accomplishments and
Programme of Work Outputs had been set were inconsistent between and within
6 GEF Evaluation Office 2009, Fourth overall performance study of the GEF: The ROtI Handbook:
Towards Enhancing the Impacts of Environmental Projects, Methodological Paper #2. Web link:
https://www.thegef.org/gef/sites/thegef.org/files/documents/M2_ROtI%20Handbook.pdf
104
M. Carbon