was drafted by November 2013, but also required a series of reviews by the
Evaluation Office and subsequent revisions, so that it was shared within UNEP
for comments as late as February 2014. Considering that the period covered by the
evaluation ended on 31 December 2012, there was a time lag of more than 1 year
between much of the information collected for the evaluation and the distribution of
its first draft report. During the first half of 2014, comments were received from
UNEP staff and data from the UNEP Programme Performance Report 2012–2013
was incorporated where appropriate to make the report as up-to-date as possible.
Because the consultants’ team had been disbanded by mid-2014, finalisation of the
report was done internally in the Evaluation Office. The report was finally
published in January 2015.
6.7 Lessons Learned on the Evaluation Approach
This evaluation has shown the importance of developing an appropriate analytical
framework, well suited for the scope and complexity of the object of evaluation.
The analytical framework and evaluation approach used for the UNEP Climate
Change Sub-programme Evaluation, combining three interlinked areas of focus
(strategic relevance, sub-programme performance and factors affecting performance), five concentric units of analysis (UNEP as a whole, sub-programme,
component, country and project) and a Theory of Change approach, allowed the
evaluation team to cover the standard evaluation criteria in a comprehensive but
concise manner, remaining strategic and without drowning in the details.
The ToC approach helped making a credible assessment of UNEP’s contribution
towards impact, sustainability and up-scaling, but did not allow this contribution to
be quantified. In other words, the evaluation could not determine to what extent
higher-level changes beyond stakeholder capacity (direct outcomes), such as
changes in environmental management practices or greenhouse gas emissions,
could be attributed to UNEP’s efforts alone, and which changes might have
happened anyway. In any case, a credible attribution of impact at the
sub-programme or sub-programme component level would have been impossible
without extensive impact assessments at the country or project level, which are
currently not available in UNEP and could not have been realistically built into the
sub-programme evaluation framework.
There appears to be a trade-off between the time that is invested in quality
assurance and stakeholder involvement during the evaluation process, on the one
hand, and the up-to-dateness of information provided and sustained stakeholder
interest in the evaluation, on the other. Strong internal stakeholder involvement
during the inception and data collection and analysis phases of the evaluation
through interviews, discussions, surveys and commenting on intermediate products,
108
M. Carbon
Evaluation Office and subsequent revisions, so that it was shared within UNEP
for comments as late as February 2014. Considering that the period covered by the
evaluation ended on 31 December 2012, there was a time lag of more than 1 year
between much of the information collected for the evaluation and the distribution of
its first draft report. During the first half of 2014, comments were received from
UNEP staff and data from the UNEP Programme Performance Report 2012–2013
was incorporated where appropriate to make the report as up-to-date as possible.
Because the consultants’ team had been disbanded by mid-2014, finalisation of the
report was done internally in the Evaluation Office. The report was finally
published in January 2015.
6.7 Lessons Learned on the Evaluation Approach
This evaluation has shown the importance of developing an appropriate analytical
framework, well suited for the scope and complexity of the object of evaluation.
The analytical framework and evaluation approach used for the UNEP Climate
Change Sub-programme Evaluation, combining three interlinked areas of focus
(strategic relevance, sub-programme performance and factors affecting performance), five concentric units of analysis (UNEP as a whole, sub-programme,
component, country and project) and a Theory of Change approach, allowed the
evaluation team to cover the standard evaluation criteria in a comprehensive but
concise manner, remaining strategic and without drowning in the details.
The ToC approach helped making a credible assessment of UNEP’s contribution
towards impact, sustainability and up-scaling, but did not allow this contribution to
be quantified. In other words, the evaluation could not determine to what extent
higher-level changes beyond stakeholder capacity (direct outcomes), such as
changes in environmental management practices or greenhouse gas emissions,
could be attributed to UNEP’s efforts alone, and which changes might have
happened anyway. In any case, a credible attribution of impact at the
sub-programme or sub-programme component level would have been impossible
without extensive impact assessments at the country or project level, which are
currently not available in UNEP and could not have been realistically built into the
sub-programme evaluation framework.
There appears to be a trade-off between the time that is invested in quality
assurance and stakeholder involvement during the evaluation process, on the one
hand, and the up-to-dateness of information provided and sustained stakeholder
interest in the evaluation, on the other. Strong internal stakeholder involvement
during the inception and data collection and analysis phases of the evaluation
through interviews, discussions, surveys and commenting on intermediate products,
108
M. Carbon