boosted learning within UNEP during the evaluation process. However, the length
of the evaluation process, due in part to the high quality standards applied by the
Evaluation Office and the time required for receiving stakeholder comments on all
evaluation products, created an important time lag between the data collection
phase and the distribution of the draft main report. This had two consequences:
information presented in the draft main report was more than 1 year old, and
internal stakeholder interest for the main report, when it was finally shared within
UNEP, appeared to be a lot less than it had been for the intermediate evaluation
products.
The evaluation team decided to cover the cross-cutting Science and Outreach
component as part of the three other components and not separately, as an acceptable way of dealing with the human resource and time constraints within the team.
This was fine in principle, but as a result, some high visibility assessment products
developed jointly by different units in UNEP under this component were not
included in the project sample, and received therefore an only cursory treatment
in the report. This undervalued some important UNEP-wide efforts and was also a
missed opportunity in terms of learning lessons from cross-divisional collaboration.
While it might not have been necessary to give the assessment of the Science and
Outreach component the same level of depth as was given to the others, one or two
projects from this component should have been included in the project sample.
As acknowledged in the evaluation report
7 under the section presenting the
limitations of the evaluation, the size of the sample of the country case studies
(six in total – or only one for most regions) was too small. Despite the logical and
practical country selection criteria, this sample could not provide a representative
and credible picture of UNEP’s strategic relevance and performance at the country
level. A larger sample size would, however, not have been possible within budget.
An alternative approach could have been to base the country case studies on
information collected from a country questionnaire sent over email, more
in-depth desk review and interviews via Skype or video-link. A rough cost comparison with the actual approach suggests that about four additional country case
studies could have been prepared using this alternative approach, bringing the
sample to a more representative two case studies per region.
As also noted in the evaluation report, the evaluation would have benefited from
more interviews with global partners and key informants outside UNEP with a good
understanding of the global climate change arena. These would have increased
diversity and credibility of views expressed in the evaluation and, possibly, generated more strategic recommendations. With hindsight, though some interesting
views from partners were collected, the perception survey was not the most
7 UNEP Evaluation Office 2014/2015, Evaluation of the UNEP Sub-programme on Climate
Change: Final report. Web link: http://www.unep.org/eou/Portals/52/SPE%20Climate%
20Change.pdf
6 An Analytical Framework for Evaluating a Diverse Climate Change Portfolio
109
of the evaluation process, due in part to the high quality standards applied by the
Evaluation Office and the time required for receiving stakeholder comments on all
evaluation products, created an important time lag between the data collection
phase and the distribution of the draft main report. This had two consequences:
information presented in the draft main report was more than 1 year old, and
internal stakeholder interest for the main report, when it was finally shared within
UNEP, appeared to be a lot less than it had been for the intermediate evaluation
products.
The evaluation team decided to cover the cross-cutting Science and Outreach
component as part of the three other components and not separately, as an acceptable way of dealing with the human resource and time constraints within the team.
This was fine in principle, but as a result, some high visibility assessment products
developed jointly by different units in UNEP under this component were not
included in the project sample, and received therefore an only cursory treatment
in the report. This undervalued some important UNEP-wide efforts and was also a
missed opportunity in terms of learning lessons from cross-divisional collaboration.
While it might not have been necessary to give the assessment of the Science and
Outreach component the same level of depth as was given to the others, one or two
projects from this component should have been included in the project sample.
As acknowledged in the evaluation report
7 under the section presenting the
limitations of the evaluation, the size of the sample of the country case studies
(six in total – or only one for most regions) was too small. Despite the logical and
practical country selection criteria, this sample could not provide a representative
and credible picture of UNEP’s strategic relevance and performance at the country
level. A larger sample size would, however, not have been possible within budget.
An alternative approach could have been to base the country case studies on
information collected from a country questionnaire sent over email, more
in-depth desk review and interviews via Skype or video-link. A rough cost comparison with the actual approach suggests that about four additional country case
studies could have been prepared using this alternative approach, bringing the
sample to a more representative two case studies per region.
As also noted in the evaluation report, the evaluation would have benefited from
more interviews with global partners and key informants outside UNEP with a good
understanding of the global climate change arena. These would have increased
diversity and credibility of views expressed in the evaluation and, possibly, generated more strategic recommendations. With hindsight, though some interesting
views from partners were collected, the perception survey was not the most
7 UNEP Evaluation Office 2014/2015, Evaluation of the UNEP Sub-programme on Climate
Change: Final report. Web link: http://www.unep.org/eou/Portals/52/SPE%20Climate%
20Change.pdf
6 An Analytical Framework for Evaluating a Diverse Climate Change Portfolio
109