documents, evaluation reports (by the UNEP Evaluation Office and UNEP partners), project design documents and progress reports etc.
The evaluation team also conducted a large number of interviews with UNEP
staff and managers at headquarters, concerned divisions and branches, in regional
offices and country offices. Country visits were organized to conduct interviews
with government officials, NGOs, development partners, and recipients of UNEP
technical and/or financial support, which enabled the evaluation team to deepen its
analysis and understanding of key internal and external factors affecting performance. The six country visits allowed the evaluation team to gauge how beneficiaries and other key stakeholder perceived programme effectiveness, sustainability
and likelihood of impact. The country visits also helped the evaluation team to
assess synergies and complementarities between UNEP climate change interventions, and also to address cross-cutting issues such as gender.
The evaluation further conducted a staff and partner perception survey. The
purpose of the survey was to collect perceptions on sub-programme relevance and
effectiveness and key factors affecting performance such as communication and
coordination between divisions, inclusiveness within UNEP in determining work
plans and budgets, human and financial resources devoted to the CCSP and its
components, engagement with partners, monitoring and reporting systems etc. The
survey was conducted online using the SurveyMonkey platform. Responses were
received from 56 UNEP staff and managers – the response rate was acceptable at
about 40 %. Only three partners responded to the survey – a response rate of less
than 15 %.
6.6 Evaluation Process
As a first deliverable, the evaluation team produced an inception report based on an
initial desk review and introductory interviews within UNEP. It included a more
detailed presentation of the evaluation background (global context, programme
framework, institutional arrangements and project portfolio); a draft Theory of
Change of the sub-programme components; and the evaluation framework
(a detailed description of the methodology and analytical tools that the evaluation
would use to answer the evaluation questions). The inception report was first
reviewed by the Evaluation Office and then shared for comments with the
Sub-programme Coordinator and the heads of functional units involved in the
sub-programme.
The data collection phase for the evaluation was expected to take place over a
relatively short timeframe from January to April 2013. However, some country
visits had to be rescheduled due to unavailability of key persons or conflicting
schedules within the evaluation team, prolonging the data collection until June
2013. The evaluation team prepared country case studies and component working
papers, which went through several rounds of comments from the Evaluation Office
(for quality assurance) and UNEP stakeholders (for fact checking). The main report
6 An Analytical Framework for Evaluating a Diverse Climate Change Portfolio
107
The evaluation team also conducted a large number of interviews with UNEP
staff and managers at headquarters, concerned divisions and branches, in regional
offices and country offices. Country visits were organized to conduct interviews
with government officials, NGOs, development partners, and recipients of UNEP
technical and/or financial support, which enabled the evaluation team to deepen its
analysis and understanding of key internal and external factors affecting performance. The six country visits allowed the evaluation team to gauge how beneficiaries and other key stakeholder perceived programme effectiveness, sustainability
and likelihood of impact. The country visits also helped the evaluation team to
assess synergies and complementarities between UNEP climate change interventions, and also to address cross-cutting issues such as gender.
The evaluation further conducted a staff and partner perception survey. The
purpose of the survey was to collect perceptions on sub-programme relevance and
effectiveness and key factors affecting performance such as communication and
coordination between divisions, inclusiveness within UNEP in determining work
plans and budgets, human and financial resources devoted to the CCSP and its
components, engagement with partners, monitoring and reporting systems etc. The
survey was conducted online using the SurveyMonkey platform. Responses were
received from 56 UNEP staff and managers – the response rate was acceptable at
about 40 %. Only three partners responded to the survey – a response rate of less
than 15 %.
6.6 Evaluation Process
As a first deliverable, the evaluation team produced an inception report based on an
initial desk review and introductory interviews within UNEP. It included a more
detailed presentation of the evaluation background (global context, programme
framework, institutional arrangements and project portfolio); a draft Theory of
Change of the sub-programme components; and the evaluation framework
(a detailed description of the methodology and analytical tools that the evaluation
would use to answer the evaluation questions). The inception report was first
reviewed by the Evaluation Office and then shared for comments with the
Sub-programme Coordinator and the heads of functional units involved in the
sub-programme.
The data collection phase for the evaluation was expected to take place over a
relatively short timeframe from January to April 2013. However, some country
visits had to be rescheduled due to unavailability of key persons or conflicting
schedules within the evaluation team, prolonging the data collection until June
2013. The evaluation team prepared country case studies and component working
papers, which went through several rounds of comments from the Evaluation Office
(for quality assurance) and UNEP stakeholders (for fact checking). The main report
6 An Analytical Framework for Evaluating a Diverse Climate Change Portfolio
107