Outreach as a cross-cutting issue rather than a stand-alone component. The portfolio under review included 57 projects and programmes classified by UNEP as
belonging to the CCSP and that were either on-going or had been initiated after
1 January 2008. A little over half (32) of these projects were completed at the time
of the evaluation, 20 were on-going and the remaining 5 were inactive or had an
unknown status. Within this portfolio, there were a number of interventions known
as “umbrella projects”, which included several, independent sub-projects contributing to the same Expected Accomplishment or (set of) Programme of Work
Outputs. If all sub-projects were counted, the total evaluation portfolio comprised
about 88 interventions. Their spread over the different thematic components was as
follows: 60 % were mitigation, 23 % were adaptation, 5 % were REDD, and 9 %
science and outreach. The remaining combined both mitigation and adaptation
objectives.
6.3 Challenges to the Evaluation
A rapid assessment of the evaluability of the sub-programme during the inception
phase had brought to light several challenges the evaluation was bound to face.
First, it was expected to assess a large, highly diverse and dispersed project
portfolio, spread over four components, managed by various branches across the
organization based in different duty stations. Second, a review of strategic documents had revealed serious issues with the results framework of the
sub-programme, namely its internal logic, the results levels at which Expected
Accomplishments and Programme of Work Outputs were pitched and the changes
in results statements, indicators and targets every 2 years. Table 6.1 presents the
results framework for the mitigation component as an illustration. Third, the
assessment of strategic relevance would prove quite challenging considering the
rapidly changing political and institutional context such as new decisions immerging from UNFCCC COPs and others.
At the same time, the evaluation would need to cope with very limited evaluative
evidence. For instance, monitoring of progress at the sub-programme level was
limited to output milestones and weak outcome indicators. Project reporting was
donor-specific, incomplete and focused on activities and outputs and, over the
period covered by the evaluation, less than one quarter of the projects in the
portfolio under review had been independently evaluated due to resource limitations and a lack of pressure from senior management and Member States. In
addition, this ambitious evaluation had to be carried out with a very limited budget,
which allowed the recruitment of only three consultants for a relatively short period
of time.
These challenges were, however, not specific to the Climate Change
Sub-programme evaluation. Similar issues were encountered by previous
sub-programme evaluations, requiring the Evaluation Office to develop an
98
M. Carbon
belonging to the CCSP and that were either on-going or had been initiated after
1 January 2008. A little over half (32) of these projects were completed at the time
of the evaluation, 20 were on-going and the remaining 5 were inactive or had an
unknown status. Within this portfolio, there were a number of interventions known
as “umbrella projects”, which included several, independent sub-projects contributing to the same Expected Accomplishment or (set of) Programme of Work
Outputs. If all sub-projects were counted, the total evaluation portfolio comprised
about 88 interventions. Their spread over the different thematic components was as
follows: 60 % were mitigation, 23 % were adaptation, 5 % were REDD, and 9 %
science and outreach. The remaining combined both mitigation and adaptation
objectives.
6.3 Challenges to the Evaluation
A rapid assessment of the evaluability of the sub-programme during the inception
phase had brought to light several challenges the evaluation was bound to face.
First, it was expected to assess a large, highly diverse and dispersed project
portfolio, spread over four components, managed by various branches across the
organization based in different duty stations. Second, a review of strategic documents had revealed serious issues with the results framework of the
sub-programme, namely its internal logic, the results levels at which Expected
Accomplishments and Programme of Work Outputs were pitched and the changes
in results statements, indicators and targets every 2 years. Table 6.1 presents the
results framework for the mitigation component as an illustration. Third, the
assessment of strategic relevance would prove quite challenging considering the
rapidly changing political and institutional context such as new decisions immerging from UNFCCC COPs and others.
At the same time, the evaluation would need to cope with very limited evaluative
evidence. For instance, monitoring of progress at the sub-programme level was
limited to output milestones and weak outcome indicators. Project reporting was
donor-specific, incomplete and focused on activities and outputs and, over the
period covered by the evaluation, less than one quarter of the projects in the
portfolio under review had been independently evaluated due to resource limitations and a lack of pressure from senior management and Member States. In
addition, this ambitious evaluation had to be carried out with a very limited budget,
which allowed the recruitment of only three consultants for a relatively short period
of time.
These challenges were, however, not specific to the Climate Change
Sub-programme evaluation. Similar issues were encountered by previous
sub-programme evaluations, requiring the Evaluation Office to develop an
98
M. Carbon