• Scope and focus: The challenges for the consultants in commissioning an
assessment with such a narrow scope are twofold. Firstly, consultants might
tend to expand their assessment to other OECD DAC evaluation criteria such as
relevance, efficiency or sustainability. In particular in a case where the climate
change relevant portfolio under review is predefined by the commissioner, the
consultants might refuse to accept this climate change earmarking by the
mandatory without additional re-verification and assessment.
Secondly, the focus on accountability for effectiveness as well as the
renouncement to develop recommendations also demands a clear management
of expectations toward the project managers. The intensive involvement of
responsible project managers often leads to the expectation that the scope of
the assessment can be widen individually and that a report on effectiveness also
produces recommendations. The SDC/SECO reports on effectiveness treat
learning clearly as a secondary objective and the formulation of recommendations is not part of the evaluation.
• Method: From a clear methodological point of view, the main challenge lies in
the late introduction of climate change earmarking SDC and SECO’s interventions, the fact that climate change benefits are co-benefits in most projects and
that results relevant for accountability toward the public are only achieved with a
significant time-lag. Earlier interventions implemented before the introduction
of the OECD Rio Markers in 2006 for adaptation and 2010 for mitigation (see
References) often do not have an explicit focus on climate change mitigation and
adaptation. As a consequence, they often lack clear climate change related
objectives, indicators and baselines. Nonetheless, they have potentially produced significant results in terms of climate change mitigation or adaptation
and are worth to be included in a report on effectiveness. As mentioned above,
the complexity and the resources needed to assess their effectiveness is however
much higher in comparison with newer project that have systematically integrated climate change into their results framework (with respective indicators
and targets) and consecutive monitoring and evaluation activities.
It is important to notice that the portfolio assessed for this analysis embraced
projects and initiatives that were not explicitly making reference to climate change.
Initially the projects and programmes implemented mainly during the earlier period
were neither fully geared towards nor openly declared as climate change relevant
interventions. Only over time, some of them were gradually oriented towards
climate change and declared as such. The introduction of the OECD Rio Markers
between 2006 and 2010 supported a clear earmarking of climate change relevant
projects. Finally the Bill to Parliament on ODA 0.5 % in 2010 specifically
earmarked some of its funds to tackle climate change. As a consequence, the precise
tracking of climate change relevant interventions was far more difficult for the first
half of the period 2000–2012 and many projects had to be classified ex post.
The challenge of time-lag between the implementation of a project and the
presence of measurable results at outcome and impact level is particularly relevant
for climate change. A report on effectiveness is a very challenging undertaking for a
90
M. Egger Kissling and R. Windisch
assessment with such a narrow scope are twofold. Firstly, consultants might
tend to expand their assessment to other OECD DAC evaluation criteria such as
relevance, efficiency or sustainability. In particular in a case where the climate
change relevant portfolio under review is predefined by the commissioner, the
consultants might refuse to accept this climate change earmarking by the
mandatory without additional re-verification and assessment.
Secondly, the focus on accountability for effectiveness as well as the
renouncement to develop recommendations also demands a clear management
of expectations toward the project managers. The intensive involvement of
responsible project managers often leads to the expectation that the scope of
the assessment can be widen individually and that a report on effectiveness also
produces recommendations. The SDC/SECO reports on effectiveness treat
learning clearly as a secondary objective and the formulation of recommendations is not part of the evaluation.
• Method: From a clear methodological point of view, the main challenge lies in
the late introduction of climate change earmarking SDC and SECO’s interventions, the fact that climate change benefits are co-benefits in most projects and
that results relevant for accountability toward the public are only achieved with a
significant time-lag. Earlier interventions implemented before the introduction
of the OECD Rio Markers in 2006 for adaptation and 2010 for mitigation (see
References) often do not have an explicit focus on climate change mitigation and
adaptation. As a consequence, they often lack clear climate change related
objectives, indicators and baselines. Nonetheless, they have potentially produced significant results in terms of climate change mitigation or adaptation
and are worth to be included in a report on effectiveness. As mentioned above,
the complexity and the resources needed to assess their effectiveness is however
much higher in comparison with newer project that have systematically integrated climate change into their results framework (with respective indicators
and targets) and consecutive monitoring and evaluation activities.
It is important to notice that the portfolio assessed for this analysis embraced
projects and initiatives that were not explicitly making reference to climate change.
Initially the projects and programmes implemented mainly during the earlier period
were neither fully geared towards nor openly declared as climate change relevant
interventions. Only over time, some of them were gradually oriented towards
climate change and declared as such. The introduction of the OECD Rio Markers
between 2006 and 2010 supported a clear earmarking of climate change relevant
projects. Finally the Bill to Parliament on ODA 0.5 % in 2010 specifically
earmarked some of its funds to tackle climate change. As a consequence, the precise
tracking of climate change relevant interventions was far more difficult for the first
half of the period 2000–2012 and many projects had to be classified ex post.
The challenge of time-lag between the implementation of a project and the
presence of measurable results at outcome and impact level is particularly relevant
for climate change. A report on effectiveness is a very challenging undertaking for a
90
M. Egger Kissling and R. Windisch