CHAPTER 1 EVALUATION DEFINITION AND PARTIES

What is evaluation?

In general terms evaluation is used in society at large to establish achievements with some accuracy. All major development agencies involved in international development assistance undertake formal evaluations of part of their development activities each year.

Definition
Danida uses the OECD Development Assistance Committee’s¹ (DAC) definition of evaluation. 

Evaluation
Danida carries out or participates in evaluations at different stages of development interventions and defines their purpose as either primarily knowledge or accountability uses, or both.

Evaluation of ongoing development activities with a primary purpose to generate information to improve the quality of the intervention typically focus on implementation issuesand and operationel activities, but may also take a wider perspective and consider effects. As they are performed usually about midway in the cycle of the intervention, these evaluations may also be referred to as mid-term evaluations by other organisations.

Other evaluations are undertaken after completion of the aid intervention to understand the factors that affected performance, to assess the sustainability of results and impacts, and to draw lessons that may inform other interventions. The terms ’formative’ or ’summative’ evaluations are also utilized to distinguish the evaluation types.

Management’s operational tools

Although there is increasing agreement in the evaluation community on the use of basic terms², differences exist and it is useful for clarity to explain the terms used in evaluation and various assessment tools related to the operational preparation and monitoring of the implementation of development interventions:

Appraisal
Some agencies use the term evaluation for prospective studies conducted prior to a project’s approval. For example the European Commission conducts ex-ante evaluations of intended programmes³.

Such studies are termed “appraisals” by Danida: “The aim of the appraisal is to provide quality assurance at an advanced stage of the preparation process of a programme or a component, but early enough for the recommendations provided by the appraisal to influence the final preparation of programme and component documents.” (Guidelines for Programme Management)⁴ Danida’s Technical Advisory Services is responsible for all major appraisals

Review
DAC defines “review” as an assessment of the performance of an intervention, periodically or on an ad hoc basis. Danida refers to the same definition in its Aid Management Guidelines. The main distinction is that a review is regarded as an internal management tool for operational monitoring of the implementation of the targets of the interventions, while the evaluation is an independent, in-depth external assessment of the objectives, implementation and results of the interventions. Major reviews are undertaken by the Technical Advisory Services, frequently as part of joint partner reviews.

Performance monitoring
The trend of public administrations worldwide to reform their inputdriven and highly procedural administrative practices in favour of resultsbased approaches to management and accountability, and the movement in the international donor community towards providing a increasingly greater proportion of funding in the form of sector or general budget support to governments of partner countries, has underlined management’s responsibility to measure and report on the achievement of agreed results. Performance measurement of set targets has become another term for the management’s monitoring function. Danida’s Quality Assurance Department undertakes regular performance reviews of the Danish Embassies.

Performance audit
The distinction between performance audit and evaluation can be thin as typically both deal with the “what” and “why” questions of programme and organisational performance.

A useful way to distinguish performance audit from evaluation generally is to highlight that performance audit is carried out primarily to provide assurance, i.e. accountability, on a direct or attest basis, with knowledgegeneration a secondary spin-off. Evaluation on the other hand is carried out usually to inform policy and decision-making as well as to produce knowledge.

Main parties to an evaluation

Successful evaluations are based on collaboration between key participants and due consideration to the interest of the different parties in the evaluation. These include the commissioning organisations, evaluators, users, and stakeholders.

Evaluations are commissioned by Danida’s Evaluation Department, alone or jointly with other donors, and with the partner country.

 Increasingly, joint evaluations are used to rationalise the evaluation process, reduce transaction costs for partner countries, improve quality of the work undertaken, and increase weight and legitimacy of the evaluation. Usually one of the collaborating agencies is responsible for commissioning the joint evaluation. (See next section on joint evaluation).

Evaluators are the organisations or individuals that carry out the evaluation, i.e. collect and analyse data, judge the value of the assistance intervention and produce the evaluation report. They are selected usually among international and Danish consulting firms and research institutions.

Stakeholders are those agencies, organisations, groups or individuals that have an interest in the development intervention or its evaluation but not necessarily a formal role in it.

Stakeholder and user participation in the evaluation process promotes con sensus and ownership in relation to development activities. Participatory approaches are capacity building processes that develop evaluative thinking and contribute to indigenous knowledge and sustainability.

Users of evaluations are stakeholders with a specific relationship to the intervention and its evaluation. They include policy makers, Danida policy and programme managers and staff, managers of assistance interventions, developing-country partners and other parties with a formal or direct role in relation to the development activities under evaluation.

To impact decision-making, evaluations address questions that are relevant to users and responsive to their information needs, i.e. evaluative information is meaningful, reliable and timely.

Inclusion of users and stakeholders in the evaluation process contributes to useful evaluations. Dialogue with stakeholders and users improves the evaluator’s understanding of, and the evaluation’s responsiveness to, their needs and priorities. The views and expertise of groups affected by the intervention and its evaluation are considered and integrated to the evaluation process whenever appropriate through mechanisms such as steering or advisory groups. 

Joint evaluation

Joint evaluations of development assistance have been conducted by various groups of donors and countries since the late 1980’s and a significant number have been undertaken since 2000. Danida has participated in joint evaluations since 1989 (see box).

What is joint evaluation?
According to the DAC Network on Development Evaluation⁵: “Joint evaluations are development evaluations conducted collaboratively by more than one agency. The focus here is not on participatory evaluation with its techniques for bringing stakeholder communities into the process, but on evaluations undertaken jointly by more than one development cooperation agency. Joint evaluations vary considerably in the number of participating agencies and in their focus, purpose and approach. Methodologies used can also differ widely, ranging from desk reviews of existing information to fieldwork in developing countries.”

Danida and joint evaluation
The changing modes of Danida’s, and the global donor community’s, development assistance increase the significance of joint evaluations. Sector programme support and national development programmes rely on collaborative, multi-donor assistance efforts with shared common objectives, joint strategies and on partner country leadership roles.

Effective evaluations of such new types of assistance call for more collaboration. Clearly, it is a waste of resources on both the donor and partner side when several donors perform individual evaluations of their support to the same sector.

Key objectives of a joint evaluation process are to ensure that the evaluation becomes an efficient learning tool, helps promote good governance, enables the partners to be fully accountable and is cost-effective. Joint evaluations can focus on vital areas and help consolidate international responses and development policy.

Types of joint evaluations
Joint evaluations, like single agency evaluations, come in different sizes and shapes: their focus spans a continuum ranging from project to policy including topics such as assistance modalities, themes, delivery mechanisms, etc. They have different purposes, cover single countries, regions or more, use a variety of approaches and methodologies, and involve developing country partners to varying degrees.

In joint evaluations including larger groups of contributors, the involvement of the majority is typically limited to a few key stages such as review and approval of the design and final product. Management and coordination is typically delegated to one or a few agencies taking a lead responsibility or to an external organisation designated to manage the study.

The following table describes the main characteristics of three broad categories of joint evaluations, based on their degree of “jointness”, i.e. the extent to which individual partners cooperate in the evaluation process.

Acknowledging the existence and typology of many complex forms of joint evaluations contributes to reducing confusion and misunderstanding when partners work together.

Benefits and challenges of joint evaluations
Joint evaluations provide a number of significant benefits to donors and recipient countries alike particularly in terms of reduced transaction costs for developing country partners and improved harmonisation of development interventions among donors; however, they also present their own set of particular challenges due primarily to the increased complexity of managing and conducting this type of evaluation.

A key advantage of joint evaluations is that they have greater credibility and broader ownership of findings among the larger development community than would be the case with single agency evaluations.

Their value lies in the comprehensiveness of findings and in the validity of lessons learned. They generally yield useful best practises, focus policy discussions among development agencies, and achieve an impetus for joint action that is out of reach of smaller evaluations. 

 On the other hand, just as co-ordination of development interventions is notoriously difficult, the same applies to evaluations. Donors prepare their evaluation programmes in response to agency needs for lesson learning and accountability, and these are geared to the planning and programming cycle of the agency. For evaluation to develop into a joint management and accountability tool, the donors need to be flexible, to adjust to the planning cycle of the partner countries and to enable them to take the lead in evaluations.

The following is a summary⁶ of the key benefits and particular challenges of joint evaluations seen from the perspective of commissioners and organisers:

Benefits

- Diversity, Legitimacy and Influence. Joint evaluations enable a diverse range of perspectives and talents on the evaluation. This can lead to high quality, transparent and credible evaluation reports, with broad ownership of findings and greater legitimacy and influence on decisionmakers.

- Broader Scope. Joint evaluations enable a broader number of evaluation questions to be addressed, given extra (jointly-shared) resources.

- Harmonisation and Alignment. Donors are increasingly prioritising partnership-based assistance modalities. Sector-wide approaches, budget support and country-led poverty reduction strategies all stress greater participation and leadership roles for partner countries. Effective evaluations of such assistance modes call for – even require – greater collaboration. Joint evaluations therefore have distinct advantages as development agencies shift their strategies away from isolated projects towards programmatic approaches. Joint evaluations also foster greater consensus among the partner agencies on upcoming development priorities and needs, thus stimulating improved coordination of future programming.

- Rationalisation and reduced transaction costs for developing countries: Joint evaluations reduce the burden and associated costs of multiple donor evaluation efforts on partner country institutions, including overlapping team visits and duplicative data collection. Joint evaluations also help to avoid conveying to partner countries too many different and sometimes conflicting evaluation messages.

- Mutual capacity development: Joint evaluations provide opportunities for agencies to learn from each other and to share their evaluation processes and techniques. They can also lead to enhanced use of local consultants, with consequent local capacity development.

- Cost-sharing: Joint evaluations are normally undertaken with shared or pooled financing, and the cost-burden is therefore divided among the various partners.

- Applicability: Joint evaluations have additional advantages and applicability in certain specific circumstances: if there are issues in the evaluation that are too sensitive or controversial for one agency alone to tackle; if the activities being evaluated have been jointly financed or implemented; when a meta-evaluation is being undertaken; and when an evaluation of the work of a multilateral organisation is being undertaken.

Challenges are often related to difficulties in harmonising differences among agencies’ objectives, operational processes and organisational cultures.

- Planning joint evaluations: A particular problem is that by the time agencies share or make public their evaluation work programmes, it may be too late in the planning cycle of any given agency to agree to undertake it jointly. Also, all agencies have limited time and resources, and getting one’s ’own’ work completed can take priority over ’joint’ efforts.

- Consensus on purpose: Agreeing the overall and political objectives for the evaluation, and ensuring that no partners are driving hidden agendas.

- Consensus on Terms of References: Agreeing comprehensive, yet manageable, Terms of Reference that accommodate the particular issues and interests of all participating agencies.

- Management: Developing joint management structures and communication processes that work effectively.

- Suitability of evaluators: Selecting evaluation teams that are mutually acceptable to all participants.

- Logistics: Coordinating schedules and travel logistics amongst the various partners.

- Evaluative process: Reaching agreement on methodologies, recommendations and reporting formats.

As well, joint evaluations create particular conditions for the external evaluators charged with carrying out a joint evaluation and some principles have been developed for organizing and managing the external evaluation team during joint evaluations.⁷

1OECD (1991), Principles for Evaluation of Development Assistance. Paris.

2 OECD (2002), Glossary of Key Terms in Evaluation and Results Based Management. Paris.

3 European Commission (2001), Guide to Ex Ante Evaluation. Bryssels.

4 Danish Ministry of Foreign Affairs. Guidelines for Programme Management. Copenhagen.

5 OECD (2006), DAC Guidance for Managing Joint Evaluations. Paris.

6 Breier, H. (2005), Joint Evaluations: Recent Experiences, Lessons Learnt and Options for the Future, DAC Evaluation Network Working Paper, OECD, Paris.

7 Freeman, T. (2004). Managing the External Evaluation Team During Joint Evaluations: Challenges and Responses. Ottawa, Canada: Goss Gilroy Inc.