CHAPTER 2 DANIDA EVALUATIONS

Evaluating development assistance

The emergence of new types of development interventions and the focus on specific priority themes in development assistance poses new challenges to evaluators.

Some of those challenges are methodological: how to evaluate support for good governance, human rights, civil service reform and private sector support.

Some are concerned with policy: how to move from project assistance to more varied and flexible modes of assistance within sector frameworks designed to address policy and institutional problems; how to improve development co-operation and partnership; and how to evaluate the increasing volume of humanitarian relief operations.

Danida’s evaluation programme includes single intervention evaluations (although these are increasingly rare), impact evaluations, sector evaluations, country programme evaluations, evaluations of forms of development cooperation, multilateral evaluations and thematic evaluations.

As the breadth, depth and diversity of Danida’s evaluation topics and collaborations expand, including joint evaluations with other donors or partner governments, their growing complexity poses increasing methodological and logistical challenges to the oversight and conduct of evaluations.

Over time Danida has shifted the focus of its evaluations from projects to more complex modes of development assistance, e.g. sector programme, country programme or thematic evaluations, and these have proved to be increasingly cost effective and with more impact on policy decisions. 

Single intervention evaluations

 A single intervention, traditionally called a development project, is defined as a time-bound intervention designed to achieve specific objectives with specified resources, usually within a short to medium term implementation schedule, and often within a broader program.

The limited scope of the single intervention makes it relatively easy to focus the evaluation.

Although overall project success is commonly determined by whether outputs are achieved on time and on budget, single interventions in development contexts are evaluated usually from the broader perspective of attainment of objectives and impact, in order to learn from the experience and to improve the design of similar future projects.

To the extent that objectives are specifically expressed and in measurable terms, single intervention evaluations present few methodological challenges.

However, a typical complicating factor is that the objectives are often complex, confounding, unrealistic, ill defined, or change as the project develops. It follows that a single intervention is not only judged on the basis of what has formally been agreed to, rather the assessment takes into account what can be achieved realistically with the resources available.

A single intervention evaluation takes into consideration, yet distinguishes between, the assessment of the project’s outputs, i.e. the quality and quantity of what it is expected to produce, and its related, broader and more complex, effectiveness and impact questions.

Sector programme evaluations

Sector programmes are a well established mode of support for Danida in which the organisational performance of relevant institutions is a key factor for successful development.

As such, sector programme evaluations focus on questions of institutional performance, processes, changes and interrelationships, and on their effects on the sector.

Questions of organisational capacity of institutions, partner country ownership and responsibility are central to evaluating the success of sector programme support interventions and their sustainability.

A key success factor in sector programme evaluations is the involvement of the partner country and its ownership of the evaluation results.

Consistent with the emphasis on development partnerships, local ownership and good governance, donors increasingly program sector assistance jointly and assist in developing an evaluation culture by incorporating joint evaluations into sector programmes.

Ultimately the goal is for partner countries to coordinate joint evaluations in their sectors.

Country programme evaluations

Country programme evaluations focus on the entire Danish assistance to one of the programme countries.

They provide an assessment of past interventions to Danida and partner countries to improve cooperation strategies, country programmes and sector interventions. As well, they generate knowledge to improve future assistance to the country and other national country programmes.

A country programme evaluation looks at the relevance of Danish assistance against Danida’s overall policy and country strategy, as well as the development policy of the partner country. It reviews the instruments used in the bilateral cooperation, the modalities, and the relative weight given to assistance for economic and social development. 

Country programme evaluations are important for policy planning at the highest level and provide a basis for bilateral negotiations. They represent an opportunity to focus on specifi c relevant issues such as the country’s dependency on development assistance, institutional capabilities, democracy, human rights, gender, and environment in a wide context.

Because of the importance of these evaluations, involvement of the partner country in the evaluation process and its ensuing acceptance, ownership and use of the evaluative information is paramount.

Country programme evaluations are truly cross-cutting, i.e. they cover all sectors and modes of cooperation, and, as a result, focussing the evaluation poses difficult challenges. The evaluation team is usually interdisciplinary, with expertise reflecting the key issues the evaluation will focus on.

To focus the country programme evaluations within the resources available, the scope is limited typically to the development issues and the strategic choices made at the national and overall levels; the economic, political and social context of the country; and the individual development agency’s strategy or the joint assistance strategy of the involved agencies.

The evaluation will take its departure from other available sector or programme evaluations and programme documentation. However, the fieldwork will cover observations and interviews with government officials, programme staff, beneficiaries and interested parties as appropriate.

Evaluations of modes of assistance

These evaluations focus on the specific instrument or channel for development assistance, e.g. research, NGOs, balance of payment support, general budget support for poverty reduction, technical assistance. Such evaluations are usually cross-sectoral and cross-country and designed to assess the efficiency of a particular mode of assistance as a means of promoting development.

Evaluations of assistance instruments are initiated to provide accountability information on past activities and to generate knowledge extract to improve future performance. The focus is usually on the efficiency of assistance delivery, but also to some extent on the effect of assistance delivered under a given mode of cooperation.

The methodological approach varies according to the form of development assistance considered. Evaluating humanitarian assistance requires a significantly different approach from evaluating research assistance or balance of payment support.

A typical problem in evaluating assistance instruments is that the individual cases being assessed often have different objectives. This makes it difficult to compare systematically the effects of different modes of development assistance and their impact. As a result, these evaluations tend to focus more on the performance of the various parties involved in development assistance and the use of the resources made available.

Evaluation of assistance instruments has a narrow perspective yet allows lessons to be drawn from larger samples of development interventions, that in turn can have wide ranging implications for policy and to improve bilateral co-operation.

Thematic evaluations

Thematic evaluations deal with selected aspects or themes in a number of development activities. There are several such themes that have been highlighted in Danish development assistance over the years. These themes are often borne out of policy statements and often termed “crosscutting issues”.

This means that they are systematically integrated in all sectors of cooperation and in all modes of assistance. Some of the factors that have been highlighted in recent years are gender aspects, institutional development, human rights, and environmental aspects.

Over the years, such cross-cutting issues have been explored on a project-by-project basis by including specific questions in the Terms of Reference. This provides a wealth of useful information when thematic evaluations are undertaken.

Thematic evaluations cut across countries, regions and sectors and have usually the greatest complexity of the evaluations. The topic is analysed in the relation to Danida’s development policy, but also in the context of international conventions and the partner’s priorities and strategies. As in all complex evaluations, a key methodological challenge lies in distinguishing between the impact of Danish assistance and the impact of national activities and policies and of other donor initiatives. 

Another common challenge in thematic evaluations is, in the preparatory phase, to draw a sample of development activities that reflects Danish development assistance in countries, sectors and forms of assistance.

Thematic evaluations are usually based on relatively large samples of development activities implemented over a relatively long period of time. Much of the study contends with older documents on file that are incomplete or marginal to the theme; as a result, fieldwork concentrates on the more recent development activities, aims at verifying information and acquiring a developing an understanding that is meaningful in current contexts.

For all their difficulties, thematic evaluations have proved useful instruments in generating specific knowledge and recommendations at the highest level of aggregation, i.e. the policy level.

Impact evaluation

Impact evaluations do not focus on the type of intervention that is being evaluated, i.e. the evaluand, but rather on results different from outputs or outcomes. Impact evaluations assess the positive and negative changes in terms of people’s welfare, produced by a development intervention, directly or indirectly, intended or unintended.

By examining whether changes in the well-being of individuals can be attributed to a specific intervention, an impact evaluation can help understand whether or to what extent an intervention has produced the intended effect, e.g. increased incomes, decreased maternal mortality, increased literacy, etc.

An impact evaluation can also help to judge the value of an intervention in relation to its cost and provide information to decide whether it should be continued, scaled-up, altered or otherwise.