Part I: Evaluating Danish Aid to Uganda

Chapter 1. Evaluation Objectives and Approach

Objectives

1.1 This is an evaluation of the entirety of Danish aid to Uganda over almost two decades. Included within its scope, as well as the bilateral country programme, are “personnel” support, NGO projects and the Private Sector Development programme.¹ Danish funding of multilaterals active in Uganda (e.g. UN agencies, the European Commission, the World Bank) is not included.

1.2 The full Terms of Reference (ToR) are at the end of this report. The evaluation objectives, as summarised in the ToR, are reproduced in Box 1.1.

Approach

1.3 Evaluating an entire donor programme requires the evaluators to go beyond an analysis of its component projects. The country programme has to be assessed as a coherent and dynamic whole, seen in the context of the country’s overall performance and of what other donors were doing. The exceptionally long time period in this case brings additional challenges. Evidence for the early years is more difficult to assemble, while both the programme’s objectives and general concepts of good practice are likely to have altered. Accordingly, the evaluators must guard against a bias towards the more recent years, and against using anachronistic objectives and performance standards when judging early performance.

1.4 The ToR set out a methodological framework to address these challenges. It involves:

Adherence to OECD DAC principles and evaluation criteria.

An organising framework based on four headings/levels to be addressed in sequence:

1. Context and framework conditions.
2. Combined donor efforts.
3. The contribution of Danish financed activities to Uganda’s development.
4. Implementation modalities and follow-up of Danish development assistance.

An Evaluation Matrix attached to the ToR (see end of report) follows these levels and proposes detailed questions/issues plus relevant indicators and data.

A study of stakeholder perceptions.

1.5 The OECD DAC evaluation criteria (relevance, effectiveness, efficiency, sustainability, impact) are reflected in Box 1.1. A key DAC (and therefore Danida³) principle is that evaluation should be independent. The evaluation team is responsible for all the findings and conclusions in this report. None of the team members has had significant direct involvement in the programme under evaluation.

1.6 The OECD DAC criteria do not resolve the difficulties of making an overall evaluation of a programme that consists of very disparate components. Box 1.2 notes the alternative (or complementary) additive and subtractive approaches that may be used. One reason why one cannot simply evaluate sectors separately and sum the results is that sectors are not symmetrical. Some (e.g. infrastructure) lend themselves more easily to quantitative economic analysis which would be impossible for others (e.g. governance, see Box 1.3). There would be a danger of bias towards what is simpler to evaluate. Also, this would ignore interactions and possible complementarity between dissimilar components of a donor’s aid programme. In the final, assessment, chapter we return to the question of coherence.

Box 1.2: Additive and Subtractive Approaches to Evaluating Country Programmes A 1999 conference on country evaluations noted alternative approaches: Two different broad approaches to country evaluation started to emerge from the conference. One looks at individual projects and aggregates the findings, and the other at the country programme as a whole. The former is essentially additive, using more conventional evaluation techniques to look at a representative selection of projects; the latter is basically subtractive, looking at the overall achievements of the country and attempting to explain what part of national success and failure can be attributed to the influence of a donor’s aid. ... an inductive approach which sought first to identify change and then to identify the causes (aid and non-aid) which explain the change would introduce more balance into the aid-centric picture of development which often emerges from aid evaluation. (OECD Development Assistance Committee, 1999b:21) The present evaluation draws on both approaches. The Danish aid programme is set clearly within the broader assessment of levels 1 and 2 of the Evaluation Matrix so as to derive a (subtractive) assessment of the Danish contribution. At the same time, we could not assess the Danish programme as a whole without understanding its individual components in some detail (Levels 3 and 4): that aspect of the approach is “additive” but recognises that the overall assessment has to go beyond a simple summation of components.

Box 1.3: Evaluating Governance Interventions There are several challenges to assessing whether donors in general in Uganda have contributed to building democracy, peace and respect for human rights. First, the instruments available for measuring progress in these fields are under-developed compared with our ability to measure changes in poverty. Assessing whether significant change has occurred in government attitudes and behaviour, in public opinion or in political culture is a much more difficult exercise than measuring the poverty headcount, or a given population’s health status or educational outcomes. Typically, changes in governance are assessed against milestones (for example, the holding of multi-party elections) that assert but cannot firmly establish a linkage to more open and accountable governance. The challenge of measurement is linked to the fact that – much more explicitly than is the case with technical sectors – donors are operating within and trying to influence political processes and institutions. Political and institutional change is slow, uneven, uncertain and reversible. Domestic factors and forces (including the impact of regional political and economic dynamics) always dominate, and they can and often do knock donor initiatives off track. With respect to Uganda, donors have learned to temper what now appears to have been an earlier over-estimation of the pace and direction of change. The lesson that donors have taken from this and similar experiences elsewhere is the need for their support to institutional change to be rooted in a careful analysis of political context and of the structural and institutional levers or drivers of change. Considerable progress has been made in donors’ capacity for political analysis, but our ability to link this to strategies for programming and policy dialogue is still very much a ’work in progress’. Finally, the way that donors have traditionally organised their support to governance initiatives also makes it difficult to assess whether this support is making a difference. Within donor agencies, much of the funding for governance has been dispersed between a relatively large number of short-term projects and within programmes that were not particularly coherent or strategic. This has mainly been the case with support to civil society, which has become an increasingly important component of donor governance strategies in the last few years. While it may be possible to evaluate the immediate outcomes of discrete projects, it has been much more difficult to reach a judgement on whether cumulatively they had longer-term or wider impact. Coordination between donors has also been limited, which has added to the general difficulty of assessing how far donors as a whole have been able to influence change. Source: Volume 4, Thematic Paper 7.

Box 1.4: The Perception Study Objective: The ToR envisaged a perception study in which: The “voice” of the stakeholders (Ugandan and other donor representatives) will be heard. [They] will be questioned of their perception (which by definition is subjective, but none the less important) of the relevance, modalities and performance of the Danish financed assistance. Method: The team sought a range of informants covering the different fields and time periods of Danish assistance. A total of 150 interviews were conducted with 186 informants (see box for details). Interviews were not tape-recorded. However, phrases presented in quotation marks in the perception study were noted at the time as direct speech. For accuracy, most interviews were conducted by at least two team members, notes were systematically written up, cross-checked and shared among all the team. Informant Categories Number of Informants Non-Danida   Central Government 40 NGOs and CSOs 33 Local Government 27 Agencies 17 Donors 14 Others 9 Sub-total 140     Danida   Danida programme/project officer 16 MFA Danida staff – Kampala 16 MFA staff – Copenhagen 14 Sub-total 46     Total 186 132 men, 54 women 130 Ugandans, 56 non-Ugandans The study team agreed a standard approach to conducting interviews and writing up interview notes. These notes were an extremely valuable resource for all components of the evaluation. The perception study report is distilled from over 135,000 words of interview notes. (The core of the report –Chapter 3 – is under 10,000 words; the more detailed annexes, about 25,000 words.) Annex 3 of the perception study gives more detail on the methodology and informants. Structure of the Perception Study Report The report systematically juxtaposes the views of Danida informants with those of other stakeholders. Recorded perceptions are organised by theme:* What kind of donor is Danida perceived to be? What kind of organisation is Danida perceived to be? How is the Uganda programme perceived to have developed? Key thematic and sector perceptions. Annexes provide more detail on perceptions of the main components of the Danish aid programme. Uses and Limitations of the Perception Study Any qualitative perception study has certain limitations: As the ToR acknowledge, the perceptions recorded are subjective. The interviewees were not a scientifically selected sample. (Inevitably, perceptions of the early years are fewer and less certain than for the more recent period. Some areas of Danida focus are covered in considerable detail; for others there is only a smattering of observations.) Many interviewees have biases by virtue of their involvement in the Danish aid programme. The perception study indicates the affiliations and perspectives of the informants quoted, but without disclosing individual identities. The process of editing and analysis inevitably reflects the judgements and biases of the study team as to what is relevant and interesting. However, the evaluation team’s own judgements and opinions are provided in the other volumes of the report. The perception study is an attempt to let the voices of stakeholders come through. The process of documenting and analysing stakeholder views was laborious. However, the study team found it a very useful way of collecting and sharing evidence for the evaluation. Interview evidence is not self-sufficient, but it is a vital complement to the documentary records available. * Informants rarely used the vocabulary of the DAC evaluation criteria and it would have been misleading to report their views under the DAC evaluation headings.

1.7 The perception study report (see Box 1.4) is presented separately as Volume 2 of the evaluation. It was compiled as an integral part of the study, drawing on all the interviews undertaken by the evaluation team. The report systematically juxtaposes the perceptions of “Danida” and “non-Danida” stakeholders. Its evidence is inherently subjective, and most interviewees have a direct interest in Danish aid. Nevertheless, it provides interesting insights, and has been a valuable complement to the data and documentary research undertaken by the evaluation team. At certain points in this report we refer to the Perception Study as a key source of evidence; but even where it is not explicitly referred to, we have drawn on the information and insights from the interviews in arriving at our judgments.

1.8 The evaluation has followed the logic of the evaluation methodology both in its work plan (Appendix B describes the study process) and in the structure of this report. Following this introduction, Part II addresses levels 1 and 2 of the evaluation framework (Uganda’s overall performance and the contribution of combined donor efforts). Part III then analyses the Danish contribution.

1.9 The chapter structure is as follows:

Part II Uganda’s Recovery and Aid in the Museveni Era
Chapter 2: Political Context and Developments
Chapter 3: Aid Relationships after 1986
Chapter 4: Economic Performance and Poverty Reduction
Chapter 5: The Aggregate Contribution of Aid

Part III: The Danish Contribution
Chapter 6: Danish Uganda Aid Strategy
Chapter 7: Danish Uganda Aid Programme (Overview)
Chapter 8: Danish Uganda Aid Programme (Main Components)
Chapter 9: Cross-cutting Issues and Aid Modalities
Chapter 10: Assessment and Lessons Learned.

1) See Chapter 7 for details.

2) MFA Danida, 1999b.

3) See MFA Danida, 1999b.