CHAPTER 3 DANIDA’S EVALUATION PROCESS

Evaluation programme

Danida’s evaluations are planned on the basis of a rolling two-year programme that is updated annually by the Evaluation Department. All relevant departments and representations are consulted and invited to propose evaluations to be included in the programme.

The Evaluation Department ensures that the programme reflects the distribution of Danish development assistance in countries, sectors, assistance instruments and priority areas.

The evaluation programme is posted on the Evaluation Department’s website http://www.evaluation.dk.

The main steps in Danida’s evaluation process, from the initial decision through to dissemination, use and follow up are described in the following sections.

Launching an evaluation

The Evaluation Department ensures that all involved parties are informed about the evaluation in due time before it is initiated. In the case of larger evaluations this is done through an approach paper that provides information on:

1. The background for the evaluation
2. The main objective
3. Outputs (reports etc. to be produced)
4. Major issues (themes or aspects to be covered)
5. Approach (methodological considerations)
6. Organisation and management
7. Preliminary work plan with a time schedule

The approach paper indicates the extent to which the partners will be involved in preparation and implementation of the evaluation, as well as possible co-operation with other donors. It is distributed for information and comments to relevant Danida departments and representations as well as partner institutions.

Where there are many interests involved, a workshop may be organised as a basis for the approach paper, and a reference group for the evaluation appointed. The composition of the reference group reflects the topic and purpose of the evaluation and is decided in consultation with relevant Danida departments.

BFT is Danida’s Technical Assistance Services ERH is Danida’s Contracts Department

Terms of Reference

The Terms of Reference provide the background for the evaluation, define its objectives and scope, the composition of the evaluation team and the timing.

The Evaluation Department drafts the Terms of Reference based on the approach paper, and ensures that the evaluation is consistent with Danida’s evaluation policy and practices.

It is essential to consider the information needs of all involved parties. To this end a workshop or seminar may be organised to bring together the Evaluation Department, the representation involved, Danida’s Technical Assistance Services (BFT), the partner country (the line ministry), the project’s management (in single intervention evaluations) and other development agencies (in sector programme evaluations). Since the partner should be involved as a matter of priority, such workshops may be organised in the partner country.

The Terms of Reference include the following chapters:

1. Background
2. Objective
3. Output
4. Scope of work
5. Methodology
6. Work plan
7. Composition of evaluation team
8. Documents available

Evaluations are usually carried out in several phases such as desk study, fieldwork, and synthesis. In such cases overall Terms of Reference provide the framework for the evaluation and one of the objectives of the desk study is to formulate detailed fieldwork plan for the evaluation.

Assigning the evaluation team

Danida, with the participation of the partner country, establishes the evaluation team by contracting with individual consultants or, as is typically the case, with consulting organisations. Competence in the field and experience relevant to the task are the key selection criteria.

All Danida evaluations are tendered according to EU procurement directives and invitations to bid are disseminated internationally, i.e. beyond the EU. Tenders are appraised on the basis of topical content, professional composition and competence of the evaluation team, and price. This tender procedure requires a preparation period that may take three to six months more than the process for evaluations below that threshold.

In cases of partner-led joint evaluations, the lead partner’s procurement policies and processes may apply.

The success of an evaluation depends on the composition of the evaluation team and the competence and personal abilities of the team members. This applies in particular to the team leader who should be the one concerned with the overall perspective, able to organise and co-ordinate the work of the team members, assess the quality and relevance of their contributions and act as a spokesperson for the team.

Members of the evaluation team are selected to represent relevant professional areas and to include professional expertise from the programme country whenever possible.

To safeguard impartiality, members of the evaluation team may not have been personally involved in the activities to be evaluated; as well, organisations conducting evaluations may not have been involved in the preparation or implementation of those activities.

Overseeing the evaluation

The Evaluation Department is responsible for managing the contractual relationship with the selected evaluation team to make sure that the evaluation is consistent with its Terms of Reference Danida’s evaluation policy, good evaluation practise and that it is carried out in cost-effective and timely fashion.

As well, the Evaluation Department is responsible for ensuring that the evaluation process and products meet the DAC Evaluation Quality Standards.⁸ 

The DAC Standards are intended to:

- Provide standards for the process (conduct) and products (outputs) of evaluations;

- Facilitate the comparison of evaluations across countries (metaevaluation);

- Facilitate partnerships and collaboration on joint evaluations;

- Better enable member countries to make use of each others’ evaluation findings and reports (including good practice and lessons learned); and streamline evaluation efforts.

The Evaluation Department may set up a reference group for each evaluation that includes representatives from the main users of the evaluation and serves to provide guidance and feedback to the external evaluation team. In the case of joint evaluations the structure and mandate of the reference group may differ.

Furthermore, to strengthen quality assurance, the Evaluation Department may use external peer reviewers.

Once the evaluation team has been assigned, the Evaluation Department monitors the conduct of the evaluation in each of the following phases:

Inception
During this first phase the evaluation team works on the basis of the Terms of Reference with the Evaluation Department to develop a detailed plan for the field study phase.

The evaluation team discusses any methodological adjustments with the Evaluation Department, whose role it is to consider significant proposed changes to what was described in the Terms of Reference and reach agreement with the evaluation team on the course of action.

At the end of this first phase, the evaluation team produces an inception report, i.e. a detailed operational plan for the conduct of the evaluation fieldwork that is submitted for approval to the Evaluation Department.

The inception report is reviewed by the Evaluation Department and the reference group and, where appropriate, consultations take place with other key partners and stakeholders before the Evaluation Department signs off on the inception report.

Fieldwork
This second phase encompasses typically data collection, analysis, discussion of findings, conclusions and recommendations through to the drafting of the final report.

The role of the Evaluation Department is to facilitate the gathering of information and general conduct of the fieldwork by the evaluation team, while monitoring that the evaluation plan is being followed and changes discussed and agreed to with the team.

As well, the Evaluation Department is available to provide guidance and assistance to the evaluation team in dealing with management, logistic and methodological issues as they arise. It acts to facilitate coordination of the evaluation with the country partner, Danish representation in the country, collaborating organisations, users and stakeholders.

Together with the reference group, the Evaluation Department considers and discusses with the team the evaluation’s findings and the preliminary conclusions and recommendations before these are included in the draft final report.

It makes sure that all relevant users and stakeholders are included in the draft discussion process as appropriate, while supporting the evaluation team’s independence.

Use of evaluation

The users of evaluations are Danida, partner country authorities, stakeholders, the media, politicians, the public, and external resources (researchers, consultants, professional agencies, etc.).

Making information available is essential to general awareness, interest and support for development assistance. The evaluation report provides a status of strengths and weaknesses of development interventions and suggests solutions to major problems. To the extent that it does so, evaluation is a piece of technical assistance in itself which is appreciated as a test of the viability of development activities and the bilateral co-operation between the partner authorities and Danida.

After the final report has been received, Danida decides what actions to be taken in the light of conclusions and recommendations presented. Salient issues are brought to the attention of Danida’s senior level management. This may eventually be used to further develop and improve overall policies and methods of work. It is the task of the Evaluation

Department to systematise such experience and formulate practical proposals on that basis.

Summaries of evaluation reports are distributed to all Danida staff members. It is the responsibility of Danida’s operational departments and Embassies to ensure that relevant past experience is built into the design and preparation of future activities.

The significance of an evaluation report is different for different users. It is essential that the final users of the information are able to utilise the evaluation system for their own needs. This implies that for the evaluation system to function properly, the demand for information needs to be carefully analysed. In an evaluation system designed to respond to demands rather than having a control function, the information is more likely to be utilised.

In order to ensure this linkage, the Evaluation Department has a role in assessing the needs of users and developing evaluation to respond to shifting policy and modes of development co-operation. Also, it is vital for the Department to assess continuously the quality of evaluation activities and explore mechanisms for feedback and improved learning.

The main feedback linkages are to make use of evaluation in planning of new development interventions, to manage existing activities, to develop policy and strategies, and to train staff members and external resources.

Apart from the evaluation report itself, feedback is provided through the summary of evaluation, the follow-up memorandum, specific studies and reports produced by the Evaluation Department, seminars and workshops, and guidelines issued by the Evaluation Department. Brief summaries of evaluation reports are submitted to the database of the OECD/ DAC Network on Development Evaluation, and all Danida’s evaluations are published and made public in the form of printed reports, summaries, and electronically on http://www.evaluation.dk.

Dissemination

Whether evaluation reports should be made public has been a matter of debate. Evaluation activities were originally not meant to be a means of accountability for development assistance. In the early 1980s, however, the policy was that the activities to inform the public to some extent should make use of the results of evaluations. The policy has since changed successively, and since 1989 all evaluations reports are available to the public.

The reaction has been generally positive. The availability of information has lifted the professional debate regarding Danish development assistance. Researchers, students, companies and individuals have benefited. Some of the debate in mass media has been more informed with the availability of evaluation reports.

Apart from distributing the evaluation report itself, the most common ways to disseminate evaluation information are through the evaluation summaries, annual reports, bibliographies, thematic reports, the web, seminars, press releases, and public debate.

Whatever channel is preferred, the best way to ensure dissemination of lessons learned and knowledge gained in evaluations is to improve both the content of reports and the presentation of material. A key benefit of good dissemination practises is transparency of development interventions and public insight into their value.

Code of conduct

Danida has developed a code of conduct to ensure the independence and credibility of evaluations by clarifying the responsibilities, proper conduct and mutual relations of consultants, the Evaluation Department, other Danida staff, and other parties responsible for the activities under evaluation. Other Danida staff includes both Embassy staff and Headquarters staff, while other parties responsible for the activities under evaluation include partner organisations and institutions in partner countries (e.g.line ministries, private sector actors, and civil society organisations), organisations in Denmark (e.g. Danish NGOs), and multilateral organisations and other international institutions supported by Danida.

Evaluators have cited two particularly difficult phases in the evaluation process where uncertainty about roles may threaten their independence: organisation of fieldwork and discussion of the draft report. The code, therefore, concentrates on issues relating to these two phases in the evaluation process, while also addressing a range of other potential conflicts.

Other key actors in the evaluation process are the target group of the activities under evaluation, other persons interviewed and contacted during the process, reference groups and peers. When the evaluation is undertaken as a joint evaluation involving several donors and partners, the described role of the Evaluation Department is also applicable to the role of other donors (the lead agency).

The code for consultants can be found in Chapter 5 – Evaluation team tasks.

Code for Evaluation Department staff

The Evaluation Department is the client, and in the case of joint evaluations with other partners: organises the assignment, controls the quality of the evaluation method and the quality of the report and is responsible for dissemination and for monitoring the follow-up of evaluation results. The Evaluation Department prepares approach papers and Terms of References, identifies consultants, facilitates contacts to Danida staff and other relevant resource persons, facilitates the collection of documents, ensures thorough assessment of the evaluation method, approves the inception report, assesses the quality of the draft report and approves the final report.

- The Evaluation Department is responsible for briefing the consultants at the inception stage on the operations, the expected role of all parties involved in the process, relevant documents and data sources and as often as necessary during the evaluation process. Contact between the Evaluation Department and consultants may take place at the initiative of either party.

- While consultants have chief responsibility for preparing the fieldwork, the task of the Evaluation Department is to facilitate the process and provide quality control of the final evaluation methodology and work plan. Consultants suggest sites for field visits, samples of interview persons, etc., but final decisions on methodology are subject to approval by the Evaluation Department. After the inception report has been approved, adjustments of the methodology described are subject to agreement between the Evaluation Department and the consultants.

- Evaluation Department staff participating in a scoping mission, the initial part of the fieldwork or in post-fieldwork debriefing meetings are responsible for facilitating the evaluation and assuring that the evaluation is proceeding correctly with regard to methodology, timing, etc. The participation and role of the Evaluation Department, and possible stakeholders, in meetings, interviews and field visits must be agreed with the consultant team leader. Stakeholders may include Embassy staff, Danida advisers and company advisers, partner representatives, other parties responsible for the activities under evaluation, and others.

- The Evaluation Department provides quality assurance of the report by checking the validity of evidence, the analytical rigour, the consistency of analyses, conclusions and recommendations, that requirements in Terms of Reference have been adequately dealt with, that the report is clearly structured and well-written, and that the report does not contain factual errors and inaccuracies.

- It is the responsibility of the Evaluation Department to synthesise comments received from Danida staff and other stakeholders into a set of consolidated comments pointing out which of the comments received are considered the most important, citing necessary changes to be made (comments relating to the issues above as well as further suggested changes, which the consultant may consider when preparing the final report).

- It is the responsibility of the Evaluation Department to ensure that relevant Danida staff and other stakeholders are kept informed about the progress of the evaluation. After the draft report has been presented, all communication between consultants and Danida staff and other stakeholders should go through the Evaluation Department.

- The Evaluation Department is required to react to all requests for assistance relating to situations in which consultants feel their independence threatened. Thus, during all steps of the evaluation process, it is an important task of the Evaluation Department to protect consultants against undue pressure from Danida staff and other stakeholders. If consultants encounter insufficient assistance or outright resistance during the evaluation process, including when carrying out the fieldwork, it is the responsibility of the Evaluation Department to contact the persons involved and ensure that proper cooperation is established. If the atmosphere during discussions of the draft report turns hostile, or arguments are not professionally sustained, it is the task of the Evaluation Department to intervene. In the extreme situation where comments can be interpreted as subtle or overt pressure against consultants to achieve specific conclusions, the Evaluation Department has a particular responsibility to take immediate action.

- If consultants suspect mismanagement, corruption or other illicit practices, the Evaluation Department must ensure that the information is passed on to relevant departments for appropriate action.

Code for Danida staff (Embassy and Headquarters staff )

Danida staff has a complex role in the evaluation process: they can request an evaluation, be the object of an evaluation, function as key resource persons during the evaluation, and/or be users of the results. They have a key role in identifying relevant evaluation topics and in assuring the usefulness of evaluation findings for the learning processes of Danida and its partners. They facilitate evaluations, and they contribute to quality assurance of the reports by pointing out factual errors and inaccuracies. Danida staff, thus, has the following responsibilities:

- When facilitating evaluations by providing contacts, references, information about activities and logistical support to the consultants, Danida staff is expected to respect the integrity of the evaluation team in making its own decisions about where to go and whom to see. Danida staff may certainly provide comments or background information on suggested sites for field visits and persons to be interviewed, but the final decision to visit the site rests with the evaluation team.

- Danida staff should assist with the identification of relevant documents, even if the material has not been specifically requested.

- When contributing to quality assurance of the report, Danida staff should observe the right of consultants to make conclusions and recommendations, which may not be shared by Danida. It is the responsibility of Danida staff to contribute to a fruitful exchange of information and views and to avoid any forms of communication, which could be perceived by consultants as undue pressure or even threats to produce certain conclusions.

- Reservations regarding the competence of evaluation team members, the quality of the fieldwork, the quality of analyses, etc. should be reported immediately to the Evaluation Department. If such reservations have not been expressed in due time, Danida staff should refrain from criticising the quality of the evaluation after its publication.

- After the draft report has been produced, all contact between consultants and Danida staff should go through the Evaluation Department. If consultants meet with Danida staff, the Evaluation Department should be present, and all correspondence between consultants and Danida staff should be copied to the Evaluation Department.

Code for other parties responsible for the activities under evaluation

With the increasing alignment of Danish development cooperation with partner management structures and with the emphasis put on partner ownership, actors in partner countries such as line ministries, private sector actors, and civil society organisations are key parties to the evaluation process. Other parties responsible for the activities under evaluation may include organisations in Denmark (e.g. Danish NGOs), multilateral organisations and other international institutions supported by Danida. Similar to Danida staff, these stakeholders have the complex role of being both the object of an evaluation, key resource persons during the evaluation, and/or users of the results. They have a key role in assuring the usefulness of evaluation findings for their own learning processes. They facilitate evaluations, and they provide quality assurance of the reports by pointing out factual errors and inaccuracies. Their responsibilities are as follows:

- When facilitating evaluations by providing contacts, references, information about activities and logistical support to the consultants, stakeholders are expected to respect the integrity of the evaluation team in making its own decisions about where to go and whom to see. Stakeholders may certainly provide comments or background information on suggested sites for field visits and persons to be interviewed, but the final decision to visit the site should be made by the evaluation team.

- Stakeholders should assist with the identification of relevant documents, even if the material has not been specifically requested.

- When providing quality assurance of the report, stakeholders should observe the right of consultants to make conclusions and recommendations, which may not be shared by their organisation. It is the responsibility of the stakeholders to contribute to a fruitful exchange of information and views and to avoid any forms of communication, which could be perceived by consultants as undue pressure or even threats to produce certain conclusions.

- Reservations regarding the competence of evaluation team members, the quality of the fieldwork, the quality of analyses, etc. should be reported immediately to the Evaluation Department. If such reservations have not been expressed in due time, stakeholders should refrain from criticising the quality of the evaluation after its publication.

- After the draft report has been produced, all contact between consultants and stakeholders should go through the Evaluation Department. If consultants meet with stakeholders, the Evaluation Department should be present, and all correspondence between consultants and stakeholders should be copied to the Evaluation Department.

8 Annex 2.