CHAPTER 5 EVALUATION TEAM TASKS

This chapter explains the expectations Danida and its Evaluation Department have of the external evaluation team with regard to the content and quality of the evaluation it is responsible for.

The information in this chapter intends to provide the external evaluation team with a clear understanding of the framework of principles, standards, roles, responsibilities and practices that Danida applies both to the methodologies and the management of its evaluations.

As such, the information is presented for each of the three main phases of the evaluation process that the external consultant team is responsible for: inception, fieldwork and reporting.

Inception: Planning the evaluation

The purpose of the inception phase is for the evaluation team to prepare a detailed operational plan, i.e. the inception report, for the next phases of the evaluation: fieldwork and reporting.

Proper planning is essential to identifying those activities required to provide well-supported answers to the evaluation questions and to avoiding other unnecessary activities and related expenditures of time, effort and money.

The planning phase provides the evaluation team with the opportunity, and responsibility, to discuss methodological specificities, fieldwork activities and reporting strategy with, and where required obtain approval from, the Evaluation Department, and as well to consult with other stakeholders.

The inception report should present:

- An overall logic model of the intervention (the evaluand), depicting the linkages between resources (inputs), intervention activities (processes), intervention results (outputs or deliverables), intended outcomes (intervention objectives), overall impacts, and their relationships in terms of the criteria of relevance, efficiency, effectiveness and impact; an explanation of how the sustainability criterion is defined and operationalised.

- The methodology: design, approach, sufficiency and appropriateness of evidence, data collection strategy and methods, analytical framework and reporting outline.

- The hierarchy of evaluation questions starting from the general ones that are presented in the Terms of References through to the specific ones that will produce data and information.

- For each specific question the basis for assessment, i.e. indicator of minimum acceptable performance.

- A matrix indicating for each specific question the nature and source of evidence.

- A schedule of activities.

- A communication and consultation plan (with stakeholders).

- In the case of evaluations with complex evaluation team organisation and logistics, e.g. joint evaluations, a systematic management plan that addresses key issues of management, coordination, authorities, responsibilities, etc.

It is expected that the external evaluation team presents, applies and documents its quality control and assurance process from the beginning of its work, i.e. at the start of the inception phase.

It should address key questions of methodology, e.g. reliability and validity issues, as well as of project management, e.g. security and confidentiality of data and information.

As well, as part of its quality control and assurance process, it is expected that the evaluation team defines, documents and presents to the Evaluation Department the structure and organisation of its evaluation file, i.e. the way in which all the information relating to the evaluation is organised and kept.

The inception phase and its attendant tasks, roles and responsibilities are organised as follows:

Inception: Methodological considerations

The Terms of References of the evaluation stipulate usually, and to varying degrees, the methodology to be employed as well as the organisation of the evaluation activities, e.g. budget, schedule and travel. However, as a result of discussions with the Evaluation Department and other stakeholders, desk study, preliminary field visit, etc., and its own evaluation expertise, the evaluation team is expected to address thoroughly key issues and questions related to methodology and management, in order to propose defensible activities, changes and/or alternatives, that will improve the quality of the evaluation.

The DAC Evaluation Quality Standards require that sound methodology be used and explained in the evaluation report.

The purpose of methodology, and the basis on which its soundness is assessed, is to produce reliable data that allow for valid evaluative judgments that are useful for learning and making decisions.

While evaluation methodology is grounded firmly in traditional social science approaches, evaluation is more than the application of social science methods to study social problems.

To make judgments and facilitate decisions the evaluation team is required to engage with stakeholders and users, in order to integrate empirical evidence with standards and values to reach evaluative conclusions.

The methods used for one evaluation can differ greatly from the next, depending on what the evaluation intends to accomplish or answer and the theories and preferences that are used.

 For example, if the priority is to make sure audiences use the evaluation, a utilisation-focused approach might be employed; if the priority is answering as unequivocally as possible “what works,” a randomized trial may be chosen; if the priority is engaging stakeholders and building evaluation capacity, an empowerment or participatory approach may be privileged.

Increasingly evaluators use “mixed methods” to optimise the configuration of components that make up the evaluation’s methodological framework as the evaluation’s methodology encompasses choice, and allows for mixing of:¹²

- Design, i.e. experimental, quasi-experimental, non-experimental;

- Approaches, e.g. theory-based, goal free, constructivist, empowerment, utilisation-focused, etc.);

- Qualitative and quantitative methods;

- Tools and techniques (e.g. logic modelling, strategic planning, concept mapping, etc.).

Methodology is also the basis on which validity and reliability of the evaluation is determined. As such, it is expected that the evaluation team develop methodology with due regard to minimising threats to validity and addressing issues of reliability.¹³

Validity
Validity is a measure of the extent to which, taken together, the evaluation’s design, data collection methods and analyses provide a reasonable basis for conclusions about the evaluation’s questions.

Four types of validity are important to consider in planning and carrying out evaluations:

- External validity – the extent to which issues related to the generalisation of conclusions are addressed; this is a particularly significant issue whenever sampling is used.

- Construct validity – the extent to which inferences can be made legitimately from the operationalisations, e.g. the specific questions and indicators, to the constructs on which they are based, i.e. the general evaluation criteria.

- Internal validity – whether observed changes can be attributed to the intervention, i.e. the cause and not to other possible causes (sometimes described as alternative explanations, rival hypotheses or counterfactuals, for outcomes and impacts).

- Conclusion validity – the degree to which conclusions reached about relationships in the data are reasonable.

12 Adapted from Weiss, H. B. (Summer 2005). The Evaluation Exchange: Editorial. Harvard Family Research Project.

13 Adapted from Trochim, W. M. K. (2002). Research Knowledge Database.

Reliability
Reliability is a measure of the quality of measurement; information is reliable if the measurement procedure yields the same results if applied repeatedly.

Reliability is a key factor for the quality of the evaluation and, as such, the evaluation team is expected to incorporate into its methodology ways of estimating the reliability of the data it gathers.

One of the most commonly used techniques for approximating the reliability of data is triangulation, applying the same measurement procedure to three or more different sources to obtain data that can be compared for similarity.

This type of reliability estimate is called internal consistency reliability; other forms of reliability estimates are:

- Parallel-forms reliability, e.g. where triangulation is applied to three different categories of data: verbal, documentary and observation.

- Test-retest reliability, used to assess the consistency of a measure from one time to another, e.g. where the same household survey is administered at periodic intervals.

- Inter-Rater or Inter-Observer Reliability, used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon, e.g. an assessment of the fairness and freedom of elections by different observers.

The evaluation team is expected to consider the relationship, i.e. the trade-offs, between issues of validity and reliability, as part of the development of the methodology.

For example, although a design that incorporates randomised control groups might be the most appropriate for dealing with issues of validity, availability and reliability of data might preclude such an approach (comparison group data, baseline data, surveys that could be substituted for baseline data, measures that are valid and reliable and so on). As well, the reality of the contexts in which most development interventions take place is such that even if such data are available they should be treated with caution.

As a result of these kinds of considerations an approach that makes use predominantly of qualitative data through interviews, observations and documentary reviews might often be more appropriate. What is expected of the evaluation team is that it undertake, and document in the evaluation file, these considerations systematically.

Sufficient and appropriate evidence
When making choices about the amount and nature of data to gather, it is expected that the evaluation team will collect only the information required to answer the evaluation questions.

Sufficiency has to do with the amount of information required to provide persuasive support for the contents of the evaluation report, i.e. will the collective weight of the evidence be sufficient to persuade a reasonable person that the observations and conclusions are valid, and the recommendations appropriate.

Some of the factors to consider when judging sufficiency are:

- The quality of the data, i.e. its relevance, reliability and validity;

- The significance of the finding and conclusion the data are intended to support, e.g. how important is it?

- How much assurance is intended, e.g. is the evaluation important for accountability purposes?

- What is the risk of making an incorrect observation or reaching an invalid conclusion?

- What is the cost of obtaining additional information in relation to its additional benefits, i.e. in terms of support for observations and conclusions?

Appropriateness of data includes questions of reliability and validity, but as well of relevance, i.e. the extent to which information bears a clear and logical relationship to the evaluation criteria and questions.

General categories of data are verbal, documentary and observational; as a general rule of thumb, observations are considered the most robust type of data followed by documentary and verbal. To the extent possible and appropriate, more than one type of data should be used to answer evaluation questions.

Overall, planning the conduct of the evaluation, and in particular the fieldwork, is an exercise in professional judgment, i.e. how best to answer the evaluation questions in light of methodological and resource constraints. (See “Shoestring Evaluation: Designing Impact Evaluations under Time, Budget and Data Constraints” by Michael Bamberger et al.)

Fieldwork: Collecting information

The purpose of fieldworks is to collect systematically and in accordance with the plan, the information required to support the formulation of conclusions, i.e. answers to the evaluation questions.

Fieldwork includes applying data collection methods, coordinating with the Evaluation Department, stakeholders and users, making adjustments to data collection as needed in response to constraints, to maintain and where possible improve the quality of the evaluation, undertaking analysis by formulating findings of fact based on the data and clearing these with concerned parties.

Data collection and choice of methods should take into account the specific challenges that may be posed by different languages in different contexts. As well, cultural sensitivities should be considered and respected both in the choice of data collection methods and in the way information is gathered.

Because evaluations often produce controversial results and may be subject to disagreements, the choice of methods of data collection and analysis may be a focus of criticisms.

It is expected that the evaluation team take into account, and communicate in its report, the strengths and weaknesses of the various methods employed, bearing in mind the considerations discussed during inception.

Fieldwork: Analysis

The purpose of analysis is to transform data into credible evidence about the development intervention and its performance. Typically, the analytical process involves three steps:

1. Organising the data for analysis, i.e. data preparation

2. Describing the data, e.g. generating findings of fact

3. Interpreting the data, e.g. assessing the findings against criteria

Preparation – Cleaning and classifying the data
Prior to and during fieldwork the instruments are used to collect information in the form of raw data, time-series, individual statements, etc. The first step in the analytical process involves preparing the data so that they meets standards of quality, i.e. validity and reliability, and that they are organised according to the evaluation questions and criteria they address.

Findings – What we saw, read and heard
The second step is to aggregate the data to constitute findings that are relevant to the evaluation questions at hand. Although judgment is involved in their formulation and selection, findings describe facts and do not constitute nor contain evaluative judgments in of themselves.

For example, as a result of collecting data on primary education a finding might be that the number of girls enrolled during a given school year was X. This might be a relevant finding to addressing the question of gender balance in the school system and contributing to forming an overall assessment of a national education policy for example. However, the finding in of itself is neither good nor bad, it is only a fact.

Conclusions – What we think
The third step in the analytical process is to interpret the findings, i.e. formulate conclusions, meaning usually in the case of evaluation to judge the findings in relation to a reasonable expectation, e.g. a standard, criterion, benchmark, target, indicator, good practice, etc.

For example, a finding that 40 per cent of girls enrolled in primary school complete their education over a given period, may be interpreted as “good” if the target was 40 per cent, “excellent” if the target was 30 per cent, “poor” if it has remained unchanged over the period and the policy was to increase the graduation rate of girls, etc.

The distinction between findings and conclusions, and its application to the processes of analysis, of clearance and of reporting are a cornerstone of Danida’s approach to ensuring the quality of its evaluation and constitute a fundamental expectation of the evaluation team’s work.

The ability for the evaluation team, the Evaluation Department as well as all other interested parties, to clearly distinguish between the agreed to facts and their interpretive conclusions, reduce the risk of general disagreement, focus critiques on specific aspects of evaluative judgment so that these can be dealt with, e.g. ultimately “agree to disagree”, contribute to the transparency and objectivity of the evaluation process as well as the credibility of the report.

Recommendations – What we suggest
The evaluation team is then expected to translate the conclusions into main recommendations which can be considered by management and subsequently translated into more specific and detailed decisions.

The evaluation team is usually not expected to provide recommendations at a detailed level. This should be the result of more comprehensive decision processes which will involve the organisation that has commissioned the evaluation and the different stakeholders.

Reporting: Preparation

Reporting the evaluation starts with its Terms of Reference. The evaluation team is bound by contract to ensure that the issues in the Terms of Reference are adequately addressed in the report.

The team leader normally drafts a table of contents at the inception stage of the evaluation, based on Danida’s requirements for reporting, the Terms of Reference and the introductory discussions with Danida. This also helps to distribute responsibilities between the members in the evaluation team.

The draft table of contents is a convenient means to organise information as the work proceeds. This includes information from initial desk studies, special studies, etc. Normally, the team prepares the outline of the draft report before departing for the fieldwork. This helps to focus the fieldwork to collect missing information, verify information, and draw and discuss conclusions and recommendations.

Usually, the team splits after the fieldwork. As a result, team members are normally obliged to complete their inputs to the draft report while in the field. This helps the team leader present a draft summary for discussion with country officials and the Embassy before leaving.

The draft of the report is reviewed by the Evaluation Department and Danida’s technical advisors and operational staff before it is distributed for wider consultation. They assess the quality of the report and submit comments regarding factual information, conclusions, and recommendations to be considered by the team before the final draft version is prepared.

This is usually done in a meeting or workshop with all relevant stakeholders where the evaluation team is debriefed. Danida then obtains comments on the final draft report from all partners and makes them available to the team. Such comments may be annexed to the report.

The team leader is responsible for submitting the final report and a summary (four pages). The structure and content of the report is discussed in the following sections. The final report should be in accordance with Danida’s layout for evaluation reports. The layout is available from the Evaluation Department.

Reporting: Structure

Because the focus of evaluations varies between sector programme, country evaluations, etc., there is no common format for reporting. However, Danida expects the evaluation team to report consistent with the DAC Evaluation Quality Standards and the following principles:

- The main report is relatively short. It contains the aggregated information and highlights. All detailed information is referred to annexes.

- Information is organised in a simple format which is easily accessible to the reader. This means that discussion of the evaluation criteria is a focus in the report.

The evaluation report consists of three levels of information. First, the executive summary is written as a self-contained paper that provides the bare essentials for decision-makers on background, major conclusions, recommendations and lessons learned.

The second level is the main report (40-50 pages) of which a substantial part is made up of the main conclusions and recommendations. These are substantiated with more detailed information only to the extent necessary.

Detailed findings are referred to the annexes. Conclusions and recommendations in the main report have references to the relevant findings in the annexes.

The third level in the report is the annexes. These provide all information necessary to substantiate major conclusions and recommendations in the main report. The Terms of Reference, the team’s itinerary, list of persons met, and list of documents used should also be annexed.

Danida’s evaluation reports are published in print. Annexes are included in the CD-ROM that accompanies the report and are also available on the website: http://www.evaluation.dk.

Reporting: Drafting and editing

Danida expects the evaluation team to maintain a professional standard in reporting. The evaluation team must employ the services of a professional copy editor.

Consultants are expected to present a draft report, which has been thoroughly copy edited, hence comments from the Evaluation Department and other stakeholders on presentation, language and structure can be reduced to a minimum. An unfinished draft is likely to provoke frustration among the readers and may result in tensions between consultants and stakeholders.

Readers have different needs for information. Some need highly aggregated information – others are interested in details. People also have different professional and educational background. The report is expected to meet the needs of all groups.

The language, use of terminology, illustrations, etc., are the main assistances for efficient communication. The report is written keeping target groups with the least specialist knowledge in mind.

Some general advice to make reports easier to read:

Brevity
- A rule of thumb is that the shorter a text, the greater the number of people who actually read it.

Structure
- The structure of the report is simple with titles and subtitles precisely identifying the topics discussed.

- A “horizontal” structure is used with not more than two levels of headings.

- The text is broken down in relatively small thematic or sequential parts, with simple and clear subtitles.

Editing
- The first sentence of paragraphs is used to make the main point, and the remainder to supplement, substantiate and discuss the main point.

- Illustrations, graphs, tables, etc. are used extensively to illustrate and elaborate on points that are emphasised.

- The details of tables and diagrams are not being repeated in written form. Instead, the reader’s attention should be focused on the important points that are derived, and related to the problem under discussion.

- Jargon and difficult words are avoided. As far as possible, the report uses words that the readers understand. Essential technical terms that may be new to the reader are defined in the text, and in a glossary.

- Long and complicated sentences are broken down in several short sentences. It is useful to give lists of items or conditions step-by-step.

- Active sentences are preferred. The more complicated passive form is used more to focus attention on a specific topic.

- Ideas are not crammed together in complex sentences with commas and brackets. Separate main ideas are presented in separate sentences.

- The meaning of abbreviations and colloquial words is explained. Abbreviations are used as little as possible.

- Simple link words are used to split sentences and indicate the direction in which the argument is moving. Link words are simple, such as: also; even so; on the other hand; in the same way. Long words like: moreover; nevertheless; notwithstanding, are avoided.

Numbers
- Only data tables or diagrams contain detailed numbers. The written text highlights only the most important numbers and says what they mean. Percentages are in most cases rounded to the nearest whole number.

- It should be possible for the reader to get the main message from a table without consulting the text. Every table has a title, table number, reference to the source of information, sample size, and full description of what each figure refers to.

- Pictures, diagrams and illustrations are used to make the report lively and interesting. These are often more effective than written text for communicating straight-forward messages.

- Use space around the text. Ease of reading and understanding is more important than reducing the volume of writing.

Feedback from the evaluation team

The evaluation team will draft a brief comment on its experience of the evaluation process so that the Evaluation Department may draw lessons on how it worked from the perspective of the consultants.

Quality control and assurance

Danida expects the evaluation team to have a systematic process for controlling and assuring the quality of its evaluation process and product. Danida contracts require that a quality manager be identified.

The DAC Evaluation Quality Standards constitutes a frame of reference for all Danida evaluations; the evaluation team must ensure that its evaluation process and product are consistent with these standards.

Expectations of quality in methodology, and to a lesser degree in management of the evaluation, have been addressed to some extent earlier in this chapter.

Another important dimension of quality has to do with the behaviour of professionals involved in Danida evaluations; the following two sections deal with overlapping expectations of professional practise and behaviour with stakeholders and other parties – Evaluation Ethics – and more specifically with Danida and related parties – Code of Conduct for Consultants.

Code of Conduct for consultants

Consultants have cited two particularly difficult phases in the evaluation process where uncertainty about roles may threaten their independence: organisation of fieldwork and discussion of the draft report. The code, therefore, concentrates on issues relating to these two phases in the evaluation process, while also addressing a range of other potential conflicts.

Consultants have cited two particularly difficult phases in the evaluation process where uncertainty about roles may threaten their independence: organisation of fieldwork and discussion of the draft report. The code, therefore, concentrates on issues relating to these two phases in the evaluation process, while also addressing a range of other potential conflicts.

Conclusions and recommendations of the evaluation report are discussed with the Evaluation Department, other Danida staff, partners and possibly a reference group, but it is the consultants who have ultimate responsibility for conclusions and recommendations.

- Consultants should be prepared to engage in discussions with Danida staff and other stakeholders without perceiving any difference of opinions as a form of pressure. There is an inherent tension in any evaluation assignment, and consultants should expect their interpretations to be challenged. Professional discussions can be difficult, and consultants should avoid feeling too easily offended or overly sensitive to criticism.

- Consultants, who believe that they have been subject to undue pressure, inappropriate behaviour or even threats from Danida staff or other stakeholders (including those responsible for the activities under evaluation) during fieldwork or while preparing the report, should report this immediately to the Evaluation Department.

- It is the responsibility of the team leader to decide, whether it is appropriate that the Evaluation Department and possible stakeholders participate in meetings, interviews and field visits. Stakeholders may include Embassy staff, Danida advisers and company advisers, partner representatives, other parties responsible for the activities under evaluation, and others. Issues relevant to the team leader’s decision include ensuring the impartiality of the evaluation, ensuring openness of discussions, and avoiding imbalance in numbers (too many ’evaluators’ present can make stakeholders uncomfortable).

- If consultants encounter or suspect corruption or other illicit practice, while carrying out their evaluation, it is their responsibility to report this to the Evaluation Department, which will ensure that Danida can take appropriate action. Only cases sustained by legally valid evidence can be reported in evaluation report.

- During the preparation and the undertaking of the fieldwork, the consultant is responsible for informing the Evaluation Department about the time schedule of the evaluation and the persons to be con tacted and interviewed. This information ensures coverage of all important sources of information and helps to avoid duplication and the evaluation becoming a burden upon a small group of key resource persons.

- After the draft report has been produced, consultants should avoid any direct contact to Danida staff and other stakeholders without prior agreement with the Evaluation Department. If consultants meet with Danida staff, the Evaluation Department should be present, and all correspondence between consultants and Danida staff should be copied to the Evaluation Department.

- Consultants should correct all factual errors and inaccuracies and make changes related to report structure, consistency, the analytical rigour, the validity of evidence, and requirements in Terms of Reference, as pointed out by the Evaluation Department. However, consultants should only agree to make changes to conclusions and recommendations of the evaluation that they regard as qualitative improvements. Where con sult ants disagree with changes proposed by the Evaluation Department, other Danida staff, and other stakeholders, they must present counter arguments. In cases of serious disagreement, the alternative assessments and counterarguments by the consultant should be presented in the report, if appropriate as footnotes. In cases of minor issues, the consultant must explain in writing to the Evaluation Department why suggested changes are not considered relevant or appropriate.

- Statements should not be made on behalf of the evaluation team if all team members have not had the opportunity to express their agreement. Unless disagreement has been clearly expressed in the report, all consultants involved in an assignment are expected to be loyal to the conclusions of the report. If internal team disagreement has not been reported, it is not appropriate that team members subsequently criticise the quality or the conclusions of the evaluation after it has been published.

- A particular case of internal team disagreement regards the situation in certain partner countries, where criticism can have serious consequences for national members of the evaluation team. Therefore, the team leader should ensure that national consultants either endorse a report in its entity or that their limited roles are clearly described in the report.

- Participation by consultants in the dissemination of evaluation findings after the publication of the evaluation report is appreciated. If consultants contribute to a public debate about the interpretation of the evaluation, however, this has to be agreed with the Evaluation Department. Particular attention should be paid to the issue of confidentiality of information obtained during the evaluation process, and consultants should not orally or in writing present information, which is not included in the report.

Evaluation ethics

Evaluation is a profession of persons with varying interests and differences in training, experience and work setting. Despite the diversity, the common ground is that evaluators aspire to provide quality information and assessment. Also that the evaluator has a responsibility versus the commissioning organisation, and the institutions, groups and individuals involved or affected by the evaluation.

Evaluators are expected to abide by international ethical standards. The following are adapted from those developed by the Canadian International Development Agency (CIDA, 1990):

Cultural intrusion
Evaluators should ensure that they are familiar with beliefs, manners and customs before going into the field. Both in data collection and reporting, it is imperative that customs regarding dress, personal interaction, religious beliefs and practices are respected.

Anonymity/Confidentiality
Evaluators must respect people’s right to provide information in confidence and ensure that sensitive data cannot be traced to its source.

Responsibility for evaluations
In developing countries, criticism can have serious consequences for a national. Evaluators must take care that those involved as local evaluators either endorse a report, or that their restricted roles are clearly described in the report. Statements should not be made on behalf of the evaluation team if other team members have not had an opportunity to disagree.

Right to privacy
Evaluators should realise that people can be extremely busy and their participation in evaluations can be burdensome. Therefore, evaluators should provide ample notice and minimise demands on time.

Supremacy of fundamental values
There is a delicate balance between certain cultural practices and the deprivation of fundamental rights and freedoms. While evaluators are expected to respect other cultures, they must also be aware of the values affecting minorities and particular groups. In such matters, the United Nations Universal Declaration of Human Rights (1948) is the operative guide.

Omissions
Ethically, evaluators have a responsibility to bring to light issues and findings which may not relate directly to the Terms of Reference. Certain other issues can cause difficulties for the evaluator and should be acknowledged and discussed with the Evaluation Department as necessary.

Evaluation of individuals
Performance evaluation is not normally a part of evaluations, though reports will touch issues such as leadership and management competence that border on evaluation of individuals. The evaluator is not expected to evaluate individuals and must balance an evaluation of management functions with this general principle.

Evidence of wrongdoing
Evaluations sometimes uncover evidence of wrongdoing. This can be complicated by an interpretation in the culture of the partner country that is different from the interpretation in the donor country. The evaluation team should consult the Evaluation Department when there are questions about what should be reported, how and to whom.

12 Adapted from Weiss, H. B. (Summer 2005). The Evaluation Exchange: Editorial. Harvard Family Research Project.

13 Adapted from Trochim, W. M. K. (2002). Research Knowledge Database.