3 Methodology and Approach
3.1 Analytical Framework
Overall analytical framework
In the context of the OECD/DAC evaluation criteria the analysis considered two broad aspects of Danida support:
- the mechanisms and processes of providing support
- the products and outcomes of successful delivery
The DAC criteria are now fairly standard across evaluation work and ensured compliance with international best practice. Data collection and analysis was based on a mixed-methods approach,combining quantitative data analysis with qualitative methods (see below).
Quantitative methods measured products and qualitative methods were used to explore issues of process, creating a better understanding of possible changes by which a particular intervention triggered a series of events that ultimately resulted in an observable changes or outcomes.
Between 2006 and 2011, support to the activities of the Centres and Networks gradually reduced as these were merged and absorbed into different structures, however the links and output they created continued to support and synergise other areas of support.
By 2011 the FFU projects and BSU had become the key focus for Danida, with some activity under Minor Studies. The position of this support which aimed to strengthen capacity and produce research output at organisations in target countries, which then fed into a general strengthening of National Agricultural Research Systems (NARS), is shown in Figure 2 as part of a theory of change diagram.
This broad and simplified theory of change links the support to capacity strengthening and research provided by the five modalities being evaluated to the wider issue of creating stronger NARS. It has been articulated here by the Evaluation to reflect the policy and focus of development support during the period under consideration.
The definition of a NARS is extensive and inclusive, involving universities, national agricultural research institutes, extension services, private sector including NGO and processers and civil society. It is also important to include, within the concept of a NARS, the enabling policy environment in which the system operates and the research which is linked to it including for example, research on the development of appropriate land rights policy and issues linked to cross-border and internal trade and markets.
The creation of stronger NARS underpins much of the approach in Africa which falls under the Comprehensive African Agricultural Development Programme (CAADP), supported by the Framework for African Agricultural Productivity (FAAP), which aims to use agricultural growth as the engine to provide economic growth and poverty reduction, and the same principle applies elsewhere.
Figure 2 Broad Theory of Change for Danida Support to Agriculture and NRM Research
Dual aspects of the Evaluation – history and future perspective
There were two broad aspects to the Evaluation. The first was to evaluate the historical performance of the activities being funded, which was a conventional look at performance, based on the DAC criteria. Measuring and assessing that the extent interventions were relevantin the context of the contemporary strategies and priorities, how effectivethey were in delivering the required results, how efficientwas the use of resources and to what extent were the desired outcomes and impactachieved and potentially sustained?
The second was to take the evaluation information and analyses and use them to look forward and consider the lessons learned from past experience and how the best use could be made in formulating a new strategy for Danida support to development researchdesigned around a new and innovative approach.
An Evaluation Matrix (see Annex B) developed during the inception phase from the 24 questions posed by the ToR, identified key issues to be addressed.
Data collection instruments and analysis
Several different tools and modalities were used during the Evaluation; these are briefly outlined below.
Document and literature review
A large number of documents were reviewed and discussed including policy, strategy, review and project reports. The key reports are listed at Annex C, and are referred to in footnotes in this report, where particularly relevant. The review of documentation allowed, inter alia,an analysis of the relevance of design, the efficiency with which resources have been used to create the necessary and sufficient products for delivery and the extent to which results have been delivered and objectives achieved. Prior to country visits (see below) the review facilitated the development of project fact sheets.
Online survey – eSurvey
An eSurvey was conducted using proprietary software designed for such work. The aim was to provide data for triangulating information collected from other sources as well as increase ownership of the evaluation process by engaging with a wide cohort of stakeholders. It was not intended as a major data-collecting instrument.
The focus of the eSurvey was stakeholders who had been, or who were currently, engaged in FFU North- and South-driven projects, 324 invitations were sent out by email of which approximately 111 (34%) were from Denmark and 213 (66%) from partner countries. A total of 86 responses (27%) were received, of which 23% were women. The majority of respondents were from universities (74%) with the bulk of the balance being national research institutes (18%) and ministries (4%). The completion rate was slightly higher from Denmark (32%) compared with respondents in partner countries (24%)
Responses to the survey are shown in full at Annex D, and are included, where appro priate, in Chapters 7 and 8; a summary of the responses to the open questionon Danida support is given also given at Annex D and have been taken into consideration in the analysis and development of recommendations.
Overall the eSurvey proved less useful than had been anticipated, see below, although it broadly confirmed information from other sources.
Key informant interviews
This tool used a semi-structured approach with a checklist, based on the Evaluation Matrix, to guide the process. It provided one of the major data collection methods.
Key Informantswere the project coordinators, country-based project leaders and contact and management level stakeholders associated with the FFU and network/BSU/KU-LIFE funding mechanisms. Interviews were conducted in person (Denmark, Burkina Faso and Tanzania), and via video and telephone links where this was not possible.
In addition to these key informants, departmental and institution heads and directors not directly engaged in activities, were interviewed to gain their perspective of Danida support in the broader picture of development partner activities and priorities.
Table 8 shows a summary of the numbers of individuals interviewed and with whom the Evaluation interacted.
A table summarising the selected projects is given at Annex E.
A total of 41 FFU projects (out of 83) and all six South-driven projects were engaged with directly. In addition the views and responses from 28 participants at a one day Emerging Issues Workshopin Copenhagen (see Annex G) provided input into the Evaluation. Many of those interviewed had interests in different activities funded by Danida, and this was especially true for the BSU initiative where the 18 people shown in Table 8 refer onlyto those directly engaged in Platform management. Overall the Evaluation took into consideration input from a diverse selection of stakeholders.
Table 8 Numbers of stakeholders interviewed during the Evaluation
|FFU Southern-driven Projects
|Centres – Management position
|BSU EC and GE – Management
|Danish Fellowship Centre
SWOT analysis workshop
In addition to interviews and site visits, SWOT Analysis Workshops in which stakeholders analysed the strengths, weaknesses, opportunities and threats of Danida support were held in Burkina Faso (15 participants), Tanzania (14) and Denmark (10). The workshops provided opportunities for informal discussions and interaction between a broad-base of stakeholders who had been through similar experiences and processes.
A summary of the recommendations made by workshop participants is given at Annex F.
In order to assess potential or actual impact at the level of end-users or beneficiaries field visits were made to two project sites in Burkina Faso and two in Tanzania.
Emerging issues workshop
After field visits and interviews had been completed, a Workshop was held in Copenhagen to share emerging issueswith key stakeholders. The objective was to sensitise both Danish and Southern partners to some of the main conclusions of the Evaluation and to seek their responses and comments. The report of the workshop is at Annex G, and findings were incorporated into this report, where appropriate.
An assessment of FFU Project Completion Reports indicated that inconsistent reporting on publications, and the sometimes long delays in research being accepted for publication, made such an analysis unrealistic. However, the Evaluation does present some information on publications as a potential indicator of the academic quality of research.
Triangulation of findings
The core tools for collecting data were the Key Informant Interviews, SWOT Analyses and Literature and Document Review.The Site Visitsand eSurveyprovided sources of additional information and data feeding into the overall evaluation and providing triangulation of the other findings.
Interplay between in-depth and overall portfolio analysis
The need to understand and document the big pictureperformance of Danida support to research in agriculture and natural resource management has been key to the Evaluation; to provide the evidence to guide and support the development of a fresh strategic approach is a key objective.
To this end the tools described in this section have been used on projects and initiatives selected using carefully developed criteria for in-depth analysis. Coverage was triangulated through the use of the multiple mechanisms described above. Two key instruments in developing the overall picture have been interaction and feedback with EVAL, UFT, the new strategy development team, Reference Group Committee and DFC as the Evaluation progressed, and a workshop on Emerging Issues (see Annex G) in the latter stages of the process.
3.2 Field Visits to Burkina Faso and Tanzania
The programmes for the country visits are shown at Annex H. Interactions with stakeholders took place through meetings with individuals and groups engaged with work supported by Danida, the SWOT Analysis Workshops and site visits. It should be noted that the larger number of North-driven FFU projects in Tanzania is reflected in the reporting on findings in Chapter 7 and the absence of South-driven FFU projects in Burkina Faso is the reason for the emphasis on Tanzania in Chapter 8.
3.3 Limitations and Challenges
The Evaluation has been limited to a fixed time period of 2006 to 2011 and five different funding mechanisms. Given the time available for the Evaluation, the number of funding instruments, their geographic coverage and the six-year time span it was not possible to make a comprehensive, detailed evaluation of all activities, and a compromise had to be established which balanced detail of coverage with the scale of the study.
To this end the focus has been on stakeholders in Denmark, Tanzania and Burkina Faso. Danish stakeholders were able to provide their perspectives of activities based in the non-visit countries (as well as Tanzania and Burkina Faso) but it was not possible to interview stakeholders in these other countries due to time and travel constraints. Every effort has been made to provide as much detail as possible and caveats are placed on conclusions where it is deemed appropriate.
The assessment of the BSU modality was limited to two of the four Platforms, and does not necessarily reflect the status or condition of the others. Recommendations and conclusions specifically address constraints and issues facing the Environment and Climate and Growth and Employment Platformsin Tanzania and Ghana. Only Tanzanian stakeholders were interviewed about these two platforms in-country although most of the Danish interviewees had had experience of both countries.
The Evaluation has also been limited to support for agriculture and natural resource management and the detailed assessment of the South-driven research projects under PRCP is limited to the five projects in Tanzania.
The lack of a strategic framework or logframe/results framework for support to development research, and for the individual modalities, has created difficulties during the Evaluation. Without a clear statement of what the support between 2006 and 2011 was intended to deliver and achieve both overall and within each of the modalities and the absence of indicators that would demonstrate success, assessments of effectivenesshave been difficult. Where indicators have been identified they have been limited to measures of product such as the number of degrees awarded or meetings held. There are few, if any baseline figures or data sets on which to base judgements of improvement.
The stakeholders, duration, size, location and institutional homes of the North- and South-driven FFU projects created a highly variable cohort of potential respondents for the eSurvey. This created difficulties in articulating simple and easily understood questions that could be answered clearly and quickly. This has limited its usefulness and it has only been possible to draw broad conclusions from responses, however nothing in the responses contradicts information collected using other methods.
Despite the limitations and challenges, the Evaluation is confident that the findings and recommendations described in this report fairly reflect the views and data made available and represent an accurate assessment of the period and modalities being evaluated.
 Relevance, Effectiveness, Efficiency, Sustainability and Impact.
 Products such as the number of PhDs completed or the number of peer-reviewed papers published.
 Outcomes such as the level and quality of new grants obtained by newly qualified researchers for research addressing demand-driven needs or the number and quality of fresh initiatives building on published research findings.
 Figure 2 is illustrative and simplified, but is included to demonstrate the role that Danida support to agriculture and NRM research has in the wider context of development, and economic growth, and the part it plays in contributing to the strengthening of NARS.
 Comprehensive Africa Agriculture Development Programme. NEPAD Midrand, South Africa. August 2002.
 Framework for African Agricultural Productivity. June 2006. Forum for Agricultural Research in Africa, Secretariat, Accra, Ghana.
 SurveyMonkey – see http://www.surveymonkey.com for further description and overview of the tool.
 North-driven projectswere those identified and led by Danish organisations, the South-driven projectswere those under the Pilot Research Cooperation Programme, and were formerly referred to as Pilot Projects.
This page forms part of the publication 'Evaluation of Danida supported Research on Agriculture and Natural Resource Management 2006-2011' as chapter 6 of 16
Version 1.0. 09-09-2013
Publication may be found at the address http://www.netpublikationer.dk/um/11214/index.htm