2 Evaluation Methodology and Approach
The methodology applied by this Evaluation is underpinned by Danida’s Guidelines for Evaluation (MFA/Danida, 2006) and the Organisation for Economic Cooperation and Development (OECD)/Development Assistance Committee (DAC) Evaluation Quality Standards. The Evaluation’s definition of the OECD/DAC Standard Evaluation Criteria (Table 2.1) is in accordance with the ToR for the assignment.
Table 2.1 Definitions of OECD/DAC Standard Evaluation Criteria
||“The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirement, country needs, global priorities and partners’ and donors’ policies”.|
||“A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results”.|
||“The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance”.|
||“The positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”.|
||“The continuation of benefits from a development intervention after major development assistance has been completed. Probability of long-term benefits. The resilience to risk of the net benefit flows over time”.|
The overall approach to data collection and analysis has been based on a mixed-methods approach, combining rigorous quantitative data analysis with qualitative data collection and study of literature. Consequently, the evaluation analysis is based on three complementary elements:
- analysis of existing (secondary) quantitative data;
- qualitative fieldwork and analysis of data/information; and
- a FFS literature study.
One clear benefit from combining qualitative fieldwork with quantitative data analysis and extensive literature study is that data triangulation can be used as a main tool for the validation and analysis process. Through data triangulation, the Evaluation has verified findings from different sources and methods to increase the credibility and robustness of the analysis.
In the following Section 2.1 the overall analytical framework, is presented. This is followed by brief descriptions in Sections 2.2-2.4 of the specific methodologies and approaches applied for each of the three above mentioned elements. For more details on the methodology and approach applied for data collection and analysis, please refer to Annex 2.
2.1 Analytical Framework
Figure 2.1 below illustrates the overall logic of the analytical frame.
Figure 2.1 Overall Analytical Framework
The FFS approach is applied within both AEC and RFLDC, however, the modalities through which FFS are implemented differ across the two components (see Chapter 3). As a consequence of this, the processing of data and information, as well as the first step in the analysis, will be undertaken separately for AEC and RFLDC (as illustrated in Figure 2.1).
Based on the individual analyses of FFS experiences from, respectively, AEC and RFLDC, and key findings from a literature study on experiences from FFS outside ASPS II (see Section 2.4), a comparative analysis of institutional arrangements and cost-benefit aspects of FFS, with particular relevance to the Bangladesh context, has been carried out.
Finally, conclusions have been drawn from the individual and comparative analyses, leading to formulation of a number of lessons learned and recommendations.
2.2 Quantitative Data Analysis
The quantitative data analysis is mainly based on data collected by independent survey teams as part of two recent external Mid-Term Evaluations carried out for, respectively, AEC and RFLDC:
- Data for the AEC Mid-Term Evaluation were collected during February 2011 through a comprehensive questionnaire (survey instrument) covering a number of outcome variables as well as a series of control variable indicators for general household characteristics (including gender, age, marital status, education level, occupation, household size and land ownership). The data collected consist of information on 1,088 FFS participating households and 228 control village households. This means that control village households have been severely under-sampled, creating potential ’common support’ (overlap condition) problems when applying statistical matching methods to the data set. The AEC Mid-Term Evaluation does not include estimates on the ’before-FFS’ situation and, unfortunately, it does not link explicitly to an AEC Baseline Study carried out in 2007.
Based on the available data, the Evaluation has to the extent possible used a post-intervention propensity score matching approach to carry out an econometric analysis for AEC. The rich information on general household characteristics in the data set has been used fully in the matching approach pursued.
- Data for the RFLDC Mid-Term Evaluation were collected during June 2010 and are comparable to those described in the AEC case. The control variables collected in terms of general household characteristics are useful, and the questionnaire is also quite comprehensive in terms of appropriate outcome variables. The questionnaire includes recall questions to establish the baseline (before-FFS intervention level). The RFLDC Mid-Term Evaluation data consist of information on 640 FFS participating households and 224 control village households. Since the amount of control village household is less than 1/3 of the total this again raises potential challenges in terms of fulfilling the overlap condition in matching procedures.
The inclusion of recall questions in the questionnaire, together with a comprehensive set of control variables (household characteristics), has made it possible for the Evaluation to carry out an econometric data analysis for RFLDC based, to a large extent, on a matched double difference approach.
The robustness of the results from the econometric data analyses has been tested at the 1% (most significant), 5% and 10% (least significant) significance level.
In addition to the above-mentioned externally collected data set, a large amount of internal monitoring data and studies has been provided by AEC and RFLDC and used for the analysis.
2.3 Qualitative Fieldwork and Studies
A key concern for the Evaluation, in relation to planning of the qualitative fieldwork, was to get the opportunity to study the full ’chain’ of selection processes in the FFS approach (i.e. from the selection of Unions and Villages down to selection of trainers/facilitators and, ultimately, the beneficiaries and topics for the FFS sessions), including the rationale and consequences related to these choices. A clear practical understanding of these selection aspects is important in order not to over or under estimate the potential impacts from FFS interventions, as well as for the analysis of various social and qualitative aspects.
The qualitative fieldwork was based on half-day studies of four FFS village ’cases’ within each region (north and northwest Bangladesh (AEC), Noakhali (RFLDC) and Barisal (RFLDC)). Each FFS village ’case’ study included a visit to a FFS village as well as to a ’control village’ (where FFS sessions had not been undertaken) within the same Union. Table 2.2 provides an overview of the number and type of villages visited during the fieldwork.
Table 2.2 Overview of village coverage for qualitative fieldwork
|FFS villages (completed FFS)
|FFS Indigenous villages
Given the practical and logistic limitations of the fieldwork coverage (four days in each main geographic area), the Evaluation aimed at selecting a diversified sample of villages to be studied. The parameters for the village case selection included:
- FFS methods: different methods applied for implementation of the FFS approach within ASPS II.
- Geographical area: FFS activities implemented in different provinces (rich/poor) and within different agro-ecological zones within the regions.
- Status/length of implementation: activities that may already have been completed some time ago (potential impact and sustainability issues), as well as more recent activities (more focus on relevance, efficiency and effectiveness aspects).
- Performance status: activities that are performing well and less well.
- Practicability of travel: travel logistics within the regions provided limitations for how much and what could be covered during the four-day visit to each region. Likewise, it was necessary to balance the time between visits to farm sites, Focus Group Discussions (FGDs) and interviews with different stakeholder groups.
The following main qualitative methods and tools were applied during the fieldwork mission for collection of data and information:
- FGDs and/or individual interviews with key stakeholders at Upazila and Union level:
- extension services in relevant departments (male and female);
- farmer organisation members (male and female); and
- NGOs, private service providers and traders.
- FGDs and/or individual interviews with key stakeholders at village level:
- FFS facilitators/trainers (male and female);
- FFS participants from completed and ongoing FFS (male and female); and
- control groups: farmers from control villages and non-FFS farmers from FFS villages (male and female).
- Direct observation of FFS sessions; FGDs with FFS participants in the field (male and female).
- Direct observation of FFS technologies/activities being implemented by graduates.
- Direct observation of Training of Trainers (ToT) and Season-Long Learning sessions.
An evaluation matrix with key evaluation questions and indicators was used to prepare standardised ’checklists’ for the FGDs with different stakeholder groups (FFS farmers, control farmers, facilitators/trainers, CBO/Farmer Club leaders etc.) to ensure that similar type of data and information would be collected across the components and geographic areas.
A total of approximately 750 FFS participants (500 completed and 250 ongoing, half male and half female) and 500 control village household members (half male and half female) have been consulted by the Evaluation through the FGDs. The men and women selected for the FGDs have been of different age and socio-cultural background, reflecting the composition of the FFS groups.
A total of 57 trainers/facilitators (39 male and 18 female) were consulted; 24 (17 male and seven female) from AEC and 33 (22 male and 11 female) from RFLDC.
2.4 FFS Literature Study
While searching for solutions to the identified FFS topics, or for improvement of earlier efforts in the ASPS II components, the Evaluation deemed it very useful to study lessons learned on challenges and opportunities from other FFS initiatives that have been presented and discussed in different formal, and informal, publications and fora.
A FFS literature study is therefore included as an integrated part of this evaluation (Annex 3). The study includes a scrutiny of existing relevant information from Bangladesh, the south and southeast Asian regions, as well as experiences from Africa where comparable problems have been faced, questions been asked and solutions been sought.
The literature study focuses on four main areas: i) mainstreaming and sustainability of FFS interventions; ii) marketing and farmer organisations; iii) cost-benefit and monitoring/evaluation of FFS; and iv) suitability of the FFS approach for non-rice topics and resource-poor rural populations.
 The overlap condition ensures that observations from FFS participating households have control village observations ’nearby’ in the propensity score distribution. Specifically, the effectiveness of matching depends on having a large and roughly equal number of participant and control observations so that a substantial region of ’common support’ (overlap) can be found. Participating units will have to be similar to control units in terms of observed characteristics unaffected by participation; thus, some control units may have to be dropped to ensure comparability.
 In addition, a one day visit to the Chittagong Foothills was included to observe the FFS experimental activities carried out here.
This page forms part of the publication 'Evaluation of the Farmer Field School Approach in the Agriculture Sector Programme Support Phase II, Bangladesh' as chapter 6 of 14
Version 1.0. 22-12-2011
Publication may be found at the address http://www.netpublikationer.dk/um/11112/index.htm