Go to content

2 Evaluation Methodology and Approach

This chapter includes a brief version of the methodology and approach applied for the evaluation. For a more detailed explanation, see Annex 6.

2.1 Analytical framework

The design of the evaluation reflects the objectives and evaluation questions as set out in the ToR. The design has been tailored to make the most out of the resources at hand, by utilizing existing data with due consideration of its strengths and weaknesses and collecting the needed additional information. A key aspect of the evaluation is to identify and assess achievements, as well as the critical factors behind positive or less positive performances. The performance of the two AGEP components covered by the evaluation are addressed in line with the ToR:

  • For IFMC, a theory-based, mixed method approach has been applied to capture results at outcome and impact level in a credible manner. This involved both a survey following up on an earlier baseline study, as well as qualitative fieldwork and programme documentation review.
     
  • For the AFSP component, the assessment builds on a combination of the recent UNDP-commissioned end-evaluation (which included a household survey) supplemented with a qualitative fieldwork mission.

The evaluation used the AGEP programme information, including the logical framework and results frameworks for IFMC and AFSP (see Annex 4) to establish an overall Theory of Change (ToC).[9] This has been key to understanding whether the support has worked as intended and whether programme assumptions materialized or not. This includes the role of contextual and external factors influencing the programme. More specifically, the analytical framework entails a mixed methods approach including both rigorous quantitative impact assessment, and contribution analysis combined with more qualitative aspects, all informed by the ToC.[10]

Evaluation questions and criteria:

The 10 Evaluation Questions (EQs) from the ToR provide the overall framework for the evaluation assignment (the EQs are presented in the Evaluation Matrix in Annex 6 together with judgement criteria and indicators). These questions, together with the evaluation objectives and key issues, are the basis of the design and methodological approach. Both qualitative and quantitative indicators have been used, as the range of issues are multifaceted and require indicators that capture this complexity (i.e. issues related to the implementation processes content/ quality, results achieved, efficiency, adaptation to context, etc.) The table below summarises the application of the evaluation matrix in the current report, how the EQs relate to the OECD/DAC criteria and in which chapter they are being addressed.

Table 3. EQs related to OECD/DAC criteria
EQs Topic Report OECD/DAC
4, 5 and 8 Implementation of FFS Chapter 5 (5.1, 5.2) Relevance, effectiveness, sustainability
1, 2, 4, 6, 7, 9 Results of FFS Chapter 5 (5.3-5.6) Effectiveness, impact, efficiency, sustainability
3 Value for money Chapter 6 Efficiency
10 Lessons learned/ recommendation Chapter 8  

2.2 Methods for data collection and analysis

The overall approach to data collection and analysis was based on a mixed-methods approach.

Household survey

A household survey was implemented within four different regions; two in the North (Rangpur and Rajshahi) and two in the South (Barisal and Chittagong (Feni District)). The survey was implemented as a follow-up to a baseline survey (implemented in 2014) with some adjustments to enhance its usefulness to the evaluation. This included mainly adding of FFS participants to the survey as most of the households included in the baseline study had not become participants in FFS. It was therefore also necessary to supplement the baseline questionnaire with recall questions, in order to gather baseline information about the FFS participants.[11] Data collected from FFS households by AGEP prior to the start of the FFS were also used to construct a proxy baseline for FFS participants.[12] The resulting data base with information regarding both the situation before and after the implementation of the FFS, and for both FFS-participants and non-participants, has been used to identify the specific results that can be attributed to IFMC.[13]

Within the four regions, 14 villages (including a total of 19 FFS groups) were identified across the baseline survey villages and the list of FFS villages. These 14 villages were located within the four above-mentioned regions. In addition, 34 non-FFS villages from the baseline survey (all located within the same four regions) were included in the control group.

Use of two different types of control groups

It should be noted, that the evaluation has made use of two different types of control groups (control households both from within (Non-FFS) and outside FFS villages (Control)). This has allowed for a more nuanced assessment of variance in results patterns, including possible spill-over effects from different types of FFS interventions. In most cases, the data analysis has been done for both Non-FFS and Control households. However, on women empowerment, the analysis has focussed mainly on FFS vs. Non-FFS data, i.e. changes observed within the FFS villages, since a number of gender and women empowerment projects are being implemented all over Bangladesh, thus the risk for contamination of control villages would be high. For land ownership the comparison is also made only between FFS and Control households, since increased land ownership among FFS households could be at the cost of less ownership among Non-FFS households within the same villages.

These are defined in the Table 4.

Table 4. Definition of households included in the survey
Households Definition
FFS household Households included in FFS, treatment group
Non-FFS household Households from FFS villages but not part of FFS
Control households Households from villages where no FFS has been implemented

The household survey effectively covered 965 households (388 FFS households and 577 control households), over the four regions as listed in Table 5.

Table 5. Households included in the survey per region

Region

FFS households

Total # of households surveyed

Rangpur 85 249
Rajshahi 109 289
Barisal 121 303
Chittagong (Feni) 73 124
Total 388 965

The evaluation has used a propensity score matching approach[14] to carry out an econometric analysis of the collected household data, based, to a large extent, on a matched double difference approach.[15] The robustness of the results from the econometric data analyses have been tested at the 1% (most significant), 5% and 10% (least significant) statistical significance level.

Qualitative data collection

The qualitative fieldwork was designed to be implemented after the preliminary results from the household survey were known. This sequencing allowed an element of follow-up on particular interesting findings and results from the survey, including more in-depth assessment of specific issues. Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs) together with site inspections were applied as the key qualitative methods by the evaluation. The fieldwork covered visit to three regions (Rangpur in the North, Barisal in the South and Chittagong Hill Tracts), where the Upazilas and villages listed in Table 6 were visited during a three-weeks period:

Table 6. Qualitative fieldwork in Rangpur, Barisal and CHT regions
Rangpur region
Upazila Village Female farmers Male farmers
Pirgacha Upazila Bara Hayat Khan village 11 0
Gunjar Khan Amintari Village 11 8
Uttar Chandipur Village 6 7
Palashbari Upazila Basudebpur, Bhagwanpur village 12 5
Balarampur Village 14 3
Purbo Gopalpur village 9 6
Paschim Goalpara village 20 18
Barisal region
Betagi Upazila Dakshin Hosnabad village 14 5
Chandkali village 11 6
Uttar Kawnia 15 4
Chittagong Hill Tracts region
Rangamati upazila Borodona village 23 4
Langadu upazila Ishaqpara village 22 9
Naniarchar upazila Jogendrapara village 20 12
Total
  188 87

The selection of Upazilas and villages for fieldwork visits was based on a wish to be able to study how the market linkage element was implemented, and to be able to cover implementation of FFS activities within different provinces (rich/poor) and within different agro-ecological zones; activities that were completed some time ago (potential impact and sustainability issues), as well as more recent activities (more focus on outcomes) as well as logistics and practicability of travel.

The following group of stakeholders was covered as part of the qualitative fieldwork:

  • 3 Upazila Agricultural Officers (UAOs), all males
     
  • 3 Upazila FFS coordinators in CHT
     
  • 3 Sub-Assistant Plant Protection Officers (SAPPOs), all males
     
  • 8 Sub-Assistant Agriculture Officers (SAAOs), all males
     
  • 2 District officers/coordinators in CHT (both males)
     
  • 26 FGDs with FFS farmers, 188 females and 87 males
     
  • 2 FGDs with non-FFS farmers in Barisal, 13 females and 6 males
     
  • 7 non-FFS members, 2 males, 5 females, in Rangpur FFS villages
     
  • 23 farmers facilitators (FFs)
     
  • 13 executive members and Business focal Points (BFPs), 7 male/5 females
     
  • 7 Farmer Organisations (FOs)
     
  • 6 UNDP technical programme officers (M&E, livelihoods, training)
     
  • 6 UNDP Master trainers

In addition to the KIIs and FGDs, the evaluation made direct observations within the visited villages of FFS technology uptake and/or any changes at village/household level resulting from FFS activities.

Data collection is summed up in the map below with indications of where the household survey and qualitative fieldwork was conducted.

2.3 Limitations and Challenges

As for any evaluation, there are some important limitations and challenges to consider. While the approach to data collection and analysis was planned to address and remedy these challenges as far as possible, there were nevertheless a range of issues that must be kept in mind.

Risk of positive bias

The risk of positive bias was considered up front; both in relation to the monitoring data, and in terms of possible “diplomatic” bias during data collection, either due to general politeness and tendencies towards confirmations; or due to risk of showcasing and preparation of informants, whether by design or unconsciously. This posed challenges for both the quantitative data collection and the qualitative fieldwork. It was not possible to carry out the household survey without some involvement of DAE staff, due to the need for mobilization of the villagers. However, in general, the survey enumerators and the fieldwork team could carry out their activities in an unsupervised manner. With regards to the use of programme monitoring data, the risk of positive bias has had the implication that the evaluation has used these only in combination with other data sources and mainly for descriptive purposes.

Programme stand-still

The fact that the evaluation investigated a programme that was not under implementation, posed limitations on both observations of practice and the dialogue regarding, for instance, selection processes, use of manuals and guidelines, etc. For instance, it would have been useful to observe training sessions with the changes in curriculum and number of training sessions that have taken place, as part of the evaluation’s relevance and quality assessment.[16]

Baseline data and “real world” limitations for the quantitative impact assessment

The household survey has required the use of supplementary data sources, such as household information sheets (filled in prior to enrolment in FFS by Upazila officers), and the incorporation of recall questions in order to capture the baseline situation for FFS participants. Furthermore, there may be nuances and effects, which do not come across as significant in the survey, simply because they are hard to detect, and not because they are not “real” results. By implication, care should be taken not to over interpret details of the responses. Rather, the key messages to be taken from the survey are thus the broader lines of changes and results. In outlining the findings, care has been taken to explain the strength of the different findings, both as they come across in the quantitative analysis, but also with consideration given to the qualitative fieldwork and other data sources.

Use of different methodologies for the AFSP and IFMC household surveys

The recent AFSP II End-Evaluation is an important data source for the evaluation, as it conveys findings regarding the results of FFS as implemented in CHT. Thus, it is important to briefly outline some of its limitations and the implications for its interpretation.

The evaluation findings build on comparison of control and treatment groups, and the report states that the only difference between the control and treatment group is that the treatment group participated in AFSP II. However, the selection of the control group is done on a very limited number of characteristics, namely land holding size and gender of the household head. Furthermore, the control group is about half the size of the treatment group, which makes it more difficult to ensure that the control group and the comparison group are similar.[17] The risk of positive bias is a concern as well. While this is also true for the follow-up survey, there are some additional issues for the AFSP survey. For instance, there are questions about positive change that have been posed in a manner that may lead to the informant feeling “invited” to provide a positive answer.[18]

In order to address potential bias, a small qualitative fieldwork exercise in CHT was added to the evaluation approach. This has allowed for validation of the AFSP end-evaluation, for the triangulation of findings, and for working with a richer picture of processes and experiences. Nevertheless, the specific findings from the AFSP evaluation should be interpreted with these limitations in mind. It should also be mentioned that, due to the use of different methodologies, different wordings in the questionnaires, etc., direct and specific comparisons between the two double-difference analyses has not been attempted (for instance in relation to income amounts).


[9] A ToC was not included in the AGEP Programme Document. See Annex 7 for an overview of the reconstructed ToC

[10] The approach serves as a way of framing the work with the ToC. More information on contribution analysis and the mixed methods approach can be found in Annex 6. The operational implications of the approach are covered in the sections on the ToC as an analytical framework, the quantitative and qualitative data collection and challenges and limitations respectively.

[11] While use of recall is less than ideal (due to the obvious problem of accuracy), care has been taken to ensure that it draws on research into what types of issues are most relevant for more detailed recall questions.

[12] It should be stressed that the qualitative field work raised issues regarding the accuracy and reliability of these data, indicating that the information may not always have been provided directly from the farmers and the register having been filled in later by programme staff (see below for more regarding challenges and limitations).

[13] A so-called double different approach. The double difference measures the difference in the observed change between participating households/ individuals and control village households/individuals, based on baseline (recall) data and ex-post data. Thus, the double difference eliminates external determinants of the outcome, in cases where these are the same for the two groups during the intervention period. The double difference approach assumes common time effects across groups and no composition changes within each group. In order to identify results stemming from FFS in CHT, the UNDP has also commissioned an impact assessment based on a double-difference approach, but this has been implemented with a somewhat different (technical) methodology (see further below).

[14] Mathematical technique used to select members of the control group that share characteristics with members of the participants group, through estimation of a statistical model based on matching characteristics (household characteristics).

[15] The information on general household characteristics (size of land, education (years), household size, number of males/females) in the data set has been used fully in the matching approach pursued.

[16] The situation also came with logistical challenges related to ensuring hard copies of programme information from the field prior to termination of the programme – a challenge programme staff was most helpful in trying to remedy – and in the availability of regional programme staff for interviews.

[17] For instance, the treatment group has more experience on farming than the control group, the number of working family members is higher, and there is a marked difference in the yield reported in the “before” scenario. Working with a smaller control sample can be perfectly fine, if one is very certain about the similarities between the two groups. This, however, cannot simply be assumed to be the case and therefore a small sample size is problematic. For comparison, the follow-up survey established a sample of 388 FFS households and 577 control households).

[18] Care must always be taken to ensure that the wording of questions do not add to the risk of positive bias, by “inviting” positive answers (for instance, it is seen as better to ask about change without indicating any direction of the change, rather than asking whether improvements have happened, asking the informant to confirm or deny). In this light, it may have felt too natural for respondents to simply say “yes” to questions like: “Do you think your knowledge and perception on improved technology after participation in the project increased?” or “Do you think that AFSP II project has increased your farm income?”).

Top

This page forms part of the publication "Evaluation of Agricultural Growth & Employment Programme (AGEP), Bangladesh – October 2019" as chapter 2 of 8.
Version no. 1.0, 2020-01-16
Publication may be found at the address http://www.netpublikationer.dk/um/evaluation_agep_bangladesh_oct19/index.html