Go to content

3 Evaluation Methodology and Framework

3.1 Overall evaluation framework

The 5Capitals approach

The “5Capitals – A Tool for Assessing the Poverty Impacts of Value Chain Development”[4] (in the following just referred to as ”5Capitals”) provides an useful approach for assessment of outcomes and impact from VCD interventions. 5Capitals is an impact assessment tool developed by the Tropical Agricultural Research and Higher Education Center (CATIE) to facilitate learning from VCD interventions.

The core of 5Capitals is an asset-based approach, where observed changes in household and business assets resulting from VCD interventions are used as indicators for improved household living standers and business performance. Changes in the stocks and flows of critical household and linked-enterprise assets (human, social, natural, physical and financial capitals) provide a more complete picture of household livelihood resilience and business viability than if focus would only be on impact indicators related to e.g. employment and income.

In addition, the 5Capitals approach provides a useful framework for understanding the role of market, political and institutional factors in facilitating or hindering favourable outcomes. The methodological framework underlying 5Capitals thereby helps to separate changes caused by interactions and interventions in value chains from those induced by the overall context. 5Capitals’ strong focus on farmers and farmer-linked enterprises together with its broader development and context perspective makes it a particular useful approach for outcome/impact assessment of VCD interventions.

The overall feature of the 5Capital approach and the related asset indicator framework is presented below.

Key household and business asset indicators for VCD impact assessment
  Household asset indicators / Business asset indicators
Natural Capital Stock of environmentally provided assets, including soil, water, stock of plants or animals
Human Capital Capacities and skills, formal education, nutritional and health status, business management and technical capacities and skills
Social Capital Rules, norms, obligations and trust embedded in social relations, structures or arrangements that enable those who share it to achieve goals they could not achieve individually
Physical Capital Tools, equipment, machinery, buildings, productive resources
Financial Capital Cash, savings, equity, credit and other financial resources

In addition to these 5 Capital areas, the framework also focuses on the “Enabling Conditions” (e.g. access to public infrastructure, services and government programmes).

The evaluation team will be using the 5Capitals Approach as an overall framework for this VCD evaluation.

3.2 Evaluation approach and methodology

The data used in the analysis of the F&B Programme interventions were collected through a quantitative survey, qualitative focus group discussions (FGDs) and semi-structured interviews. The strength of combining these types of data is the ability to validate the findings of either method and to expand the understanding of the study tasks. In addition, combining different data types allows eliciting causation between variables and improving confidence in the results.

Qualitative and quantitative phases of data collection were mixed. The first semi-structured interviews with value chain facilitators, programme staff and grant beneficiaries were conducted prior to the survey and they were crucial for formulating survey questions. The qualitative information from this stage also served to understand the positioning of the F&B sector within a broader cultural, economic and political context. These interviews comprised two cold store operators, two farmers, four programme officers and one expert for F&B production involved in lectures and training sessions with producers. The survey was followed by FGDs with a number of grant beneficiaries and non-beneficiaries to further the understanding of the programme effects.

The Survey

The quantitative survey comprised 435 respondents from the F&B sector in southern Serbia. The survey took place in August and September 2015 in five districts in the south of Serbia: Jablanica, Nisava, Toplica, Pcinja and Pirot. The survey was conducted in Serbian language by eight enumerators who even though having previous survey experience underwent a two-day training prior to the survey.

The survey sample was created in two stages. The first stage included stratified random sampling of grant beneficiaries from the list of grant applicants obtained from the F&B Programme office. The list of grant beneficiaries obtained from the F&B Programme office contained names of 851 applicants that had been approved for grants and planting material in calls for applications 1, 2, 3 and 4. A closer inspection of the list revealed that some applicants still have not received the funds in which case they were replaced in the sample by a randomly drawn applicant from the same district who has in fact received the funds.

The sample was stratified by district with 30% of grant beneficiaries from each district apart from district Pirot where due to a very small number of grant beneficiaries, all of them were added to the sample. The targeted number of grant beneficiaries for the survey was 200 and the sample included 70 names more to account for possible non-responses. The replacements were made in around 10% of the cases either due to a non-response or if it was established that a respondent still had not received the funds. The final sample comprises 193 grant beneficiaries.

The second sampling stage involved identifying a comparison group. The list of all F&B producers in Serbia could not be obtained from the Statistical Office of the Republic of Serbia, as this would violate confidentiality regulation. Then, the list of producers that have not applied for the grant from survey districts was made by eight consultants that have worked with the F&B Programme on providing information about the programme and helping with grant applications. Some of the consultants also work for local municipality and have access to lists of registered producers. These eight lists were compiled and a random sample of non-beneficiaries was drawn from the final list[5]. The list of non-beneficiaries also includes producers with unsuccessful applications and applicants who still have not received funds. The survey sample comprises 242 non-beneficiaries of which 64 have applied but have not yet received the grant, as shown in Table 1.

Table 1: Survey sample grant beneficiaries and non-beneficiaries
  Grant beneficiary Total
Grant approved No Yes  
No 178 0 178
Yes 64 (grant not yet disbursed) 193 257
Total 242 193 435

The largest share of the sample comprises household farms (427 respondents). All enterprises in the sample are registered as private limited company. The sample also comprises one co-operative. Compared to the list of grant applicants, the sample has not entirely preserved the composition of the main legal ownership forms, as shown in Table 2. The main difference is the lower number of enterprises.

Table 2: Composition of the sample and the list of approved applicants by legal ownership form
Legal ownership form List of grant beneficiaries Sample
  Number (%) Number (%)
Household farm 810 95.2 427 98.2
Limited liability company 28 3.3 7 1.6
Sole proprietor company 9 1.0 0 0.0
Co-operative 4 0.5 1 0.2
Total 851 100 435 100

Apart from asking about the participation in the F&B Programme, the survey questionnaire asked about production, costs, assets, inputs, marketing, labour, land, finance, income sources and membership in farmer groups or associations in 2015 and 2012. The questionnaire also included several qualitative questions in the form of open-ended and preference rating scales questions. These were used to complement the quantitative questions and numerical findings. In formulating questions, care was taken to adopt the concepts and measures to local conditions.

Limitations

The potential bias introduced by the sampling and survey method could come from four sources. First, the reliance on consultants to identify producers for the comparison group was unavoidable, as the official list of F&B producers could not be obtained. Relying on these sources could lead to systematic exclusion of certain types of producers from the survey, e.g. of remote, smaller or poorer ones. To address this concern of excluding the remote respondents, an attempt was made to create a comparison group that would include approximately the same number of respondents per village which is the lowest administrative unit in Serbia. Table 3 shows a rather uniform distribution of respondents by district and grant holding status. The exceptions appear to be Toplica and Pirot where higher number of applicants have applied but still have not received the grant so they were reclassified as a comparison group. The effect of excluding smaller and poorer producers from the comparison group could be that the size of the programme effect would be underestimated and results therefore could be interpreted as a lower bound of the programme impact.

Table 3: Distribution of grant holders by district
  All Grant beneficiaries Grant non-beneficiaries
  Number Percent Number Percent Number Percent
Jablanica 81 18.62 40 20.73 41 16.94
Toplica 193 44.37 82 42.49 111 45.87
Nisava 78 17.93 42 21.76 36 14.88
Pirot 37 8.51 8 4.15 29 11.98
Pcinja 46 10.57 21 10.88 25 10.33
Total 435 100.00 193 100.00 242 100.00

The second form of bias that could negatively affect the reliability of results could come from the use of recall data. To address this concern, the recall information for 2012 was verified against the business plans that the grant applicants made but this was not possible for the comparison group. In addition, the number of recall questions was not large and it concerned aspects of production that producers are quite likely to know, such as the size of landholding or seasonal labour salary. Even though the size of the recall period is reasonable, errors in the recall may bias results by understating the 2012 values, due to which the situation in 2015 and the programme impact may be overstated. This is a concern if only the group of grant users suffers from the recall problem, but the sample also contains a comparison group with likely the same direction of recall bias, so the recall problems may not affect the results.

The third form of bias may come from the difficulties in eliciting responses to sensitive questions such as income level. This bias is likely downward oriented so the end results could possibly have lower values than in an ideal setting. It is, however, not possible to estimate the magnitude of such bias in the context of this study.

The fourth bias could come from systematic differences among beneficiaries and non-beneficiaries participating in the survey. There is a risk that that those who have participated in the F&B Programme may differ systematically from those that did not participate in aspects that are not directly related to their social-economic profile such as their motivation/attitude, connections, network. If more talented individuals have self-selected into the programme, this would be overestimating the impact of the programme. But by how much we are not able to measure. In the survey design, the evaluation team has tried to mitigate this risk by including a large number of producers with unsuccessful applications and applicants who still have not received funds (see above).

Focus group discussions and key stakeholder meetings

FDGs were organised as follow-up to the quantitative field survey with beneficiaries and non-beneficiaries of the F&B Programme investment grant scheme. The FGDs were organised in 4 different municipalities in south Serbia: Bojnik, Vladicin Han, Aleksinac and Blace. The number of participants in each FGD varied between five and nine persons and the length of each FGD was approximately 1½ - 2 hours. The FGD participants were selected randomly among the survey respondents from the area.

In addition to the FGDs, visits were made to processing and cold storage centres within the region and interviews were conducted with the business owners. In total, eight FGDs with beneficiaries and non-beneficiaries and eight visits to processing and cold storage centres were conducted.

Finally, interviews and meetings were conducted with key stakeholders and resource persons, including management and staff from the F&B Programme, representatives from the GoS (Ministry of Agriculture (MoA)) and Directorate for Agrarian Payment (DAP)) as well as from the Danish Ministry of Foreign Affairs.

Other activities

In addition to the fieldwork activities, the evaluation team has reviewed relevant documentation, statistics and data.

The evaluation team was invited to make a presentation of the preliminary survey findings at the Steering Committee Meeting in Nis on 18 September. This allowed for useful discussions and interaction with the committee members and for feedback from key stakeholders on the preliminary evaluation findings.


[4] Tropical Agricultural Research and Higher Education Center, CATIE, 2012.

[5] As the lists for the comparison group were made in different ways (some from personal knowledge of producers in the area, some from official lists), it is not possible to assess the representativeness of the list.

Top

This page forms part of the publication "Evaluation of Danida Support to Value Chain Development – Serbia Country Study" as chapter 3 of 8.
Version no. 1.0, 2016-07-05
Publication may be found at the address http://www.netpublikationer.dk/um/evaluation_value_chain_development_serbia/index.html