Go to content

Annex B: The Methodology of the Evaluation

1. Introduction

The Consultant’s outline of the methodology for the Evaluation of the Danida Business-to-Business (B2B) Programme 2006-2011 was first presented in the Technical Proposal (25.10.2013). The methodology evolved further during the Inception Phase – November 2013 to January 2014 and the preliminary version was presented in the Inception Report (31.12.2013). The methodology was tested first during the visit to Uganda in February 2014 and underwent consequently some refinements, and later on during the visit to Bangladesh in March 2014 – resulting in a consolidated version of the methodology. EVAL submitted an Update of the Evaluation Methodology to the Evaluation Reference Group (ERG) on 2 May 2014. The comments received from ERG and EVAL were incorporated. The final version was submitted on 12 May 2014. Since then, the random sample survey has been largely completed and the Evaluation has now moved into its final phase – leading to some further refinements of the Evaluation Methodology. This note presents the updated version of the Evaluation methodology as at 23 June 2014.

2. Theory of Change

The Evaluation Team’s view is that the ‘reconstructed’ Theory of Change as presented in the Inception Report still remains valid – representing the ‘best case scenario’. The Theory of Change was reconstructed based on the B2B Programme’s objectives. Given the level of investments in each of the B2B Programme countries, the macro-level impact on poverty reduction and in promoting economic growth and social development has been minimal and was thus left out of the analysis. The thrust of the Theory of Change as regards long-term outcomes and impacts is therefore on the local level, i.e. in those urban and rural communities in which the B2B partnerships have been established and functioned; and the extent to which the B2B partnerships have had systemic impact on the business sector in which they operate, such as technology transfer and market development. ‘Market access’ turned out to be an essential short-term and long-term outcome, and has subsequently been added in the Theory of Change diagram. A further elaboration of the Theory of Change is presented in Annex D.

The application of the ‘Context-Mechanism-Outcome’ (CMO) approach in relation to the B2B Programme has been assessed.[69] There is great variation in contexts between the B2B Programme countries and within the countries depending on location and sector. Based on the experience from Uganda and Bangladesh, the number of contextual parameters has been reduced compared to the Inception Report. The revised “Contextual parameters and assumptions” is attached as Annex B.1. The contextual factors are used in the country studies (e.g. size of companies, previous international experience, type of collaboration, etc.) in relation to outcomes. The same was the case for the random sample of collaborations. The contextual factors are especially relevant for assessing a number of judgement criteria in the Evaluation Matrix (attached as Annex C to the Synthesis Report) and for considering how they influence the results chain:

  1. Theory of Change Outputs: The efficiency for promoting partnerships from the pilot to the project phase (Judgement Criterion (JC) 2.2), and how the contextual factors may have influenced the design of the pilot phase (JC 2.3) and the project phase (JC 2.4) to make the collaborations compatible with the local context;
     
  2. Theory of Change Short-term Outcomes: The influence on the choice of technology and associated knowledge (JC 4.1), and how this led to improving the local companies’ performance (JC 4.2); and
     
  3. Theory of Change Long-term Outcomes: How they may influence the long-term results and if possible changes in the composition of the contextual factors over time have had an impact – positive or negative – on companies’ business opportunities (JC 6.1).

Country contextual parameters will be used on the entire programme portfolio when this is possible, i.e. number of projects, ratio of projects to total collaborations, grants and disbursements. It has been discussed between the Evaluation Team and EVAL to include sector as a contextual factor at the portfolio level analysis. In the portfolio documentation received initially (B2B database), sector was not listed as a parameter for the partnerships, but DGG has since come up with a list of B2B/DBP projects with an indication of sectors. The Evaluation Team has made an effort to use the information on sectors in the portfolio analysis. Information on sectors is included in JC 2.3, 2.4, 3.3, and 6.1 in the Evaluation Matrix.

The mechanisms that convert B2B interventions to outcomes have: 1) a human dimension where the partners decide and agree on the sequence of interventions leading to the pilot and project phases subject to their respective interests, and ultimately to the continuation or discontinuation of the collaboration; and 2) a market dimension, which the partners can only influence to a very limited extent. Whereas the human dimension has a certain measure of predictability in relation to the prevailing context, the market dimension is much less predictable and market changes may influence the outcomes either positively or negatively – and may in the worst cases result in the partnership to fail. There may be many other reasons for a partnership to fail, which are not due to the market situation, e.g. not understanding the context appropriately, taking unnecessary risk, inappropriate preparation, etc. The Uganda and Bangladesh country studies also showed examples of Danish companies using the B2B as a ‘survival mechanism’ – in most such cases the partnership failed. The team has assessed what motives drove the partners to engage in international partnerships and how the B2B Programme was seen to support such motives. Furthermore, it will be assessed what type of benefits the partners expected to achieve – either immediate, short term, or long term – and how the motives and corresponding choices influenced the commercial viability of the local company as well as the Danish company. The motivation and risk elements of the partnerships will constitute an important part of the Evaluation.

A results framework “Ratings and criteria” has been developed for the partnerships’ performance in conjunction with the country studies, please see Annex B.2. Each of the collaboration (pilot and project) that has been assessed will be rated in accordance with the results framework – provided that adequate information is available – obtained through project documentation and interviews. The project ratings will be filled into an Excel sheet by country. The results framework will provide an overview of the per formance by project in the sample, which can be extrapolated with some degree of probability by country and for the entire portfolio. This overview will provide the basis for relating the B2B Programme’s performance to the prevailing contextual factors by locality and country as they evolved over the time span of the evaluation period.

3. The Evaluation Matrix

The Evaluation Matrix as first presented in the Inception Report was prepared based on the 20 Evaluation Questions (EQs) presented in the Terms of Reference (ToR). The number of EQs in the Evaluation Matrix was reduced to nine by transforming 19 of the ToR’s EQs to Judgement Criteria (JC). The main overall EQ from the ToR corresponds to EQ8 in Table 3.1 below. A 10th EQ dealing with ‘Value Added’ was added, which is often is used in EU evaluations. However, it was found that there is limited coordination among the development partners engaged in this kind of assistance and thus limited value added. In consequence, EQ10 has been taken out. EQ1 to EQ5 are related to the project period, whereas EQ6-EQ9 are related to the post-project period, i.e. when project funding from the B2B Programme has ceased.

The country visits to Uganda and Bangladesh gave no rise to change of the overall structure of the EQs, see Table 3.1. The structure of the JCs also remains unchanged, except for: 1) JC 3.1 where the indicator has been changed to target B2B stakeholders more broadly instead of companies only; and 2) a new JC 6.4 has been added, “Unintended negative effects from implementing the B2B Programme”. A number of minor changes and editorial revisions have been made in the JCs’ indicators and means and source of verification. Notes have been added for the JC 6.1, 6.3 and 8.2 indicators concerning data availability. The E-survey has been maintained as a means and source of information in the Evaluation Matrix, and will be discussed further in Section 8 below.

Table 1: B2B Evaluation Questions by criteria
Criteria Evaluation Question
Relevance EQ1: To what extent has the B2B Programme been consistent with private sector development requirements in the partner countries and with Danida’s private sector policies?
Efficiency EQ2: How efficiently were the B2B Programme instruments used in creating partnerships and how did external factors influence the results?
EQ3: To what extent did the management of the B2B Programme provide an efficient framework for: delivery of services to companies, utilisation of resources, and accounting for results?
Effectiveness EQ4: How has the B2B Programme led to knowledge and technology transfer in the local partner company and what were the resulting short-term outcomes?
EQ5: How has the B2B Programme led to improved conditions for employees and the wider population and what were the resulting short-term outcomes?
Impact EQ6: What long-term effects have the B2B Programme had on the local partner companies and specific business sectors, and how have these influenced local communities, and the national enabling environment?
EQ7: What long-term effects have the B2B Programme had on the Danish partner companies?
EQ8: To what extent and how has the B2B Programme contributed to poverty reduction by creating growth and employment in Danida partner countries?
Sustainability EQ9: To what extent have the benefits derived from the B2B Programme continued after project completion?

4. Country Studies

As part of the Preparatory Phase for the country visits to Uganda and Bangladesh, an Assessment Sheet was developed with the purpose of: 1) stating the basic facts for each partnership (pilot and project) based on available project data; 2) recording the outcome of interviews with the Danish and local partner; and 3) assessing the partnership’s performance.

The key uncertainty all through the initial phases has been availability and trustworthiness of data. The availability of data would again determine the factors and the causes that the Evaluation could explore with any likely degree of significance in findings. Uganda was an excellent test in this regard, illustrating a number of important methodological challenges. Most of these were also present in Bangladesh:

  • Basic data about partnerships are often lacking, or of poor quality. This includes information about financial results of the cooperation, investments, internal operations and even employment. After the Application process, all data provided in project reporting is based on self-reporting. One can trust or not trust such data, but as the B2B process generally contains few external checkpoints, an evaluator must show great caution in handling this information. It is clear from indicator reports generated from MFA’s PDB that the data contains substantial errors (data that missing, yearly figures that have been incorrectly aggregated, etc.). The Evaluation Team will include a comparison of the results from the indicator system with an assessment of results by the team based on the interviews conducted with project partners. This will provide the evidence base for a final ‘judgement’ of the B2B indicator system.
     
  • Reporting from B2B projects – in the form of Quarterly and Final project reports – is scattered and in many cases only available through extended excavation, taking substantial time. This is partly due to the fact that the embassies are only requested to save such reports in hardcopy at the archives at the embassies. In a number of cases, in particular those that have gone wrong, there is hardly any progress reporting from project operations at all, except a few lines in the embassy’s Project Completion Report (PCR).
     
  • Indeed, data regarding investments including those in internal and external environment cannot be used, as they are not only highly susceptible to manipulation, they also suffer from being interpreted differently from case to case. CSR activities are another area where it is difficult to get hard third-party evidence apart from what the team can observe during a visit to the production site.
     
  • Two sets of data that are somewhat more reliable are turnover[70] and employment. Both can be externally verified, which the team did on some occasions through annual accounts and manpower lists. Of course, the challenge of how much of this can be attributed to B2B support remains an issue, but these two figures at least give “ballpark” numbers for operational status of the B2B collaboration.
     
  • Information is triangulated from four different sources, namely the Danish partner, the local partner, the embassy, and from documents. The views from these sources differ considerably in some cases. The Danish companies and the embassy documents tend to be more positive to the outcome of the collaborations than the local partners’ verdict. This was the case both in Uganda and in Bangladesh.
     
  • Finally, diversity in the portfolio is such that “one will be hard pressed to find any project with similar characteristics, similar backgrounds, same capacities and working in the same markets. All have their own story to tell, and the outcomes in terms of both business and development impacts are equally diverse” (Draft Uganda Country Report). The factors that seem to determine success or failure of collaborations are many and varied, often specific for each case. While some are related to ‘soft’, psychological factors such as trust and individual entrepreneurship drive, the destiny of other collaborations are better explained by market conditions and market developments.

In order to make the Bangladesh data sample similar to the Uganda study, the team decided to include all the pilot phase collaboration in addition to those that went to the project phase. As in Uganda, the team was not able to meet with representatives from all the collaborations. Some partners were travelling and unavailable for meetings, some refused to meet the team, and some companies had simply disappeared. Getting hold of Danish partners proved as complicated as for the Ugandan and Bangladeshi partners. Therefore it has not been possible to make a full triangulate of all projects with all four data sources as listed above. However, for most B2B projects the team has at least information from two and often three sources. In those cases where information is highly doubtful, or not properly quality checked, the team has refrained from giving ratings.

In addition to meeting B2B partners, the team met with other donors, with sector organisations, with other private companies, and with Funds involved in social business investment. Through these interviews, the team has among other issues tried to establish ‘the counterfactual’ in Uganda and Bangladesh, i.e. what would have happened without the support of the programme.

The lack of well-defined hard data implies that the Evaluation to a substantial degree will have to use qualitative ratings and conclusions. Each partnership is thus rated across a number of evaluation criteria, in addition to being categorised in relation to basic features of each company. A third set of factors is the B2B particulars, as for instance size of B2B support, years of B2B implementation, and which phase each collaboration went through. All these ratings and criteria are entered into an Excel results framework by country.

Power Point Presentation/debriefing note were prepared at the end of the country visits, which formed the basis for the debriefing meetings with the embassy and EVAL – and in Uganda with some of the B2B partners as well. Draft Country Reports for Uganda and Bangladesh have been submitted. The Table of Contents for the country reports is shown in the box below.

Table of Contents for the Country Reports

Summary: Results and Conclusions

Introduction and Background

Methodology

The Investment and Business Environment

The B2B Portfolio in Brief

Key Results

Analysis towards the DAC Criteria

5. The Random Sample

The 20% random sample consisting of 87 collaborations (seven of which were chosen from the Uganda portfolio and seven from the Bangladesh portfolio, which are included in the country studies) has been chosen from the total population. The random sample consists of 36 pilots and 49 projects. No collaborations were drawn from three of the 17 B2B Programme countries (the random process did not select projects from Indonesia, Nepal and Burkina Faso).

Out of the 87 collaborations, the team succeeded in compiling adequate information for 80 collaborations through accessing information from the ExtraNet complemented with additional information from embassies and partners interviewed. Three of the selected collaborations turned out to be PSD projects and inadequate information was available for four projects. Collection of information from the ExtraNet encountered similar problems as for the Uganda and Bangladesh collaborations due to non-availability of the essential documents – in some cases there are few documents related to the implementation of the projects (quarterly reports or embassy PCRs) – the intentions of the projects can only be assessed through applications documents. The outcomes and results were established through interviews with Danish and local partners. Once the overview was attained for the respective partnerships in the Programme countries, an interview was undertaken with the embassy’s B2B Programme Coordinator in order to triangulate information, to address specific issues arising from the assessment of the random sample collaborations, as well as general issues of relevance for the country in question.

The same approach for assessing collaborations as applied for the Uganda and Bangladesh portfolios was used for the random sample. The Assessment Sheet was slightly modified based on the experience from the country studies.

The following approach for contacting the partners was adopted:

  1. An email was written – using the mail address obtained from the application/EVAL – with an introduction letter from Danida and brief interview guide attached. If the email bounces, Google will be used to find an alternative email address;
     
  2. If no reply has been received within 3-4 days, a follow-up email will be sent;
     
  3. If still no reply is received, a call will be made to the company – using the number from the application or from a Google search.

Once contact was established with a relevant person with knowledge about the B2B collaboration, the interview was conducted and the gathered information filled into the Assessment Sheet. An interview would not be pursued if no relevant contact person was identified after the above three mentioned attempts. The assessment of the random sample will be done by country.

The outcome of the collaboration assessments will be entered into the results framework by country (ref. Section 2) for those collaborations for which adequate information has been gathered from project documents and interviews – the same results framework as applied for Uganda and Bangladesh country studies was used for the random sample. A statistical analysis was undertaken with a view to generalising the performance for the entire population for selected type of results, e.g. turnover, employment, investment, etc. Hence, the evaluation will estimate the aggregated results at programme level through the combination of Uganda and Bangladesh case studies and the random sample.

6. The Success Cases

The industry associations, the embassies, and DGG have selected a number of ‘success cases’, which from their perspective demonstrate good examples of well performing partnerships. The purposes of including the success cases in the Evaluation were to: 1) determine the scope of the development effects; 2) analyse the constellation of the contextual factors; and 3) see how the cases relate to identified important issues, see Section 7.

Regarding the first purpose, selected outcomes of the successful projects have been aggregated, which – together with the other case material – provided an indication of what the B2B Programme has ‘at least’ achieved in terms of outcomes. This was only possible in relation to the outcome indicators: number of jobs and company turnover – and only for those cases where based data was considered reliable. Regarding the second purpose, the team has looked into which contextual factors that in particular prompted the partnerships to be successes. The team has identified success cases as part of the country studies and the random sample, and has correspondingly assessed the contextual factors that contributed to the success. A comparative analysis of the team’s criteria for determining a success case and the externally selected success cases has been made. Some of the externally selected success cases were included in the country study samples and the random sample and will thus be subject to the general evaluation assessment, whereas those not included will not be part of the overall assessment, as they may distort the evaluation in a positive direction.

7. The Issue Paper

The Issue Paper was prepared following the completion of the country reports from Uganda and Bangladesh. The issues were drawn from the country context and assessment of the case countries’ portfolio. The issues were arranged corresponding to the Evaluation Matrix, i.e. EQs and JCs and thus be compatible with the Theory of Change. The country reports have an elaborated Chapter 7 “Analysis on the DAC criteria” with summaries for each EQ, which were the main source for formulating the issues.

The first draft of the Issue Paper was prepared in parallel with the assessment of the random sample collaborations. The Issue Paper was submitted to the ERG for comments on 27 May and a meeting was held on 12 June 2014. The paper will be updated if the assessment adds further issues or rectify those already identified. The Issue Paper will also relate to those changes that have been introduced in the Danida Business Partnership (DBP) Programme that replaced the B2B Programme. The Issue Paper has fed into the E-Survey, the focus group discussion, and the Synthesis Report.

8. E-Survey

The E-Survey covers all collaborations in the portfolio and addresses a few essential issues. The issues for the E-survey were relatively few and were selected amongst the most essential issues from the Issue Paper. The team had some doubts about the response rate due to the difficulty of identifying partners that have concrete knowledge of the partnership and their willingness to respond. Those partners that have been interviewed might generally be positive and are thus likely to respond. The E-Survey was sent to 745 Danish and local partners on 2 June 2014 and the response rate has so far been 22%. The E-survey provided an opportunity to have the views of partners that have not been approached previously. The design of the E-survey was made in consultation with EVAL at a time when the assessment of the random sample was near completion.

9. Focus Group Discussion

A focus group discussion was held on 28 May 2014 based on selected issues from the consolidated Issue Paper in order to validate the team’s findings. It was agreed to deal with factors that contributed to successful implementation of the partnerships and generated sustainable impacts. DGG, the embassies and the Danish business associations have selected a number of ‘success cases’ (ref. Section 6). It was decided to invite the Danish partners that had been or still were involved in the selected success cases jointly with representatives from DGG, EVAL, DI and the Danish Federation of SMEs. The discussion suffered from the absence of local partners and their viewpoints. The focus group discussion was arranged in consultation with EVAL. Bilateral interviews will be conducted with staff from the industrial associations and Danida to follow-up on the core issue raised.


[69] Reference is made to the Terms of Reference, Section 5 ”Approach and methodology”.

[70] Turnover is for most companies a less contentious figure than “profit”. Local companies would only indicate very roughly whether they earned money or not, some refusing outright by claiming profitability figures to be “business secrets”.

Top

This page forms part of the publication "Evaluation of Danida Business-to-Business Programme 2006-2011 – Evaluation 2014.05" as annex b
Version no. 1.0, 2014-11-14
Publication may be found at the address http://www.netpublikationer.dk/um/14_danida_btb_programme_2006_2011/index.html