Go to content

Annex C: Evaluation Matrix

Introduction

The Terms of Reference for the B2B Evaluation specify 20 evaluation questions (EQs) under the headings of the OECD/DAC evaluation criteria. These questions were transformed into nine broad EQs and a number of judgement criteria (JC) related to each EQ – ensuring that the intended scope of the Evaluation was maintained.

A 10th EQ concerning value added (often used in EU evaluations to assess the synergy of member states’ interventions) was initially included. However, it was found that there is no coordination among the development partners engaged in this kind of assistance and thus limited value added. Consequently, the 10th EQ was abandoned. The resulting nine questions are shown in Table 1 below. EQ1 to EQ5 are related to the project period, whereas EQ6 to EQ9 are related to the post-project period, i.e. when project funding from the B2B Programme has ceased. The Evaluation Matrix was first presented in the Inception Report and has since undergone some refinements as the Evaluation progressed.

Table 1: B2B Evaluation Questions by criteria

Criteria

Evaluation Question

Relevance

EQ1: To what extent has the B2B Programme been consistent with private sector development requirements in the partner countries and with Danida’s private sector policies?

Efficiency

EQ2: How efficiently were the B2B Programme instruments used in creating partnerships and how did external factors influence the results?

EQ3: To what extent did the management of the B2B Programme provide an efficient framework for: delivery of services to companies, utilisation of resources, and accounting for results?

Effectiveness

EQ4: How has the B2B Programme led to knowledge and technology transfer in the local partner company and what were the resulting short-term outcomes?

EQ5: How has the B2B Programme led to improved conditions for employees and the wider population and what were the resulting short-term outcomes?

Impact

EQ6: What long-term effects have the B2B Programme had on the local partner companies and specific business sectors, and how have these influenced local communities, and the national enabling environment?

EQ7: What long-term effects have the B2B Programme had on the Danish partner companies?

EQ8: To what extent and how has the B2B Programme contributed to poverty reduction by creating growth and employment in Danida partner countries?

Sustainability

EQ9: To what extent have the benefits derived from the B2B Programme continued after project completion?

Evaluation
Criteria

Judgement
Criteria

Indicators

Means and Source
of Verification

EQ1: To what extent has the B2B Programme been consistent with private sector development requirements in the partner countries and with Danida’s private sector policies?

Relevance

1.1 The extent to whicht he B2B Programme has been relevant for addressing constraints of private sector companies in the partner countries.

Defined binding constraints in the eligible countries. (Specifically for BGD and UGA).

Matching B2B investment profiles against these constraints.

Hypothesis: B2B Programme is too marginal to address such constraints, except in very specific circumstances when projects have systemic impact.

Business environment reports for BGD and UGA.

B2B Country Reviews.

B2B project reports by Programme countries.

Interviews with partner companies in UGA and BGD.

Interviews with informed observers such as WB office, IFC, chambers, etc.

 

1.2 The extent to which the B2B Programme stimulated the creation of international partnerships that would not otherwise have occurred.

Degree of additionality of the B2B partnerships – including the additionality of the technology provided and the knowledge attained compared to the local context.

Scope and substance of the B2B partners’ strategic alliances. Motives for engaging in the B2B Programme.

Hypothesis: B2B stimulated partnerships in countries that would otherwise not have been accessed by the Danish partner.

Interviews with partner companies.

E-survey.

 

1.3 The appropriateness of the B2B Programme for – through partnerships between Danish and partner country companies– promoting the overall objectives of Danidasupport in relation to private sector development, i.e. poverty reduction through private sector growth and employment.

The degree to which the intended development impacts are incorporated in the application documents.

The degree to which the intended impacts have been attained, e.g. direct and indirect employment generated and level of increase of employees’ income and welfare.

Note: Since the B2B Evaluation is a post-evaluation; the appropriateness of the Programme concept will be compared to the actual development effects as dealt with in EQ8 and EQ9.

Interviews with business sector officers at Danish embassies in programme countries.

Review of: B2B applications, project completion reports and annual reports after the project phase. Interviews with partner companies.

EQ2: How efficiently were the B2B Programme instruments used in creating partnerships and how did external factors influence the results?

Efficiency

2.1 The efficiency in using B2B Programme instruments to establish the initial contact (Contact phase) between companies in Denmark and in the partner countries (“matchmaking”).

The efficiency in promoting B2B partnerships from the contact phase to the pilot phase.

Scope of participation of Danish companies in BGD and UGA; yearly basis.

Cost for the Contact phase for Danida, direct and indirect.

Rate of Contact participants that go to the Pilot phase.

Data from embassies in BGD and UGA.

Cost data from Danida/Embassies (BGD &UGA).

Specific analysis of companies participating in Contact phase in Bangladesh and Uganda.

Interviews with companies in BGD, UGA and Denmark.

 

2.2 The efficiency in promoting B2B partnerships from the pilot phase into the project phase.

Did any specific factors (for instance company type, motivation, financial incentives, power relationship between partners, type of partnership project, or other) systematically come into play in this process?

To what extent did the prevailing contextual factors influence the transition from pilot to the project phase?

Rate of partners moving from the pilot to the project phase.

Reasons for differences in programme countries and systematic assessment against defined context parameters.

Company business motivation: Markets, efficiency/ resources/ costs, competitive factors, or business environment.

Scope and substance of the strategic alliances and level of additionality. Business alliance structure: JVs, buy/ sell, and agent/ franchise/ licence, TA.

Hypothesis: Efficiency dependent on type of Danish companies (size, sector, experience, intentions, motivations, robustness, etc.) and on embassy competence/ attitudes.

Portfolio analysis.

Review of pilot partnership reports/ completion report.

In-depth interviews companies in BGD, UGA and Denmark.

Sample analysis from companies in overall portfolio and interviews in person.

Questions in E-survey.

 

2.3 The partners’ efficiency in adapting to external and internal factors in the design of the Pilot phase.

The response should be based on a comparative analysis of programme achievements between B2B projects with differences in external factors (local economic factors, enabling environment, sector, etc.).

Appropriateness of identifying relevant contextual factors by country and local area.

The degree to which contextual factors have appropriately been applied in the design of the Pilot phase.

The strategic choices by the embassies in prioritising certain sectors and excluding others.

Assessment of prevailing contextual factors by country.

Assessment of the quality of the CBC (Concept for business cooperation) for capturing the local business environment and in providing the basis for results-based monitoring.

Interviews with partnership companies.

E-survey.

 

2.4 The efficiency in adapting to external and internal factors in the design of the Project phase.

The response should be based on a comparative analysis of the programme achievements between B2B projects with differences in external factors influencing the programme (local economic factors, enabling environment, sector, etc.).

Appropriateness of identifying relevant contextual factors by country and local area.

The degree to which contextual factors have appropriately been applied in the design of the Project phase.

Assessment of prevailing contextual factor by country.

Interviews with partnership companies.

Questions in E-survey.

EQ3: To what extent did the management of the B2B Programme provide an efficient framework for: delivery of services to companies, utilisation of resources, and accounting for results?

Efficiency

3.1 The extent to which the administration and management of the B2B Programme was well-balanced between ensuring control of public funds, providing easy access for private companies and providing the framework for an efficient use of Danida/ embassy administrative resources.

Stakeholders’ perception of Danida HQ’s and embassies’ facilitating role in providing appropriate facilitating frameworks.

The quality and appropriateness of the application system.

The quality of the appraisal process.

The quality of the monitoring of the programme.

Analysis of embassy systems in BGD and UGA.

Analysis of Danida’s HQ monitoring system.

Analysis of the documentation system in B2B from application to PCR.

Interviews with B2B partners.

Interviews with Danish business associations.

Interviews with GRV and embassy B2B/ DGG coordinators.

E-survey.

 

3.2 The extent to which the documentation and monitoring system of the B2B Programme, and the way it has been administered, was a useful basis for assessing progress and documenting results at individual project level, country level and programme level.

Quality of the results reporting (quarterly reports; completion reports).

Degree of coherence of results reporting with reality.

Use of system – the number of reports that has been made based on the system.

Hypothesis: Great variation between countries due to interest and competence of embassies.
Too strong focus on the application process with weak progress reporting and monitoring.

Pilot and project progress reporting.

Embassies use of progress reporting and establishing the indicator reports.

Embassies’ annual reports.

TSA’s Country Review Reports.

Concepts for Business Development.

Interviews companies BGD and UGA and sample of portfolio projects.

Interviews embassy staff and Danida staff.

 

3.3 The circumstances under which the B2B Programme provide the best results in terms of achieving its objectives in relation to human and financial inputs (Programme costs/ Value for money).

Share of collaborations, which result in a viable post-project collaboration– and the underlying factors for this.

Cost of the collaboration (Contact, Pilot, and Project) in relation to achieved results at the end of the project period.

NB: Some projects that continue after the project period may have achieved additional results.

Comparative assessment of B2B projects with differencesin external factors influencing the performance (local economic factors, enabling environment, sector, etc.).

Assessment of the VfM for BGD and UGA

Comparative assessment of VfM in Programme countries in relation to number of pilots and projects.

EQ4: How has the B2B Programme led to knowledge and technology transfer in the local partner company and what were the resulting short-term outcomes?

Effectiveness

4.1 The extent to which the B2B Programme support led to adoption of new knowledge and technology in the partner company– and the particular circumstances that facilitated this process.

To what extent did the prevailing contextual factors influence the adaption of new knowledge and technology?

Share of local partner companies that have acquired new technology and knowledge, including management practices – and how this has happened.

Process leading to upgrading of staff’s skills and competence.

Review of project files: BGD, UGA and random sample.

E-survey.

Interviews with partners and staff/ employees.

 

4.2 The extent to which the partnership, through the adoption of new knowledge or technology orotherwise, led to improved performance, increased employment and/or increased turnover of the local partner company.

To what extent did the prevailing contextual factors influence companies’ improved performance?

Short-term changes in companies’ productivity, production and marketing resulting in: increased turnover, profit, employment, and investment.

Performance of JVs or other forms of collaboration.

Review of project files: BGD, UGA and random sample.

E-survey.

Interview with partners.

Correlation analysis with selected contextual parameters.

EQ5: How has the B2B Programme led to improved conditions for employees and the wider population and what were the resulting short-term outcomes?

Effectiveness

5.1 The extent to which the support under the B2B Programme led to improved occupational health and safety conditions for employees.

Measures taken to improve OHS in the production chain.

Hypothesis: The improvements are related to the sector in which the B2B takes place.

Review of project files: BGD, UGA and random sample.

Physical inspection of the working environment in selected companies in BGD and UGA.

Interview with employees and other stakeholders in selected companies in BGD and UGA.

Interview withpartners.

E-survey.

 

5.2 The extent to which the support under the B2B Programme led to environmental improvements.

Improvement of the external environment through reduced pollution and waste stemming from adaptation of new technology and knowledge in the local partner company, e.g.:

  • Cleaner inputs
  • Efficient use of raw materials
  • Treatment of waste
  • Achievement of standards compared to local and national regulations.

Hypothesis: The improvements are related to the sector in which the B2B takes place.

Review of project files BGD, UGA and sample.

Physical inspection of the working environment in selected companies in BGD and UGA.

Interview with employees and other stakeholders in selected companies in BGD and UGA

Interview with partners.

E-survey.

 

5.3 The extent to which CSR interventions, and other measures to improve the general conditions – introduced as part of B2B partnerships – were effective for employees (internal) or the wider population (external).

Increased awareness of HIV/AIDS in the company and surrounding community.

Equal employment opportunities for women, men and youth.

Adherence to national laws and ILO conventions/ Decent Work Agenda
– promotion of workers’ rights and human rights.

Adherence to UN Compact
– promotion of sound business practices (including the fight against bribery and corruption).

Review of project files: BGD, UGA and random sample.

Interview with employees and other stakeholders in selected companies in BGD and UGA.

Interview with partners.

E-survey.

EQ6: What long-term effects have the B2B Programme had on the local partner companies and specific business sectors, and how have these influenced local communities, and the national enabling environment?

Impact

6.1 The long-termeffects of the B2B partnerships on the development of the local partner company (e.g. investment, turnover, income, employment, productivity, and competitiveness); and how country or company contextual factors systematically influenced the long-term effects.

Performance of the six measured indicators in the B2B:

  • Turnover
  • Investment in the collaboration
  • Male employment
  • Female employment
  • Environmental investment
  • CSR activities

Performance of JVs or other forms of collaboration in realisation of the intended benefits by sector.

Note: The degree of the reliability may vary between the indicators, but they will all be maintained in the Evaluation
– commenting on their re usefulness.

Review of project files:BGD, UGA and randomsample.Interview withpartners.E-survey.
 

6.2 Target groups benefitting from the programme and the degree to which the sewere the intended beneficiaries.

Has there been a counterproductive selection bias related to gender, population groups, geography, or other factors?

Target groups – partner company employees and non-company stakeholders
– benefitting positively and type of benefits;

Target groups – partner company employees and non-company stakeholders
– benefitting negatively and type of inconveniences/ disadvantages.

Review of project files: BGD, UGA and random sample.

Interviews with partners.

Interviews with employees and other stakeholders in BGD and UGA.

E-survey.

 

6.3 Discernible long-term effects beyond the local partnership company in the vicinity of the local company in terms of technology adoption, CSR, environmental aspects, occupational health and safety, or other.

Diffusion of knowledge and technology that have been adopted by non-participating companies.

Diffusion of OHS, CSRand environmental management practices to the wider community.

Spin-off effects in terms of market development or market creation.

Changes in industrial standards.

Note: The amount of information available may not be adequate for conducting a comprehensive assessment of the long-term effects beyond the local company.

Review of project files: BGD, UGA and random sample.

Interviews with B2B partners.

Interviews with embassy B2B/DBP coordinators.

E-survey.

Country analyses based on information from World Bank, IFC, WEF, etc. in UGA and BGD.

 

6.4 Unintended negative effects from implementing the B2B Programme.

Distortion of the local market conditions by favouring a single company.

High risk taking due to the high grant percentage leading to untimely dissolution of collaborations.

Review of project files: BGD, UGA and random sample.

Interviews with partners.

E-survey.

EQ7: What long-term effects have the B2B Programme had on the Danish partner companies?

Impact

7.1 The long-term effects of the B2B partnership on the Danish partner company in terms of other international strategic alliances, increased access to markets, improved competitiveness, or other.

To what extent did the prevailing contextual factors in Denmark influence the Danish companies’ interest for strategic alliances
– as offered by the B2B Programme?

Impact of other forms of internationalisations due to the B2B.

Market access.

Competitiveness.

Review of project files: BGD, UGA and random sample.

Interviews with partners.

Focus group discussions with Danish enterprises.

Interviews with Danish industrial organisations.

Correlation analysis between performance and selected contextual factors.

 

7.2 The Danish partner companies’ level of investments in international strategic alliances.

Investments in JVs and similar collaborations. Quantitative data of actual investments versus targets.

Review of project files: BGD, UGA and random sample.

Interviews with partners.

EQ8: To what extent and how has the B2B Programme contributed to poverty reduction by creating growth and employment in Danida partner countries?

Impact

8.1 The impact of the B2B Programme in terms of poverty reduction or sustained changes in livelihood for the people directly affected by the programme through employment, capacity development or CSR activities.

Change in number of employees, income levels, and welfare that can be attributed to the programme.

Degree of contribution to socio-economic development in local communities where B2B partnerships have been established.

Review of project files: BGD, UGA and random sample.

Interviews with partners.

E-survey.

 

8.2 The indirect impact of the B2B Programme on poverty reduction, increased employment and growth through market changes and effects on the wider economy.

Systemic effects:

  • Crowding in
  • Copying
  • Sector growth
  • Backward/forward linkage
  • Technology diffusion

Note: There may be a limited number of projects that have continued beyond the B2B project period for which it is possible to determine the contribution to the indirect impacts.

Review of project files: BGD, UGA and random sample.

Interviews with partners.

Interviews with embassy B2B/DBP coordinators.

Assessment of selected ‘success cases’.

E-survey.

Country analyses and sector analysis in BDG and UGA.

 

8.3 The impact of the B2B Programme on business sectors, the national enabling business environment, and economic growth.

Change in business sectors’ perception of new technology, OHS and CSR.

Interview with generaland sector-based business associations in BGD and UGA and other selected partner countries.

Interviews with non-B2B partners in BGD and UGA.

Interviews with local and national officials in BGD and UGA.

EQ9: To what extent have the benefits derived from the B2B Programme continued after project completion?
Sustainability

9.1 The extent to which partnerships have continued beyond the period supported by the B2B Programme, and the particular circumstances that has led to the continuation.

Rate of survival of JVs, TA, Buy/ Sell, and agent/ franchise/ licence.

Interviews with partners in BGD and UGA and sample of projects in the Portfolio.

E-survey.

Correlation analysis with selected contextual parameters.

 

9.2 The extent towhich there has been a transition of the partnership to other types of Danida support, e.g. from sector programmes, mixed credit (Danida Business Finance) or the Investment Fund for Developing Countries (IFU).

Number of projects with parallel or subsequent funding from IFU, level of investments.

Number of companies attempting such funding.

Interviews with companies in BGD and UGA and sample from the overall portfolio.

Interviews with IFU and Danida Business Finance.

 

9.3 The extent to which local partner companies have benefitted from the partnership after the partnership has been dissolved following the completion of the project phase.

The continued use of the technology introduced through the B2B partnership and attainment of similar types of benefits as derived from the support during the Project phase.

Interviews with former local partners that continue to operate the local company.

 

9.4 The extent to which Danish partner companies have benefitted from the partnership after the partnership has been dissolved following the completion of the project phase.

Increased exposure to the international market and cooperation with new partners in developing countries.

Reduction in cost of production due to off shoring or outsourcing.

Interviews with former Danish partners that continue to operate the Danish company.

Top

This page forms part of the publication "Evaluation of Danida Business-to-Business Programme 2006-2011 – Evaluation 2014.05" as annex c
Version no. 1.0, 2014-11-14
Publication may be found at the address http://www.netpublikationer.dk/um/14_danida_btb_programme_2006_2011/index.html