Royal danish ministry of foreign affairs - Go to the frontpage of um.dk   Publication  
 
 
     
 
 

2 Methodology

2.1 A conceptual framework

Drawing on the ToR and the lessons learned during the inception and scoping phases, a conceptual framework was devised and documented to guide the case study approach and analysis, with the specific aim of providing direction and consistency of approach to the Country Teams during the main study phase. The Conceptual Framework document is given as Annex B with this chapter providing a methodological overview, the selection process for identifying the case studies, information sources, evaluation tools and the role of the Theory of Change in the study. The validity and the study limitations are also described and discussed.

2.2 Methodology overview

The Country Study was divided into an Inception period (Phase 1) which included a Scoping Study, followed by the detailed Case Studies phase (Phase 2). The findings from this study, together with the findings of the other two Country Studies, provide the primary source material for the Synthesis Phase (Phase 3). The objectives, timing and outputs of each phase are given in the following table.

Table 1 Methodological Overview
Phase 1:
Inception
Phase 2:
Country Studies
Phase 3:
Synthesis
Objectives
  • Understand different stakeholders perceptions of policy dialogue
  • Understand the context for CSO action
  • Provide recommendations for the policy processes which will provide the most useful insights into what works and what does not
  • Understand the current portfolio of DP support
  • Review the relevance, effectiveness and efficiency of the selected policy processes in Bangladesh:
  • Local governance
  • Education policy
  • Minority land rights
  • Food security (mini review)

Other case studies were conducted in Mozambique and Uganda.

  • Analyse and draw lessons learned from the country case studies
  • Situate findings within the debate on civil society engagement
  • Identify cross cutting findings and conclusions
  • Present findings to broad group of DPs
Timing
July-November 2011 December 2011-March 2012 May-September, 2012
Main methods
  • In country participatory workshops with CSO representatives
  • Interviews with key informants in country
  • Workshops with University students and media
  • Meetings and interviews with DP representatives
  • Secondary data review
  • Review of policy processes in each country
  • Interviews and focus group discussions with stakeholders
  • Observation of civil society engagement in action
  • Review of project proposals, strategies and evaluations
  • Findings validation workshop
  • Sharing findings with DPs in country
  • International sharing workshop in Kampala
  • Interaction with ICSOs e.g. BetterAid, Open Forum
  • Meta-analysis
Output
  • Inception Report
  • Bangladesh Country Report
  • Synthesis Report
  • International presentation of the findings

2.3 The case study approach

A case study approach is used to assess policy processes to provide a more holistic understanding of the collective and diverse roles played by different actors within a particular process. The selection of policy processes for the case studies involved a careful consultative procedure based on the relevance of the policy process for the country and DPs as well as diversity of CS action involved in order to provide the best possible basis for learning lessons.

It is important to note that the cases were selected to help identify lessons learned regarding civil society effectiveness in policy dialogue within the policy themes as a whole rather than to examine the specific support of the commissioning DPs. The policy processes comprise a mix of CS action, only some of which is directly related to the specific programmes of the commissioning DPs. The lessons learned therefore cut across all forms of support and cannot be attributed to specific DP action. It is also important to recognise that they are not representative of the ’universe’ of CS action which is extremely broad and diverse.

Phase 2 Case studies (policy processes) were selected through a consultative process in Dhaka with the following criteria in mind:

  • Range of CSOs involved (to understand the diversity of CSOs and to ensure at least some of those policy processes finally selected would include ’less usual’ CSOs such as Trade Unions, faith based groups, professional associations and diaspora groups)
     
  • range of CS action (to review the diversity of action from formal to informal (invited and claimed) so that this range could be captured in at least some of the case studies)
     
  • the level at which CS action takes place (to ensure that at least some of the case studies included local, national and international experience and which involved action outside the capital)
     
  • types of funding modalities (to be able to choose at least some case studies which would allow review of the benefits and constraints of different modes of funding)
     
  • inclusion of CSOs currently funded by the DP reference group
     
  • the relevance of the policy process (to people living in poverty and to the particular country context) i.e. policy processes which are of key importance to development and where CSOs have played a role
     
  • effectiveness of the policy process (outcomes achieved bearing in mind that much could also be learned from mixed or poor achievements)
     
  • availability of documentation on the policy process.

The details of this selection process can be found in the Bangladesh Scoping Study Report. A typology of the CSOs participating in the study is provided in Annex G of this report.

2.4 Information sources

For each policy process, a variety of sources of information were identified as follows:

  • The key CSOs (regarded as ’movers and shakers’) as well as others operating in the same context which had not engaged (documentation review of project proposals, evaluations etc, interviews and observation)
     
  • sources of funding and support (DPs, fund managers, INGOs) for engagement in policy dialogue (documentation review of policies, disbursements and evaluations etc., interviews)
     
  • the key government participants to policy dialogue in the selected policy process areas (interviews)
     
  • research institutions, ’think tanks’ and CS activists (interviews).

2.5 Evaluation tools

In order to facilitate a comparison of the analysis done in the case studies and to ensure more analytical rather than descriptive reports the team used common analytical frameworks.

Evaluation Framework: The Case Studies were undertaken using a common Evaluation Framework (see Annex C) comprising 18 evaluation questions derived from the ToR. The framework detailed specific evidence which would be required to answer the questions. Over 60 face to face interviews were conducted in the Bangladesh Case Study using the evaluation questions as guidelines as well as sharing and debating the theory of change and policy process analysis charts. These included meetings with CSOs, activists, government staff, politicians and locally elected representatives as well as donors active in the thematic areas. Workshops and FGDs were held with a variety of formal and informal CSOs and media representatives (see Annex D for details).

Appreciative enquiry principles (see Annex H) were used in interviews and focus group discussions (FGD) to establish achievements and success in the different policy processes. This helped participants recognise that change had happened even in some cases where external factors seemed to be insurmountable hindrances and where there were high levels of frustration at the lack of progress.

The policy dialogue cycle tool depicted in Figure 1 (Chapter 1) was used to help locate entry points for CS action.

The Power Cube: Another key analytical tool used in the study is the Power Cube which provides a framework to analyse how power operates in the spaces and places for engagement. The diagram below provides a graphic representation of the different manifestations of power. The concept of closed, invited and claimed spaces have been explained above. The visibility of power is categorised as i. visible (i.e. the formal rules, structures and procedures which govern engagement), ii. hidden (i.e. the actual influence those engaging have over others within the engagement space) and iii. invisible (i.e. the power dynamics assumed by participants from their socialisation and societal norms). The conceptual framework helped in the analysis of power relations, levels of operation and understanding of spaces for CS engagement. (See also Annex H).

Figure 2 The Power Cube

Figure 2 The Power Cube

Source: Gaventa, 2003

Field observations were carried out and included observation of a variety of CS-state engagements (see Annex D for the list of persons who participated in the study).

2.6 Theory of Change as a conceptual framework for the Case Studies

The study took an evaluative approach based on Theory of Change (ToC). ToC is a based on programme theory and is an approach which seeks to understand processes of change beyond the measurement of results to include more explicit reflection on the assumptions behind technocratic causal frameworks. In particular it examines the context, actors and processes of change to support learning about what constitutes effective strategies. Developing ToCs for civil society engagement in policy dialogue work has proved especially challenging as the complex nature and dynamics of both civil society action and its engagement with the State is not amenable to linear logic. The array of formal and informal, consensual and dissenting voices as well as the wide range of different incentives for and interests of policy dialogue stakeholders provides a complex web of interactions where causal relationships are hard to distinguish.

ToC is supposed to provide a flexible framework for critical and adaptive thinking rather than a product.[11] There are many interpretations and visual representations of ToC available in recent literature but the fundamental principles are similar and include the need to understand i. the context, ii. the actors, iii. the desired-for change and iv. the linked events/processes leading to change.

Evaluation and attribution

Establishing attribution is the most challenging element of any study on policy influencing. Policy and practice change is a result of highly complex interacting forces and actors.

Different constellations of actors engage and disengage, work continuously over long periods of time or exploit moments of opportunity and undertake a wide variety of activities to influence change. Tipping points can be reached in a multitude of different ways.

The case studies used ToCs to capture the different elements contributing to change in policy and practice. These helped to ensure that the multiplicity of actions and actors were taken into account when trying to establish attribution and provided a focus for discussion among different actors regarding their relative contributions. However, they also served to highlight how linear and short-term models of change may lead to exaggeration of success as the contribution of others before and in parallel are generally overlooked. This alerted the team to the need for cautious interpretation of reported success in interviews, project reports and evaluations of individual organisations.

As well as examining impact level outcomes, the teams purposefully examined process outcomes as legitimate markers of achievement. These include legislation, creation of new or expanded participatory space and official platforms for civil society engagement, behaviour and attitude change of service providers and duty bearers.

Scope of work

The evaluation inevitably was limited in scope by practical considerations. While having the advantage of examining the complete cycle of policy dialogue it nevertheless was limited by selection of just a few policy processes. All three case studies looked at elements of governance which provided cross-cutting information for comparative purposes.

The time horizon suggested in the ToR was policy dialogue in the last five years. While this provides information on CSOs currently active and, in particular the ’movers and shakers’ identified in the ToR (3.1) it may have constrained the need to view the long-term perspective of change. Many of the achievements have not resulted from recent engagement but from longer term ’drip-drip’ actions as well as incremental changes in the enabling environment. This limitation has been mitigated somewhat by the fact that all team members have long-term experience of the country context, civil society participation and CS action.

Validity of findings

Recognising the complex and often politically charged environment in which policy dialogue takes place, the team was cautious about attribution and accepting accounts of processes at face value. They exercised care to triangulate findings in a number of ways:

  • Purposeful inclusion of a range of CSOs in each policy process, including ’movers and shakers’ as well as those apparently less active
     
  • interviews with Government (supply-side), key informants not connected with CSOs (independent view) and DPs
     
  • document review (especially during Phase 1) including websites, newspaper clippings, YouTube
     
  • exposure to civil society engagement in action (meetings, debates, public hearings, TV Talk Shows etc.)
     
  • verification workshops with mixed participants representing different stakeholder groups to confirm and extend study findings
     
  • circulation of draft country reports to a variety of stakeholders for comment and further development.

The research team was able to draw on their own recent assignments to supplement this study including:

  • End of Programme Evaluation of CAMPE (January-February 2012)
     
  • Advisory preparation of Aparajita project (empowering women elected representatives) (May-August, 2011)
     
  • Evaluation of DFID’s Support to Civil Society (February 2011)
     
  • Mid-term Review of Transparency International, Bangladesh (November/ December 2011)
     
  • Research on Mobilising Resources for Women’s Rights undertaken as part of the Pathways of Women’s Empowerment Consortium (2010-11).

Where possible different points of view are provided in the text in order to provide balanced accounts.

2.7 Country specific limitations – Bangladesh

The Opposition Party called a national Rally on March 12th to which the ruling party responded to by effectively closing travel in and out of Dhaka for the preceding days and calling their own supporters to rally the following day. This resulted in cancellation of the planned trip to Chapainawabganj to observe a live Open Budget Meeting and inhibited movement in Dhaka.

SDC organised a major review of it local governance portfolio coincident with this study. This resulted in their unavailability as well as confusion among study participants about which study we were under and was burdensome to the same respondents. CIDA was closing its fiscal year at the end of March and was busy meeting many urgent deadlines making it difficult to meet on the CHT and food security issues although we did interact on education. All DPs shared with us that headquarters demands for paper work and hosting various delegations, missions and evaluations have increased and that they are increasingly under stress and unable to participate as they would like in local processes.

The Directorate of Primary Education and the various donors involved in the third Primary Education Development Plan (PEDP III) were busy with field visits and preparatory work for the May Joint Annual Review Mission. This made it difficult to meet the relevant persons in both Government and donor agencies. One of the key CSO players in the Primary Education sector (CAMPE) was preoccupied by its ’end of project review’ and negotiations with DPs for its next phase of project funding, making it difficult to have enough time with them.

The distance from CHT to Dhaka meant that although CSO representatives were interested to participate in the Dhaka workshops it would have involved three days of travel. The Parliamentary Standing Committee for local governance was not available during the period of the study.


[11] Review of the Use of Theory of Change in International Development, Isabel Vogel, April 2012.




This page forms part of the publication 'Support to Civil Society Engagement in Policy Dialogue' as chapter 5 of 20
Version 1.0. 03-01-2013
Publication may be found at the address http://www.netpublikationer.dk/um/11191/index.htm

 

 
 
 
 
  Ministry of Foreign Affairs of Denmark © | www.um.dk