Royal danish ministry of foreign affairs - Go to the frontpage of um.dk   Publication  
 
 
     
 
 

2 Methodology

2.1 A conceptual framework

Drawing on the ToR and the lessons learned during the inception and scoping phases, a conceptual framework was devised and documented to guide the case study approach and analysis, with the specific aim of providing direction and consistency of approach to the Country Teams during the main study phase. The Conceptual Framework document is given as Annex B with this chapter providing a methodological overview, the selection process for identifying the case studies, information sources, evaluation tools and the role of the Theory of Change in the study. The validity and the study limitations are also described and discussed.

2.2 Methodology overview

The Country Study was divided into an Inception period (Phase1) which included a Scoping Study, followed by the detailed Case Studies (Phase 2). The findings from this study, together with the findings of the other two Country Studies, provide the primary source material for the Synthesis Phase (Phase 3). The objectives, timing and outputs of each phase are given in the following table.

Table 1 Methodological Overview

(How the Uganda Country Study fits in to the overall programme of study)

Phase 1:
Inception
(including Scoping Study)
Phase 2:
Country Studies (Case
Studies of Policy Processes)
Phase 3:
Synthesis
Objectives
  • Understand different stakeholders perceptions of policy dialogue
  • Understand the context for CSO action
  • Provide recommendations for the policy processes which will provide the most useful insights into what works and what does not
  • Understand the current portfolio of DP support
  • Review the relevance, effectiveness and efficiency of the selected policy processes in Uganda:
  • Governance & Accountability, Anti-Corruption
  • Gender-based legislation
  • Forest Management and Governance

Other case studies conducted in Bangladesh and Mozambique

  • Analyse and draw lessons learned from the country case studies
  • Situate findings within the debate on civil society engagement
  • Identify cross cutting findings and conclusions
  • Present findings to broad group of DPs
Timing
September-November 2011 December 2011-March 2012 May-October, 2012
Main methods
  • In country participatory workshops with CSO representatives
  • Interviews with key informants in country
  • Workshops with keystake holders
  • Meetings and interviews with DP representatives
  • Secondary data review
  • Review of policy processes
  • Interviews and focus group discussions with stakeholders
  • Observation of civil society engagement in action
  • Review of project proposals, strategies and evaluations
  • Findings reviewed in validation workshop
  • Sharing findings with DPs in country
  • International sharing workshop in Kampala
  • Interaction with ICSOs e.g. BetterAid and Open Forum
  • Meta-analysis
Output
Inception Report Uganda Country Report

Country reports produced for Bangladesh and Mozambique
Synthesis Report

International presentation of the findings

2.3 The case study approach

A case study approach is used to assess policy processes to provide a more holistic understanding of the collective and diverse roles played by different actors within a particular process. The selection of policy processes for the case studies involved a careful consultative procedure based on the relevance of the policy process for the country and development partners as well as diversity of CS action involved in order to provide the best possible basis for learning lessons.

It is important to note that the cases were selected to help identify lessons learned regarding civil society effectiveness in policy dialogue within the policy themes as a whole rather than to examine the specific support of the commissioning DPs. The policy processes comprise a mix of CS action, only some of which is directly related to the specific programmes of the commissioning DPs. The lessons learned therefore cut across all forms of support and cannot be attributed to specific DP action. It is also important to recognise that they are not representative of the ’universe’ of CS action which is extremely broad and diverse.

Phase 2 Case studies (policy processes) were selected through a consultative process in Kampala, based on findings of a scoping study, which identified seven key policy issues of concern in Uganda which CSOs have been engaged in the last five years with the following criteria in mind:

  • Range of CSOs involved (to understand the diversity of CSOs and to ensure at least some of those policy processes finally selected would include ’less usual’ CSOs such as Trade Unions, faith based groups, professional associations and diaspora groups)
     
  • range of CS action (to review the diversity of action from formal to informal (invited and claimed) so that this range could be captured in at least some of the case studies)
     
  • - the level at which CS action takes place (to ensure that at least some of the case studies included local, national and international experience and which involved action outside the capital)
     
  • types of funding modalities (to be able to choose at least some case studies which would allow review of the benefits and constraints of different modes of funding)
     
  • inclusion of CSOs currently funded by the DP reference group
     
  • the relevance of the policy process (to people living in poverty and to the particular country context) i.e. policy processes which are of key importance to development and where CSOs have played a role
     
  • effectiveness of the policy process (outcomes achieved bearing in mind that much could also be learned from mixed or poor achievements)
     
  • availability of documentation on the policy process.

The details of this selection process can be found in the Uganda Scoping Study Report (July and September 2011). ’Chapter 7, Key policy areas and suggested areas for the case study’. (The Scoping Study can be requested from Danida. Write to eval@um.dk)

2.4 Information sources

For each policy process, a variety of sources of information were identified as follows:

  • The key CSOs (regarded as ’movers and shakers’) as well as others operating in the same context which had not engaged (documentation review of project proposals, evaluations etc, interviews and observation)
     
  • sources of funding and support (DPs, fund managers, INGOs) for engagement in policy dialogue (documentation review of policies, disbursements and evaluations etc., interviews)
     
  • the key government participants to policy dialogue in the selected policy process areas (interviews)
     
  • research institutions, ’think tanks’ and CS activists (interviews).

A key feature at this stage was to ensure full stakeholder participation in the process (see Box 1).

Box 1 Stakeholder participation

Following the scoping study and selection of the three policy areas, a stakeholder mapping was done by the team to identify the key players and stakeholders in the selected policy areas at national level and district level. Participants selected included CSOs at national level, INGOs, the media, DPs, Government Ministries, Agencies and Departments (MDAs),[11] Representatives from Parliament, district-based NGOs/CSOs and Community Members through community-based organisations (CBOs). The three districts of Soroti and Lira in North Eastern Uganda and Buikwe District (Mabira area) were mapped as providing good examples where the case studies could be followed-up to the grass roots level. The districts also established the link between national level and district level issues.

Key informant interviews were held in all districts especially with district and sub-county authorities, while Focus Group Meetings were held with community members. In general people at all levels including government representatives were very open in their discussion and forthcoming in giving information.

2.5 Evaluation tools

In order to facilitate a comparison of the analysis done in the case studies and to ensure more analytical rather than descriptive reports the team used common analytical frameworks.

Evaluation Framework: The Case Studies were undertaken using a common Evaluation Framework (see Annex C) comprising eighteen evaluation questions derived from the ToR. The framework detailed specific evidence which would be required to answer the questions.

Key evaluation questions: Key evaluation questions were developed along the four main areas of focus of the study on i) Enabling and disabling environment for CSO engagement in policy dialogue and factors affecting their engagement, ii) CSO Effectiveness in relation to accountability and legitimacy, iii) Results and outcomes focused on policy process outcomes and policy changes as well as CSO achievements and; iv) DP Support on Policy dialogue focused on CSO perspectives of DP support, donor perspective of DP support and DP support and enabling factors (see Annex F). The key questions were developed into a generic topic guide for each category of interviews.

The policy dialogue cycle tool depicted in Figure 1 was used to help locate entry points for CS action.

The Power Cube: Another key analytical tool used in the study is the Power Cube which provides a framework to analyse how power operates in the spaces and places for engagement. The diagram below provides a graphic representation of the different manifestations of power. The concept of closed, invited and claimed spaces have been explained above. The visibility of power is categorised as i. visible (i.e. the formal rules, structures and procedures which govern engagement), ii. hidden (i.e. the actual influence those engaging have over others within the engagement space) and iii. invisible (i.e. the power dynamics assumed by participants from their socialisation and societal norms). The conceptual framework helped in the analysis of power relations, levels of operation and understanding of spaces for CS engagement.

Figure 2 The Power Cube

Figure 2 The Power Cube

Source: Gaventa, 2003

Field observations were carried out and included observation of a variety of CS-State engagements (see Annex D: List of persons who participated in the study).

2.6 Theory of Change as a conceptual framework for the case studies

The study took an evaluative approach based on Theory of Change (ToC). ToC is a based on programme theory and is an approach which seeks to understand processes of change beyond the measurement of results to include more explicit reflection on the assumptions behind technocratic causal frameworks. In particular it examines the context, actors and processes of change to support learning about what constitutes effective strategies. Developing ToCs for civil society engagement in policy dialogue work has proved especially challenging as the complex nature and dynamics of both civil society action and its engagement with the State is not amenable to linear logic. The array of formal and informal, consensual and dissenting voices as well as the wide range of different incentives for and interests of policy dialogue stakeholders provides a complex web of interactions where causal relationships are hard to distinguish.

ToC is supposed to provide a flexible framework for critical and adaptive thinking rather than a product.[12] There are many interpretations and visual representations of ToC available in recent literature but the fundamental principles are similar and include the need to understand i. the context, ii. the actors, iii. the desired-for change and iv. the linked events/processes leading to change.

Theory of Change developed for Uganda case studies

The team developed a ToC based on the findings from the scoping study. The team’s ToC helped to understand CSO policy dialogue in Uganda and the connections between CSO strategies, outcomes and actions, illustrated in Figure 2 below. The ToC has been used to investigate and identify the main goals of the policy issues, the main factors in the enabling environment, the CSO strategies that have been most effective and to map out a checklist of indicators to assist in collecting the evidence, as illustrated below.

The long-term goal for policy dialogue across the three case studies

The ’long-term goal’ for policy engagement by CSOs in Uganda identified by the evaluation team across the three policy areas is the Attainment of Effective Governance in Uganda. The specific policy outcomes per case study are identified as follows:

  • Case study 1: Governance and Accountability: Policy Dialogue on Anti-Corruption, with the goal to achieve a “A well governed, accountable and corruption free society”
     
  • Case study 2: Justice law and Order Sector: Policy Dialogue on Gender Responsive Legislation aimed at “Equal and non-discriminatory legislation in Uganda”
     
  • Case study 3: Natural Resources Management: Policy Dialogue on Forestry Management/Governance aimed at “Sustainable management of forests in Uganda”

The practice in the past has been that CSOs work with government-provided spaces which are seen as ’a given’ by the constitution and the Local Government Act with provisions that allow all parties to participate in government processes. The team recognised that some CSOs supported government-driven processes and contribute ideas, hoping that the system would change for the better. However, the study found that some CSOs have now realised that for them to be effective and influence change in policy dialogue, they have to work from the grass roots up. The evaluation looked at CSOs operations at national, district and community level and how they work with the media, and other like-minded CSOs in coalitions or networks and policy makers at the different levels, especially the parliament.

Following the findings from the Uganda Scoping Study, the ToC for Uganda policy dialogue is illustrated in the diagram below. The ToC is used in the study to trace how: CSO strategies have contributed to policy outcomes. Individual ToCs have been developed for each case study.

Figure 3 Illustration of Theory of Change for CSO Policy Dialogue in Uganda

Figure 3 Illustration of Theory of Change for CSO Policy Dialogue in Uganda

2.7 Key focus areas for the study

Enabling environment for policy dialogue

The enabling environment is seen as one of the key preconditions that must be in place for the CSOs to operate effectively. Four areas identified as crucial for effective dialogue assessed by the evaluation were:

  • The Legal and Regulatory framework
     
  • Political Context: Government responsiveness and extent of open space for policy engagement
     
  • Existence of self-regulation and strong CSO institutions to engage government in dialogue (technical competence, strong institutions). For example, issue based coalitions to provide a supportive environment for CSOs to advance policy issues
     
  • Extent of funding resources for CSOs to engage in policy dialogue are available (human, finances, time).

Effective strategies for CSO engagement

The evaluation looked at the following checklist of strategies for policy dialogue in Uganda, among them:

  • Identification of policy gaps through research and analysis of existing policies
     
  • Creating awareness and sensitising people about the issues
     
  • Coalition building on different issues of interest and networking among themselves
     
  • Media advocacy using the media as the interlocutor on public issues of concern to aimed at mobilising public opinion about different areas of interest. They use findings and materials on media talk-shows
     
  • Filing petitions and holding campaigns
     
  • Publication of policy briefs aimed at influencing policy makers
     
  • Court cases filed on behalf of the citizens – public interest litigation
     
  • Social mobilisation especially at community levels and holding demonstrations to express their concerns.

Indicators for policy gains

For each case study, the team set out to identify evidence of contributions made by CSO using guidance of the change outcomes. Attribution is very difficult because there are many players in the field of policy dialogue. The team however noted that in policy dialogue, the advocacy processes are important in contributing to intermediate results. Furthermore, some outcomes may occur after a very long time and may not be measurable during a particular CSO programme.

The team drew up a check list of indicators of long-term policy outcomes and the process outcomes which reflect gains in policy dialogue as indicated below:

Examples of long-term policy changes
  1. Pro-poor laws are enacted
  2. Percentage of reduction in gender based violations
  3. Increased budget allocations to sectors
  4. Ruling on Public interest litigation cases (cases won)
Examples of process outcomes identified for the study
  1. Participatory pro-poor budgeting processes
  2. Increased coalitions building and collaboration among CSOs/CBOs
  3. Increased collaboration between CSOs/CBOs and government bodies/agencies
  4. Commitment of policy makers to use/advance CSO proposals
  5. Prevention of actions that are anti-policy
  6. Joint implementation of policy
  7. Research-documents-policy briefs, documents, IEC materials, press-releases etc
  8. Petitions written and filed/submitted to relevant authorities
  9. Public interest litigation cases filed
  10. Scorecard processes: community based monitoring – actors become effective together
  11. Establishment of commissions of inquiry
  12. Institutional reforms/reshuffle/recruitments
  13. Effective Participation in invited dialogue spaces
  14. Capacity built for CSOs engagement in advocacy.

The team was not limited to, but also used the above short list to identify outcomes of policy dialogue and to collect associated evidence where it exists.

2.8 Study limitations

Evaluation and attribution

Establishing attribution is the most challenging element of any study on policy influencing. Policy and practice change is a result of highly complex interacting forces and actors. Different constellations of actors engage and disengage, work continuously over long periods of time or exploit moments of opportunity and undertake a wide variety of activities to influence change. Tipping points can be reached in a multitude of different ways.

The case studies used ToCs to capture the different elements contributing to change in policy and practice. These helped to ensure that the multiplicity of actions and actors were taken into account when trying to establish attribution and provided a focus for discussion among different actors regarding their relative contributions. However, they also served to highlight how linear and short-term models of change may lead to exaggeration of success as the contribution of others before and in parallel are generally overlooked. This alerted the team to the need for cautious interpretation of reported success in interviews, project reports and evaluations of individual organisations.

Box 2 Challenges for measuring success in policy dialogue

From discussions amongst the team, the extent to which “success” is measured in policy dialogue was found to be challenging. The gathering of “Evidence” is limited because some dialogue processes may last for many years as in the case of the gender dialogue on domestic law which has taken over 40 years in Uganda. For some policies, the outcomes may take years to be realised while for some other laws, the processes may never be completed as seen from the discussions which were stalled or bills withdrawn from parliament. To address this challenge, the team agreed to document important process outcomes, for example the actions taken at each stage of policy engagement. Lastly, the question of “attribution” in a context where many players participate still remains a challenge even for the central participants of CSOs and Government. Our view as a team is that CSOs, Government, the media, the DPs and individuals contribute different building blocks in policy dialogue and irrespective of the importance of a stakeholder in the process success in policy dialogue does not rest on only one participant.

As well as examining impact level outcomes, the teams purposefully examined process outcomes as legitimate markers of achievement. These include legislation, creation of new or expanded participatory space and official platforms for civil society engagement, behaviour and attitude change of service providers and duty bearers.

Limitations found in the three case studies

While the team would have liked to have gone into the studies in more depth, and to have followed-up on different issues arising from the meetings, time was the constraining factor. Some case studies such as forestry governance were well documented, while the gender based legislation has not yet fully documented their processes. The case study on corruption is limited by the inconclusive nature of the policy dialogue process on corruption in Uganda. The capacity of CSO to document the evidence of their success especially in the gender policy dialogue and anti-corruption was more limited. Among the challenges of the study was the difficulty in identifying the contribution of each CSO, because of challenges of attribution in advocacy where many players are involved. To address the challenges of attribution, the evaluation team identified a checklist of indicators facilitate the team to identify the policy gains made by individual CSOs. Even where a CSO led intervention was well documented and researched, with proper analysis, the outcomes specific to a single CSO were not easily visible because most of the policy dialogue outcomes outlive the CSO programme duration and can only be realised over a longer time horizon.

Challenges when introducing the Theory of Change

For most CSOs, the ToC concept was not understood and the discussions often quite superficial, especially because many were not familiar with the framework within which the ToC is developed. The general response was more towards explanation of their purpose, vision of policy dialogue, goals and specific strategies. Very few people expressed a clear strategy or vision, but were rather driven by passion and the strategic links between different elements of their work were not clearly articulated. The exception to this view was in the forestry sector, where strategic planning and organisation of CSOs in the natural resources, environment and forestry sectors is relatively well advanced.

Scope of work

The evaluation inevitably was limited in scope by practical considerations. While having the advantage of examining the complete cycle of policy dialogue it nevertheless was limited by selection of just a few policy processes. All three case studies looked at elements of governance which provided cross-cutting information for comparative purposes.

The time horizon suggested in the ToR was policy dialogue in the last five years (see Box 2 above). While this provides information on CSOs currently active and, in particular the ’movers and shakers’ identified in the ToR (3.1) it may have constrained the need to view the long-term perspective of change. Many of the achievements have not resulted from recent engagement but from longer term ’drip-drip’ actions as well as incremental changes in the enabling environment. This limitation has been mitigated somewhat by the fact that all team members have long-term experience of the country context, civil society participation and CS action.

Validity of findings

Recognising the complex and often politically charged environment in which policy dialogue takes place, the team was cautious about attribution and accepting accounts of processes at face value. They exercised care to triangulate findings in a number of ways:

  • Purposeful inclusion of a range of CSOs in each policy process, including ’movers and shakers’ as well as those apparently less active
     
  • interviews with Government (supply-side), key informants not connected with CSOs (independent view) and DPs
     
  • document review (especially during Phase 1) including websites, newspaper clippings
     
  • exposure to civil society engagement in action (Annual General Meetings)
     
  • verification workshops with mixed participants representing different stakeholder groups to confirm and extend study findings
     
  • circulation of draft country reports to a variety of stakeholders for comment and further development.

[11] Government: Included Ministry of Finance, Planning and Economic Development (MFPED), Justice Law and Order Sector Secretariat (JLOS), Ministry of Gender, Labour and Development (MGLSD), Inspectorate General of Government (IGG), Office of the Auditor General (OAG), Ministry of Water and Environment (MOWE), National Forestry Authority (NFA), Directorate of Ethics and Integrity (DEI), and for Parliament, representatives from Budget Office, Public Accounts Committee (PAC), Environment and Naturals Resources Committee and Uganda Women Parliamentarians Association (UWOPA).

[12] Review of the Use of Theory of Change in International Development, Isabel Vogel, April 2012.




This page forms part of the publication 'Support to Civil Society Engagement in Policy Dialogue' as chapter 5 of 19
Version 1.0. 07-01-2013
Publication may be found at the address http://www.netpublikationer.dk/um/11194/index.htm

 

 
 
 
 
  Ministry of Foreign Affairs of Denmark © | www.um.dk