Annex B Conceptual Framework
Conceptual framework for case study analysis 1 Overview
This evaluation revolves around three key issues:
- CSO effectiveness: What are the ways in which CSO engagement in (country) policy dialogue is most effective – and what does this mean for how this can be facilitated in the future?
- Enabling and disabling conditions: What are the enablers and barriers to CSO engagement (at country level) – and how could they be addressed?
- DP policies and strategies: How can DPs most effectively support and facilitate (directly and indirectly) increased civil society engagement at country level?
The following figure shows the three main elements, the concepts that are used to analyse them and the linkages that will be investigated through this study.
The key concepts that have been studied during inception (indicated in blue) include:
- CSO strategies to engage in policy dialogue
- Policy dialogue
- Outcomes of policy dialogue
- Enabling and disabling conditions affecting CSO effectiveness.
The key linkages which will be investigated through case studies during the main phase (shown in red) include:
- Key enabling and disabling factors and how they affect CSO choice of strategies
- Policy dialogue: How CSOs access and use the space for policy dialogue? and What entry points they use into policy cycle?
- What are the successes and failures of CSO engagement in policy dialogue? and What are the (process) outcomes with regard to policy change?
Below, we present the key concepts for this evaluation, and then explain how we will investigate the linkages between them through the case studies. Since most of the evidence for this evaluation is collected through case studies the use of checklists and standardised reporting formats is critical to enable comparative analysis during the synthesis stage. We therefore developed detailed typologies and checklists for analysis of the key concepts which will help us to identify common features across case studies. The checklists are presented below.The framework for analysis of DP policies and strategies (7 Cs) is presented separately. The evaluation will look at DP support from different angles: From a general perspective, whether DP policies and strategies (in principle) support effective CSO engagement in policy dialogue; and from a country perspective, whether DP support practices enable (or perhaps prevent) a more effective role of CSOs.
2 Key concepts
2.1 CSO strategies to engage in policy dialogue
Based on suggestions from CIDA during inception and other sources we have developed a typology of CSO engagement in policy dialogue. The typology contains a number of strategies, which CSOs use – directly or indirectly – to influence policy makers.
This includes highly visible strategies, like advocacy, campaigning and demonstrations, but also less-visible strategies, such as networking and evidence-based studies. Policy dialogue is often perceived as direct engagement between CSO and government only, but there are other ways (particularly highlighted by Northern CSO consulted during inception) through which CSO contribute to policy processes, for example through training, education, community mobilisation and projects that are piloting innovative practices. Donors often tend to focus on the formalised dialogue, which is more visible to them, but country stakeholders emphasised that it is often the informal forms of dialogue that are effective. Checklist 1 shows the different forms of CSO engagement, clustered into four main types.
Checklist 1 Typology of CSO engagement in policy dialogue
|Typology of CSO engagement in policy
dialogue (as used during scoping studies)
|Questions for case|
Direct & formalised dialogue
- Advocacy and campaigning
- Participation in sector or PRSP planning
- Monitoring, reporting, social accountability
- Evidence-based studies and research
Direct & informal dialogue
- Ad-hoc communication at central level
- Ad-hoc communication at local level
- Insider lobbying
- Networking and coalition building
- Demonstrations and mass action
- Policy analysis and debate
Indirect contribution to dialogue
- Information, education and training
- Projects piloting innovative practices
- Community mobilisation for feedback and advocacy
- Community mobilisation for policy implementation (no feedback mechanisms included)
- Service delivery
How effective are these strategies on their own and in combination to achieve policy change?
Relevant evaluation questions: EQ6, EQ11,
We used this typology to identify the main types of dialogue (formal and informal) that will be covered through the case studies. For example, the Mozambique study selected “Budget Planning and Monitoring” as a case for direct and formal dialogue, and “Dissemination of the law on violence against women” as a case for direct and informal dialogue.
The case studies will revisit the typology in order to determine which strategies (on their own or in combination) have been effective in influencing policy dialogue.
2.2 Policy dialogue
Policy dialogue is a broad concept which different stakeholders understand and interpret in different ways. For foreign governments and donors, policy dialogue often refers to the (formal) dialogue at government level. For country stakeholders, policy dialogue both refers to dialogue between government and civil society and within civil society. The Uganda Scoping Study thus distinguishes between “vertical” and “horizontal” dialogue.
It is important to understand the process nature of policy dialogue. Policy dialogue involves ongoing negotiation of ideas, relations and power; thus, it is a process for establishing legitimacy (as pointed out by the Uganda study), for mutual learning and for influencing. The process nature of policy dialogue also means that it extends beyond “policy making” into implementation, review and revision of policies. The ToR for this evaluation thus demand a study of policy dialogue throughout policy development and implementation.
In the context of this evaluation, dialogue is understood as a way of influencing policy processes. In order to review the linkages between policy dialogue and policy processes, the evaluation uses two main tools: The policy cycle tool is useful to conceptualise how policy processes work and what the entry points for influencing are. The power cube tool and the concept of space which it contains are useful to analyse the power relations that define the space for policy dialogue. The use of these tools can help to explain why CSO engagement has been effective (or not).
The power cube tool can help to explain how CSOs have been able to access and use spaces for influence (and power), such as policy dialogue. The power cube distinguishes between invited, claimed and closed spaces for participation. The conceptual aspects (and terminology) of the power cube are useful to map the inclusiveness of spaces for CSO participation. But the nature of policy processes transcending several spaces are often difficult to capture within the categories suggested by the power cube.
The policy cycle tool describes the phases of policy development and implementation at iterative process (see figure below). Effective CSO strategies use various entry points into the policy cycle to influence policy processes.
Figure 4 Possible CSO entry points into policy cycle tool
2.3 Outcomes of policy dialogue
For the case studies it is important to break down the concept of influence into (intermediate) outcomes from specific CSO strategies that can already be observed and long-term policy changes. Intermediate (process) outcomes are important to trace CSO influence in policy dialogue. In some cases it may be possible to link policy changes, like the adoption of new policies or the implementation of policies, directly linked to CSO inputs, e.g. through provision of policy papers of proposals that have been taken up.
In other cases, CSOs only had an indirect influence, e.g. through framing issues or raising awareness through media campaigns. However, in most cases it may only be possible to measure intermediate (process) outcomes of CSO strategies. Intermediate outcomes leading to more effective engagement of CSOs in policy dialogue include strengthened organisational capacity, strengthened alliances and strengthened base of support. In the checklist below we present the possible outcomes of CSO strategies. The checklist will serve as guidance for the identification of (intermediate and policy change) outcomes through the case studies.
Checklist 2 Measuring influence – Possible outcomes of CO engagement in policy dialogue
|Intermediate (process) outcomes
||Inputs into policy dialogue
Strengthened organisational capacity
- Improved management including transparency and accountability
- Improved capacity to communicate messages
- Increased voice and demands for accountability
- Increased participation in civil society-state space
- Increased number of partner supporting an issue
- Improved level of collaboration
- Improved harmonisation of efforts
- Increased number of strategic alliances
Strengthened base of support
- Increased public involvement in an issue
- Changes in voter behaviour
- Increased media coverage
- Increased awareness of messages among specific groups
- Increased visibility
Direct Inputs intopolicy dialogue
- “White papers”
- Policy proposals
- Lessons from pilots projects
- Policy briefings
- Watchdog function
Indirect inputs intopolicy dialogue
- Setting an agenda
- Framing issues
- Media campaign
- Policy development
- Policy adoption
- Policy implementation
- Policy enforcement
Shift in social norms
- Changes in awareness of an issue
- Changes inperceptions
- Changes in attitudes and values
2.4 Enabling and disabling conditions affecting CSO effectiveness
For “civil society to flourish it requires a favourable enabling environment, which depends upon the actions and policies of all development actors – donors, governments and CSOs themselves.” For the purpose of this evaluation we understand “enabling and disabling conditions” as the key parameters defining the space for policy dialogue and the opportunities for CSOs to participate. The Scoping Study have conducted a systematic review of dimensions the defining the enabling environment in the context of case study countries, based on documents review and using the checklist below. The extent to which these conditions affect CSO engagement in policy dialogue varies between countries. It will therefore be important to document the key barriers and enablers for CSO effectiveness in a way that allows comparative analysis during the final synthesis. We therefore use a structured reporting framework to document findings, based on the checklist below. The case studies will revisit the analysis of the enabling and disabling conditions done during the scoping studies in order to identify the factors that have affected CSO engagement in policy dialogue. Based on this analysis, the final synthesis will then elaborate the common and differing elements that present barriers to an effective role of CSOs.
Checklist 3 Enabling and disabling conditions
|Enabling and disabling conditions for CSOs
(as used for scoping studies)
|Questions for analysis|
of case studies
- Legal and judicial system and related mechanisms through which CSOs or their constituencies can seek legal recourse
- Democratic parliamentary system and opportunities for CSO to build alliances with members of parliament
- Power and power relations (between CSO and Government; relations between CSOs and citizens, CSOs and other CSOs and the private sector)
- Measures to promote philanthropy and corporate social responsibility
- Mechanisms to ensure the promotion and protection of the rights to expression, peaceful assembly and association, and access to information
- CSO-specific policies such as CSO legislation and taxation regulations including charitable status provisions
- Regulations and norms promoting CSO transparency and accountability to their constituencies
- Access to funding (and role of donors); abilityto mobilise resources (financial, skills, people, in kind contributions)
- Ethnic and social issues, economic structures
- CSOs own capacity and commitment
|Whether certain aspects of the enabling framework can explain the success or failure of CSO strategies. (EQ15)?|
How elements of the enabling framework define the space for policy dialogue.
To what extent DP strategies address critical aspects of the enabling framework in order to support an effective CSO role in policy dialogue (EQ 16)?
3 Establishing linkages through case studies
3.1 A “practical” theory of change for case studies
After the conceptual building blocks have been established (through the scoping studies), the case studies will focus on interrogating the linkages between CSO strategies on policy dialogue and policy change outcomes. The case studies will make use of existing documentation to the extent possible; however, we expect that the linkages will mainly be assessed on the basis of information derived from stakeholder interviews and focus groups. Analysis therefore needs to be systematic and involve steps for crosschecking and verification.
The purpose of the case studies will be to provide an in-depth analysis of how CSO strategies have contributed to policy outcomes. One challenge in measuring influence through policy dialogue is that organisations often claim to be influential (also to justify the support they receive) and that the evidence to support these claims often relates to low-level outcomes or even outputs. Furthermore the very nature of policy work, involving multiple interventions by numerous actors and a wide range of external factors, complicates the analysis of causality and attribution. It will therefore be critical to establish plausible links between CSO strategies and policy change. This will be done through a “practical” theory of change, which we will develop through a participatory process involving various stakeholders and sources to enable crosschecking and verification.
The theory of change is a technique to structure our understanding how CSO strategies have contributed to policy outcomes. As a visual tool the theory of change depicts the pathways that lead from specific activities of individual CSOs to wider policy changes, thus establishing causal linkages through interactive stakeholder analysis.
A “practical” theory of change Outcomes (influence)
A major aspect in developing the theory of change is to test the plausibility of perceptions (and claims) around policy dialogue outcomes, using a two-way approach:
- Working forwards from strategy to outcomes (b): We review CSOs and their achievements vis-à-vis objectives and any evidence on outcomes achieved. This will be done through meta-analysis of the available data in CSO reports, using the checklist on outcome indicators above (see Checklist 2). Claims about outcomes and impacts made in the documentation can be cross-checked through interviews and focus group discussions. Simultaneously the team will also assess what issues led to identified policy changes by a process of tracing and uncovering the steps through which outcomes have been generated, exploring how and why decisions or practices were executed and what the role of the different stakeholders were in that process. This will be done through the process analysis tool.
- Working backwards from impact to outcomes (c): This means we identify key policy changes (impacts) and identify the role that CSOs have played in it.
As a first step we will review the available literature (studies, evaluations etc.) to establish wider policy changes. We will then interrogate any linkages between those changes and the outcomes that CSOs have achieved through group discussions, which involves a wider range of (non CSO) stakeholders, including representatives from government, think tanks etc. We will use force field analysis as a tool to understand the dynamics of change and the role different actors have played in it.
Factors affecting CSO effectiveness (a): The final element of the case study analysis will be the identification of factors that have affected CSO engagement in policy dialogue. Naturally, this part of the analysis will be done in conjunction with the analysis of CSO strategies and outcomes (c). Key factors will be identified through CSO focus group discussions, using participatory tools, such as SWOT or force field analysis. Based on our initial understanding for documents review and scoping studies we have identified key factors explaining CSO effectiveness, which are presented in the table below. Checklist 4 includes presents key factors for consideration during the case studies, some of them directly linked to the “enabling conditions” (space, government attitude); others are CSO-related factors (CSO legitimacy, capacity and networks). The case studies will use these (and any additional factors identified during the study) to construct a theory of change around the issues that have been influenced by CSOs.
Checklist 4 Factors explaining CSO effectiveness
|Factors affecting CSO engagement in policy dialogue
||Questions for case study analysis|
|Factors relating to the enabling conditions:
||What are the key factors influencing whether CSO engage in policy dialogue (EQ 14)?|
What are the main enabling and constraining factors that affect CSO engagement (EQ 15)?
To what extent have DP support strategies addressed these factors (EQ 15)?
Spaces for policy dialogue
- Transparent, accessible and inclusive space
- Regular and systematic opportunities for participation, covering all stages of policy process
- Shared principles, including recognition of the value of each stakeholder group’s voice, mutual respect, inclusiveness, accessibility, clarity, transparency, responsibility and accountability
- Attitudes and behaviour
- Capacities, skills and knowledge
|Factors relating to the policy process itself:|
Policy issue and process:
- Nature of the policy issue (e.g. how controversial)
- Timing of policy process
- Access to information
|CSO internal factors:|
CSO legitimacy, capacity and networks
- CSO strategic clarity and focus on opportunities
- CSO capacities, funds and knowledge
- CSO Strategic alliances and networks
- CSO sound evidence and analysis
- CSO legitimacy
Analysis, crosschecking and verification: The final analysis will bring together the various elements of the case studies, establishing a plausible link between CSO strategies, policy dialogue and outcomes. As part of the final analysis the evaluators will use analytical tools, such as power cube and policy cycle tool, to analyse the various elements that contribute to CSO effectiveness. The power cube will be used to analyse the inclusiveness of spaces for policy dialogue; the policy cycle tool to determine which entry points CSOs have used to influence policy dialogue. The analysis will be done during the final verification workshops, which will include a wider range of stakeholders, including representatives from government, media, INGOs and academics. During the final verification and feedback workshops the team will also present their theories of change for the selected policy areas for verification by a wider group of stakeholders.
3.2 Process for case studies
Case studies will be conducted through nine steps which are illustrated in the figure below.
Figure 5 Process for case studies
 The term “CSO effectiveness” emphasises the effectiveness of CSOs as development actors (see OECD 2010, Civil society effectiveness).
 OECD 2010: CS effectiveness and adapted from ODI 2006. Policy engagement – How CS can be more effective.
 Adapted from Jane Reisman et al. A guide to Measuring advocacy and policy, Organisational Research Services, 2007.
 OECD 2010: Civil society effectiveness.
 Based on Advisory Group 2008, p. 17-18; Jacqueline Wood & Real Lavergne. 2008 Civil Society and Aid Effectiveness.
 Adapted from Jacqueline Wood and Real Lavergne. 2008. Civil Society and Aid Effectiveness
– An exploration of Experiences and Good practice, p. 11; ODI 2006. Policy engagement
– How CS can be more effective, p. 15-16.
This page forms part of the publication 'Support to Civil Society Engagement in Policy Dialogue' as chapter 14 of 19
Version 1.0. 07-01-2013
Publication may be found at the address http://www.netpublikationer.dk/um/11194/index.htm