Annex B Methodology and Conceptual Framework
Evaluation of civil society engagement in policy dialogue – conceptual framework to guide case study approach and analysis
The purpose of this paper is to present the key conceptual elements for this evaluation, the linkages between them and how they will be approached through the case study. The paper will serve as guidance for country teams during the main study phase.
This evaluation revolves around three key questions:
- CSO effectiveness: What are the ways in which CSO engagement in (country) policy dialogue is most effective – and what does this mean for how this can be facilitated in the future?
- Enabling and disabling conditions: What are the enablers and barriers to CSO engagement (at country level) – and how could they be addressed?
- DP policies and strategies: How can DPs most effectively support and facilitate (directly and indirectly) increased civil society engagement at country level?
In order to answer these questions, the evaluation will have to develop an in-depth understanding of what CSO strategies for engagement in policy dialogue are, what outcomes they have achieved and what factors have contributed to their success or failure. In addition it has to review how DPs have supported CSO engagement in policy dialogue and how relevant and responsive their support of CSO was within the country context. In-depth analysis of policy processes and CSO engagement in them will be done through case studies.
The case studies will look at the links CSO effectiveness in policy dialogue, the enabling and disabling factors and the role that DP support has played. The three main conceptual elements for this evaluation and the specific concepts that will be used to analyse them are shown in the figure below.
Figure 3 Overview of key concepts and linkages for this evaluation
The key concepts that have been studied during inception include:
- Types of CSO strategies to engage in policy dialogue
- Policy dialogue and what it means within a given context
- The enabling environment and how it defines the space for policy dialogue.
The key linkages which will be investigated through case studies during the main phase include:
- Key enabling and disabling factors and how they affect CSO choice of strategies
- Policy dialogue: How CSOs access and use the space for policy dialogue, and
- What entry points they use into policy cycle?
- What are the successes and failures of CSO engagement in policy dialogue?
- What are the (process) outcomes with regard to policy change?
In addition the figure contains several variables that influence CSO strategies and their outcomes on policy dialogue. They will be an important part of the explanatory models describing how CSOs have influenced policy change (Theory of Change, see below).
Below we present the key concepts for this evaluation, and then explain how we will investigate the linkages between them through the case studies. Since most of the evidence for this evaluation will be collected through case studies of different policy areas set in the contexts of three different countries we will use checklists and standardised reporting formats to analyse and present the key concepts for this evaluation. This approach will support comparative analysis during the synthesis stage. We therefore developed detailed typologies and checklists for analysis of the key concepts which will help us to identify common features across case studies.
The evaluation will look at DP support from different angles: From a general perspective, whether DP policies and strategies (in principle) support effective CSO engagement in policy dialogue; and from a country perspective, whether DP support practices enable (or perhaps prevent) a more effective role of CSOs – thus becoming part of the enabling and disabling factors. The latter will be done as part of the case studies. Analysis of DP policies and strategies at HQ level will be done through an institutional assessment tool (7 Cs) which is presented separately.
2. Key concepts
2.1 CSO strategies to engage in policy dialogue
Based on suggestions from CIDA during inception and other sources we have developed a typology of CSO engagement in policy dialogue. The typology contains a number of strategies, which CSOs use to – directly or indirectly – influence policy makers. This includes highly visible strategies, like advocacy, campaigning and demonstrations, but also less-visible strategies, such as networking and evidence-based studies. Policy dialogue is often perceived as direct engagement between CSO and government only, but there are other ways (particularly highlighted by Northern CSO consulted during inception) through which CSO contribute to policy processes, for example through training, education, community mobilisation and projects that are piloting innovative practices. Donors often tend to focus on the formalised dialogue, which is more visible to them, but country stakeholders emphasised that it is often the informal forms of dialogue that are effective. This evaluation understands that there are different ways of engaging in policy dialogue. In order to be able to assess the effectiveness we need to understand (and structure) the diversity. Checklist 1 thus shows the different forms of CSO engagement, clustered into four main types.
Checklist 1 CSO strategies for engagement in policy dialogue
|Types of CSO strategies in policy dialogue
(as used during scoping studies)
|Questions for case study analysis|
|Direct & formalised dialogue
Participation in sector or PRSP planning
Support social accountability
Evidence-based studies and research
Direct & informal dialogue
Ad-hoc communication at central level
Ad-hoc communication at local level
Protests and demonstrations
Policy analysis and debate
Indirect contribution to dialogue
Information, education and training
Projects piloting innovative practices
Community mobilisation for feedback and advocacy
Community mobilisation for policy implementation (no feedback mechanisms included)
|How effective are these strategies on their own and in combination to achieve outcomes on policy change, given the existing enabling and disabling conditions?|
Relevant evaluation questions: EQ6, EQ11,
The case studies will cover different types of dialogue, both formal and informal.
We therefore used this typology to guide the selection of policy areas where different types of dialogue. For example, the Mozambique study selected “Budget Planning and Monitoring” as a policy area, where for direct and formal dialogue, and “Dissemination of the law on violence against women” as a case for direct and informal dialogue.
The case studies will revisit the typology in order to determine which strategies (on their own or in combination) have been effective in influencing policy dialogue, given the existing enabling and disabling conditions.
2.2 Policy dialogue
Policy dialogue is a broad concept which different stakeholders understand and interpret in different ways. For foreign governments and donors policy dialogue often refers to the (formal) dialogue at government level. For country stakeholders, policy dialogue both refers to dialogue between government and civil society and within civil society. The Uganda scoping study thus distinguishes between “vertical” and “horizontal” dialogue.
It is important to understand the process nature of policy dialogue. Policy dialogue involves ongoing negotiation of ideas, relations and power; thus, it is a process for establishing legitimacy (as pointed out by the Uganda study), for mutual learning and for influencing. The process nature of policy dialogue also means that it extends beyond “policy making” into implementation, review and revision of policies. The ToR for this evaluation thus demands a study of policy dialogue throughout policy development and implementation.
In the context of this evaluation dialogue is understood as a way of influencing policy processes. In order to conceptualise how policy processes work and what the entry points for influencing are the evaluation uses the policy cycle tool. The policy cycle tool describes the phases of policy development and implementation at iterative process (see figure below). Effective CSO strategies use various entry points into the policy cycle to influence policy processes.
Figure 4 Possible CSO entry points into policy cycle tool
2.3 Enabling environment for CSO effectiveness
For “civil society to flourish it requires a favourable enabling environment, which depends upon the actions and policies of all development actors – donors, governments and CSOs themselves.” The scoping study have conducted a systematic review of dimensions the defining the enabling environment in the context of case study countries, based on documents review and using Checklist 2 below.
Checklist 2 Enabling environment
|Elements of an enabling environment
(as used for scoping studies)
|Questions for analysis of case studies|
|Legal and judicial system and related mechanisms through which CSOs or their constituencies can seek legal recourse
Democratic parliamentary system and opportunities for CSO to build alliances with members of parliament
Power and power relations (between CSO and Government; relations between CSOs and citizens, CSOs and other CSOs and the private sector)
Measures to promote philanthropy and corporate social responsibility
Mechanisms to ensure the promotion and protection of the rights to expression, peaceful assembly and association, and access to information
CSO-specific policies such as CSO legislation and taxation regulations including charitable status provisions
Regulations and norms promoting CSO transparencyand accountability to their constituencies
Access to funding (and role of donors); ability to mobilise resources (financial, skills, people, in kind contributions)
Ethnic and social issues, economic structures
|Whether certain aspects of the enabling framework can explain the success or failure of CSO strategies. (EQ15)?|
How elements of the enabling framework define the space for policy dialogue.
To what extent DP strategies address critical aspects of the enabling framework in order to support an effective CSO role in policy dialogue (EQ 16)?
What other factors have influenced CSO engagement in policy dialogue (EQ 14, EQ 15)
For the purpose of this evaluation we understand “enabling environment” as the formal conditions under which CSOs develop their strategies. More specifically, certain elements of the enabling environment will determine the space for CSOs to participate in policy dialogue. The power cube is useful to conceptualise the power relations that – as part of the enabling environment – define the space for policy dialogue. It can help to explain how CSOs have been able to access and use spaces for influence (and power), such as policy dialogue. The power cube distinguishes between invited, claimed and contested spaces for participation. The conceptual aspects (and terminology) of the power cube are useful to map the inclusiveness of spaces for CSO participation. But the nature of policy processes transcending several spaces is often difficult to capture within the categories suggested by the power cube.
2.4 Enabling and disabling conditions
After the scoping studies it was felt that the concept of enabling environment was somehow restricted to covering the formal conditions for policy dialogue only. The conclusion was that a wider concept was needed to also cover the informal conditions that facilitate or restrain CSO engagement in policy dialogue. It was suggested to use the concept of enabling and disabling conditions instead which would cover a wider range of factors, including those relating to DP support and CSO internal factors. Checklist 3 (below) provides a selection of factors which have been identified during the inception phase.
The practical way of broadening the analysis beyond the concept of environment will be to look back at the contextual factors (both formal and informal) that have shaped CSO strategies and outcomes as part of the case studies. The case studies will revisit the analysis of the enabling environment prepared during the scoping studies in order to identify the formal factors that have determined the space for engagement in policy dialogue (using Checklist 2). Furthermore, the case studies will identify any additional factors that have affected CSO strategies and outcomes (using Checklist 3).
The identification of factors that have affected CSO engagement in policy dialogue will be a major element of the case study analysis. Naturally, this part of the analysis will be done in conjunction with the analysis of CSO strategies and outcomes. Key factors will be identified through CSO focus group discussions, using participatory tools, such as SWOT or force field analysis. Based on our initial understanding from documents review and scoping studies we have identified key factors explaining CSO effectiveness in policy dialogue. Our preliminary understanding is that CSO effectiveness is determined by a number of factors, some of them are external, and others are internal. Checklist 3 presents key factors for consideration during the case studies, some of them directly linked to the “enabling conditions” (space, government attitude); others are CSO-related factors (CSO legitimacy, capacity and networks). The case studies will use these (and any additional factors identified during the study) to identify which factors are key for CSO effectiveness and integrate them into the theory of change for a given policy area.
Checklist 3 Factors explaining effective CSO engagement in policy dialogue
|Factors affecting CSO engagement in policy dialogue
||Questions for case study analysis|
|Factors relating to the enabling conditions:
||What are the key factors influencing whether CSO engage in policy dialogue (EQ 14)?|
What are the main enabling and constraining factors that affect CSO engagement (EQ 15)?
To what extent have DP support strategies addressed these factors (EQ 15)?
Spaces for policy dialogue
- Transparent, accessible and inclusive space
- Regular and systematic opportunities for participation, covering all stages of policy process
- Shared principles, including recognition of the value of each stakeholder group’s voice, mutual respect, inclusiveness, accessibility, clarity, transparency, responsibility and accountability
- Attitudes and behaviour
- Capacities, skills and knowledge
|Factors relating to the policy process itself:|
Policy issue and process:
- Nature of the policy issue (e.g. how controversial)
- Timing of policy process
- Access to information
|CSO internal factors:|
CSO legitimacy, capacity and networks
- CSO strategic clarity and focus on opportunities
- CSO capacities, funds and knowledge
- CSO Strategic alliances and networks
- CSO sound evidence and analysis
- CSO legitimacy
3. Establishing linkages through case studies
3.1 Towards a “practical” theory of change for case studies
The scoping studies have established the main conceptual building blocks; in the following, the main study will interrogate the linkages between CSO strategies on policy dialogue and policy change outcomes through a case study approach.
The purpose of the case studies will be to provide an in-depth analysis of how CSO strategies have contributed to policy outcomes. One challenge in measuring influence through policy dialogue is that organisations often claim to be influential (also to justify the support they receive) and that the evidence to support these claims often relates to low-level outcomes or even outputs. Furthermore the very nature of policy work, involving multiple interventions by numerous actors and a wide range of external factors, complicates the analysis of causality and attribution. It will therefore be critical to establish plausible links between CSO strategies and policy change. This will be done through a “practical” theory of change for each policy area, which we will develop through a participatory process involving various stakeholders and sources to enable crosschecking and verification.
The theory of change is a technique to structure our understanding how CSO strategies have contributed to policy outcomes. As a visual tool the theory of change depicts the pathways that lead from specific activities of individual CSOs to wider policy changes, thus establishing causal linkages through interactive stakeholder analysis.
Figure 5 Linking strategies to outcomes through a “practical” theory of change
A major aspect in developing the theory of change is to test the plausibility of perceptions (and claims) around policy dialogue outcomes, using a two-way approach:
Working forwards from strategy to outcomes: We review CSOs and their achievements vis-à-vis objectives and any evidence on outcomes achieved. This will be done through meta-analysis of the available data in CSO reports, using the checklist on outcome indicators above (see Checklist 2). Claims about outcomes and impacts made in the documentation can be cross-checked through interviews and focus group discussions. However, where documentation is limited, the use of other techniques, such as Appreciative Inquiry, can be used to inquire into the aspiration of CSOs and pathways towards achieving those. To triangulate CSO self-perceptions with other sources, we will conduct short “reality checks” by visiting other organisations, communities etc. as feasible and appropriate. Through participatory analysis the team will assess what issues led to identified policy changes by a process of tracing and uncovering the steps through which outcomes have been generated, exploring how and why decisions or practices were executed and what the role of the different stakeholders were in that process. This will be done through the process analysis tool.
Working backwards from impact to outcomes: This means we identify key policy changes (impacts) and identify the role that CSOs have played in it. As a first step we will review the available literature (studies, evaluations etc.) to establish wider policy changes. We will then interrogate any linkages between those changes and the outcomes that CSOs have achieved through group discussions, which involves a wider range of (CSO and non CSO) stakeholders, including representatives from government, think tanks etc. Force field analysis will be a useful tool to understand the dynamics of change and the role different actors have played in it through a process of interactive analysis.
3.2 Outcomes of policy dialogue
For the case studies it is important to break down the concept of influence into (intermediate) outcomes from specific CSO strategies that can already be observed and long-term policy changes. Intermediate (process) outcomes are important to trace CSO influence in policy dialogue. In some cases it may be possible to link policy changes, like the adoption of new policies or the implementation of policies, directly linked to CSO inputs, e.g. through provision of policy papers of proposals that have been taken up. In other cases, CSOs only had an indirect influence, e.g. through framing issues or raising awareness through media campaigns. However, in most cases it may only be possible to measure the intermediate (process) outcomes of CSO strategies that will eventually lead to more effective engagement in policy dialogue. Intermediate outcomes leading to more effective engagement of CSOs in policy dialogue include strengthened organisational capacity, strengthened alliances and strengthened base of support.
The checklist below will serve as guidance for the identification of (intermediate and policy change) outcomes through the case studies.
Checklist 4 Measuring influence – Possible outcomes of CSO engagement in policy dialogue
|CSO intermediate (process) outcomes
||CSO inputs into policy dialogue
Strengthened organisational capacity
- Improved management including transparency and accountability
- Improved capacity to communicate messages
- Increased voice and demands for accountability
- Increased participation in civil society-state space
- Increased number of partner supporting an issue
- Improved level of collaboration
- Improved harmonisation of efforts
- Increased number of strategic alliances
Strengthened base of support
- Increased public involvement in an issue
- Changes in voter behaviour
- Increased media coverage
- Increased awareness of messages among specific groups
- Increased visibility
Direct Inputs into policy dialogue
- “White papers”
- Policy proposals
- Lessons from pilots projects
- Policy briefings
- Watchdog function
Indirect inputs into policy dialogue
- Setting an agenda
- Framing issues
- Media campaign
- Policy development
- Policy adoption
- Policy implementation
- Policy enforcement
Shift in social norms
- Changes in awareness of an issue
- Changes in perceptions
- Changes in attitudes and values
4. The case study approach
4.1 Process for case studies
The advantage of using case studies for this evaluation is that they will enable an in-depth and contextualised analysis of complex concepts and linkages surrounding CSO engagement in policy dialogue by focussing on a specific policy area. Case studies tend to take a more open approach which allows factors and issues that are not anticipated or well understood at this stage to be explored. The evaluation will conduct two to three case studies in each country. The case study approach needs to be flexible and adaptive, based on the conceptual framework outlined above.
The case studies will make use of existing documentation to the extent possible; however, we expect that the linkages will mainly be assessed on the basis of information derived from stakeholder interviews and focus groups. Analysis therefore needs to be systematic and involve steps for crosschecking and verification.
The case study process will used nine basic steps which are illustrated in the figure below.
Figure 6 Process for case studies
4.2 Principles for data collections
Triangulation: Time and resources for the country studies are limited. The teams will need to focus their efforts on capturing a variety of data sources on each topic and triangulate findings between different resources and perspectives to the extent possible. The main data sources that will be consulted include the following:
- CSOs working within the policy areas: The selection of CSOs for case studies will include different types of CSOs (national, local, networks, CBOs etc.) and CSO strategies (as identified through the typology above). CSO own documents and reports will provide evidence on their strategies, the activities conducted and any results achieved. Gaps within the written documentation will need to be filled in through CSO oral accounts. Focus groups with CSOs selected as case studies will help to identify the key enabling and disabling factors that have led to their success or failure. These findings must be crosschecked through consultation of other sources, such as those listed in the following.
- Other civil society actors engaged in the policy area: Representatives from movements, associations, self-help groups, campaigns etc. will be a valuable source for gaining additional insights on how the existing space for policy dialogue has been used by other organisations. These sources should be used to the extent possible to triangulate findings from case studies, in particular with regard to the enabling and disabling conditions. In addition, journalists and parliamentarians with a good knowledge of the policy area should be consulted as source of information and for verification of findings.
- Members of CSO constituencies should be consulted where possible to clarify issues around case study CSO strategies, in particular with regard to questions around CSO accountability and legitimacy.
- Independent think-tanks and experts with a specific knowledge of the policy can provide analysis into what has been achieved (outcomes) and what the key barriers have been. They may also have (independent) views on what the achievements of different types of CSOs have been. The team will identify academics and/or consultants as resources persons.
- Government departments at central and local level with specific responsibilities within the policy area can provide (written and oral) information to verify outcomes on policy changes (e.g. budgets that have been revised; decisions that have been taken; plans that have been developed through a consultative process). The team should in particular look out for those in charge of innovative government initiatives that are likely to spearhead future policy change. In addition visits to government department might be required to cross-check CSO information on barriers resulting from government action. (Government laws and regulations contributing to the enabling and disabling conditions have already been reviewed as part of the scoping studies, but the team might identify additional documents in relation to the selected policy process.)
- Donors and International NGOs will be consulted not only as stakeholders for this evaluation, but also as a source of information. They may have undertaken previous analysis on certain policy issues already and they probably have a good overview of who the main actors are, which can guide the selection of CSOs for case studies.
- Media reports and websites are also an important source to consult during the preparation of case studies.
Any additional sources will be identified for specific policy areas as part of the case study preparation.
Selectivity: Because of the limited time and resources available the team needs to be selective in the way it uses different sources. Selectivity means that the team has to be conscious what the minimum amount of sources is to allow qualified findings. The implication of this is that the quality and utility of individual sources must be critically assessed and potential biases be addressed.
Spread: What the available sources are will depend on the country and policy issues. Whatever the sources are, it is important to ensure a good spread across a variety of sources, geographical, social, economic and political. Within the short time available a good spread can be achieved through careful selection of informants (during preparation), use of online communication tools (Skype) or phone interviews and use of focus groups.
Innovation: The teams should be innovative in their approach to data collection, look outside those data sources that have been well covered by previous studies and consult people, organisations and initiatives that may bring in a fresh perspective and add new insights.
Labour division: For each team, team members will spread out to cover different policy areas. There will be similar issues cutting across several policy areas (such as the enabling and disabling conditions) where team members will be able to collect data from different sources (and cross-check their findings.
4.3 Analysis, crosschecking and verification
The final analysis will bring together the various elements of the case studies, establishing a plausible link between CSO strategies, policy dialogue and outcomes. As part of the final analysis the evaluators will use analytical tools, such as power cube and policy cycle tool, to analyse the various elements that contribute to CSO effectiveness. The power cube will be used to analyse the inclusiveness of spaces for policy dialogue; the policy cycle tool to determine which entry points CSOs have used to influence policy dialogue. The analysis will be shared and further deepened during the final verification workshops, which will include a wider range of stakeholders, including representatives from government, media, INGOs, parliamentarians and academics. During the final verification and feedback workshops the team will also present their theories of change for the selected policy areas for verification by a wider group of stakeholders.
 The tool will also be used at the country level, but with a perspective of synthesising findings per donor at HQ level. The tool will focus on the six donors participating in this evaluation.
 OECD 2010: CS effectiveness and adapted from ODI 2006. Policy engagement – How CS can be more effective.
 OECD 2010: Civil society effectiveness.
 Based on Advisory Group 2008, p 17-18; Jacqueline Wood & Real Lavergne. 2008 Civil Society and Aid Effectiveness.
 Adapted from Jacqueline Wood and Real Lavergne. 2008. Civil Society and Aid Effectiveness
– An exploration of Experiences and Good practice, p. 11; ODI 2006. Policy engagement
– How CS can be more effective, p. 15-16.
 Adapted from Jane Reisman et al. A guide to measuring advocacy and policy, Organisational Research Services, 2007.
This page forms part of the publication 'Support to Civil Society Engagement in Policy Dialogue' as chapter 14 of 20
Version 1.0. 03-01-2013
Publication may be found at the address http://www.netpublikationer.dk/um/11191/index.htm