Royal danish ministry of foreign affairs - Go to the frontpage of um.dk   Publication  
 
 
     
 
 

Annex C Evaluation of Support to Civil Society Engagement in Policy Dialogue – Evaluation Framework (revised November 2011)

Evaluation questions Indicators (specific evidence required) Data sources and methods for data collection Reporting format
2. Enabling environment for CSO engagement in policy dialogue and key changes over the past five years within case study countries?

2.1 The legal, political and financial freedoms of CSOs and how they have changed over the last five years

2.2 The relationship between government and civil society in practice – including the power dynamics at play and how this has changed over the last 5 years

2.3 Key issues determining the enabling environment
Analysis of factors that contribute to CSO effectiveness in the country context

Changes of the last five years

Analysis of power relations and how these affect the space for policy dialogue

Use Checklist 3!
Country documents describing the legal and political framework for CSOs.

Existing analysis of enabling framework (from DPs, think tanks, CSOs)

CSO feedback on enablersand constraints
Country report (revised from scoping study)
Level 3 (Case studies) – CSO effectiveness
CSO strategies:

6. How do the CSOs (selected for case studies) engage in policy dialogue (within the chosen policy areas)?

6.1 What strategies are used by CSOs to achieve their objectives on policy dialogue?

6.2 What is the scope of policy dialogue? What does it cover?

6.3 To what extent do CSO use networking or cooperation with other CSO as part of their strategies? Is there an advantage in having joint NGO platforms or does this rather dilute their impact on agenda setting?

6.3 What is the intervention logic behind the CSO strategies/approach? What do they want to achieve and how?
Types of CSO strategies (see Q4) (Use Checklist 1)

Theories of change for case study CSOs (Phase 2)

Analysis of policy dialogue space as part of the case study (Phase 2)

Whether NGO networks and platforms are effective for policy dialogue
CSO strategy documents

CSO focus group discussions
Country report
Accountability and Legitimacy:

7. To what extent is CSO engagement in policy dialogue supported by their mandate?

7.1 Whose interests do CSO engaged in policy dialogue represent? How do they obtain legitimacy?

7.2 To what extent are CSOs engaging in policy dialogue accountable to their constituencies? How transparentare CSO procedures and operations? What are the feedback mechanisms?

7.3 How do CSOs obtain legitimacy to speak for the people they claim to serve or represent? To what extent are CSOs’ political demands supported by “numbers” (size of constituencies)?
Whether the CSOs’ mandate supports engagement in policy dialogue

Whether there the CSOs are accountable to their constituencies on their engagement in policy dialogue

Whether the CSOs have established feedback mechanisms with their constituencies

Whether CSOs have the “critical mass”to support their political demands

Whether CSOs present the interests of poor and marginal groups
CSO mission statements

CSO institutional visits and interviews

Cross checking through interviews with groups representing CSO constituencies

Cross checking through interviews with independent thirds (e.g. think tanks, parliamentarians)
Country report
Results (Process outcomes and policy changes):
11. How effective are the CSOs in asserting influence on government (at national and local level) through policy dialogue? How effective are CSOs in influencing policy change? How effective in holding government accountable for policy the implementation?
Extent to which policies changes occurred in selected policy areas

Evidence that CSOs have been substantially engaged in policy dialogue

Evidence that CSOs contributed to policy change in selected policy areas

Evidence that CSOs are holding government to account for the implementation of policies

Use Checklist 2!
Review of policy outcomes documented by CSO

Review of available analysis of policy processes (DP sources and evaluations; independent research and studies; media)

Stakeholder workshop (including government representatives, think tanks, parliamentarians, other relevant organisations etc.) to review policy change and contributions
Country report

Separate documentation of process outcomes and policy changes (with evidence)

Documentation of CSO workshop

Documentation of stakeholder workshop
Results:

12. How effective are the CSOs in achieving their own specific policy objectives?
Evidence that CSOs achieve their stated policy objectives

Cases where CSOs failed to achieve their objectives (and why)
Review of results documented by CSOs

CSO focus groups (workshops), using process analysis, theory of change.

Crosscheck findings through stakeholder interviews/ workshop
Country report

Documentation of CSO workshop

Documentation of stakeholder workshop
Results:

13. What were the unexpected results of policy dialogue?
Evidence that CSOs have achieved results beyond their stated policy objectives Same as Q11 Same as above
Enabling and disabling factors:

14. What are the factors influencing whether CSOs engage in policy processes or not? Why are some CSOs who – given their constituency and profile could engage in policy work – not doing so?
Key factors (drivers, breakers) influencing CSO engagement in policy processes

Practices that have enabled CSO outcomes in policy dialogue
CSO workshops using toolssuch as

– power cube
– SWOT analysis
– Force field analysis

Synthesis of key factors determining outcomes of CSO engagement
Country report

Documentation of CSO workshop
Enabling and disabling factors:

15. What are the main enabling and disabling factors that affectthe relevance and effectiveness of CSOs in policy dialogue, both in general and in relation to CSOs own goals and objectives? (e.g. what role do aspects of the enabling environment, CSO capacity, resource constraints and degree of networking play?)
Analysis of enabling and constraining factors affecting CSO strategies and results

Use Checklist 4!
Same as Q 15 Country report

Using separate template on enabling factors (from scoping study
Level 4 – DP support on policy dialogue (country level)
DP support:

17. How responsive are DP strategies to the priorities of the CSOs and what role did this play in the effectiveness of CSOs?
Lessons (and examples) on responsive support; Lessons (and examples) on responsive support: what worked and what didn’t? Feedback from CSOs and other stakeholders (Country web survey) Country report

(Feedback form/survey for synthesis)
DP support:

18. What value do specific support strategies add? In particular, what value does support provided through different channels (Northern CSOs, local CSOs) add? What value does DP engagement in policy dialogue add?
Whether DP strategies support diversity of CSOs

Whether DP strategies support learning, innovation and focus on results

Whether partnerships with Northern CSOs provide opportunities for networking, dialogue and information sharing?
Feedback from CSOs and other stakeholders (Country web survey) Country report

(Feedback form/survey for synthesis)
DP support:

8. What is the relevance of DP support vis-à-vis CSO priorities on policy dialogue?

8.1 What do DP perceive as the main needs of CSOs, particularly in relation to policy dialogue?

8.2 To what extent has DP support been driven by CSO demands?

8.3 To what extent does DP support respond to changing conditions for policy dialogue? To what extent is DP support informed by sound contextual analysis?

8.4 To what extent do DPs pursue their priorities through support of CSO engagement in policy dialogue (whose agenda)? Or where relevant: do what extent do Northern CSOs pursue their agenda through cooperation with local partners (who sets the agenda)?
Whether DP support is aligned to CSO priorities (priorities evidenced through CSO internal strategies, planning and communication)

Whether DPs are responsive to CSOs demands

Evidence of DP analysis and response to changing framework conditions

Cases where CSO priorities changed in response to DP support

Whether DPs (or Northern CSOs) pursue their strategic priorities through CSO support in policy dialogue
DP and CSO interviews

(Country web survey)
Country report

(Feedback form/survey for synthesis)
DP support:

16. To what extent have DP support strategies addressed the enabling and constraining factors that CSO face?
Evidence that DP strategies have addressed those enabling and constraining factors Feedback from CSOs and other stakeholders (Country web survey) Country report

(Feedback form/survey for synthesis



This page forms part of the publication 'Support to Civil Society Engagement in Policy Dialogue' as chapter 15 of 20
Version 1.0. 03-01-2013
Publication may be found at the address http://www.netpublikationer.dk/um/11191/index.htm

 

 
 
 
 
  Ministry of Foreign Affairs of Denmark © | www.um.dk