Glossary

Impact Assessment Terminology

Causality, how x causes y. Expectations with regards to causality is typically expressed in the intervention logic captured in linkages in the results chain from activities through to outputs, outcomes, and impacts. The causal linkages in the chain together with sound logic underpin and supplement the attribution analysis. Examination of causality also answers questions concerning the relation between outputs and impacts.

Attribution, an assessment of the degree to which impacts can be linked back to the outputs delivered by, and “credited” to, the interventions. At the impact level, attribution is generally accepted to be at the level of some contribution of outcomes in combination with many other important factors. The impacts referred to are those indicated by the double difference estimates described below.

Counterfactual, an assessment of what would have been the situation (the status of impacts) if the intervention had not taken place.

The treatment group, households (HHs) or individuals who have been affected by the interventions.

The comparison group, HHs/individuals who have NOT been participating themselves, but who prior to the intervention possessed similar observable characteristics as those in the treatment group.

Propensity Score Matching, mathematical technique used to select members of the comparison group through estimation of a statistical model based on matching characteristics which relate directly to the probability of participation.

Pipeline Approach, a technique for comparison group selection where the comparison group will be composed of individuals who have been selected (eligible) to participate, but have not (yet) been involved nor have benefited from intervention activities. This assumes that such a pipeline exists, that there has been no change in selection criteria, and that applicants have not been ranked for participation.

Double Difference Measurement, the double difference measures the difference in the observed change between the treatment and comparison group, based on baseline and end-data. Thus the double difference eliminates external determinants of the outcome,
in cases where these are the same for the treatment and comparison group during the intervention period. The double difference approach assumes common time effects across groups and no composition changes within each group.

Contamination, can arise from two sources – “own” and “external”. Contamination can come from the intervention itself due to spill over effects (own contamination). This type of contamination can occur if the comparison group is selected from a geographical area too close to the intervention area. Comparison groups from distant locations can also be contaminated through interventions by other agencies (external contamination).

Selection bias, is bias introduced from the way beneficiaries have been selected for participating in the intervention. When beneficiaries are not randomly selected, but some kind of selection process has taken place, then the comparison group should not be randomly selected either, but rather drawn from a population with same characteristics as the participants group using the same selection criteria.

Statistical significance, in statistics, a result is called statistically significant if it is unlikely to have occurred by chance. In this analysis, the significance level is used to measure the statistical strength of a data finding. The significance level is here the risk of concluding a data relationship that may not exist. Frequent levels of significance used for statistical testings are 10% (0.1), 5% (0.05), 1% (0.01) and 0.1% (0.001). If a significance test gives a value lower than the test levels, the null hypothesis (a hypothesis that an observed difference between two data sets is random/due to chance) is rejected. Such results are referred to as being ’statistically significant’. For example as in this report, if an observed difference between data from a treatment group and a comparison group is found to be significant at the 10% level, it means that the null hypothesis (that the observed difference is by chance/random) can be rejected with 90% certainty. The lower the significance level, the stronger the certainty that the null hypothesis can be rejected. Cases with relatively few observations (data) and large variation, increase the uncertainty and makes it more difficult to reject the null hypothesis.