Alternatives to the Conventional Counterfactual: Summary of Session 713 Think Tank - American Evaluation Association, Orlando 2009

This note provides a summary of the main points raised during the discussion on alternative to conventional counterfactuals. The Think Tank had three objectives: to ask how project or programme outcomes and impacts cam be assessed when it is not possible to construct a statistically matched control group; to consider the real-world challenges of using comparison group designs and; to share experiences on promising approaches for assessing outcomes without a statistical comparison group. The purpose of a counterfactual in evaluations is to address the question "What would have been the situation of the project population if the project had not taken place." One method for answering that question is to conduct a Randomized Control Trial (RCT) or to use quasi-experimental designs with a pretest/posttest comparison of the project and a matched control group. However, the complexity of many community interventions, RealWorld budget, time and data constraints, and the fact that many evaluations are not commissioned until late in the project cycle, mean that conventional statistical evaluation designs are often not feasible and alternatives to the conventional statistical counterfactual are an essential tool in the evaluator¡¦s arsenal. The increasing demand on evaluators to assess general budget support and complex, multi-donor, national-level programs where project level evaluation designs can rarely be applied further increases the need for alternative ways to assess the outcomes and impacts of these complex programs.