Review of Joint Evaluations and the Future of Inter Agency Evaluations

Executive Summary

This is a review of primarily three pilot phase Inter agency Real time Evaluations (IA RTE’s) as described in the Terms of Reference (TOR) in the annexes. The review is to inform the drafting of a note to the Inter agency Standing Committee Working Group (IASC WG) regarding the future of IA RTE’s. The review differs from an evaluation in many respects. It does not apply standard evaluation criteria. The role of the reviewer is to reflect views collected from IA RTE Interest Group (IG) members and from the review field missions. The methodology was comprised of a desk review, interviewing and a workshop .

Summary Findings and Conclusions

1. The pilot IA RTE’s were poorly rated regarding their ‘instrumental’ use i.e. in terms of recommendations implemented and corrective actions taken as a direct result of each IA RTE. Few examples of such direct benefit were encountered during the review. Constraints relate to ownership, perceptions, methods, dissemination/‘communication’ and follow-up of results. Poor direct use is a generic problem of evaluations of humanitarian action.

2. Other, indirect, types of use have been noted, though these are not currently part of the primary objectives of the IA RTE’s. A significant number of examples were found where the evaluations influenced the operations (and those working in them) by means other than the final report. Such less visible use is recognized as being important and in some cases determinant in bringing about change (see examples in the main text).

3. The vast majority of those consulted recognize a need for some sort of real time evaluation mechanism. The potential value of an appropriate IA RTE mechanism is widely recognized, albeit from differing perspectives, be they ‘learning’, ‘accountability’ or both. Broad agreement exists within the IG for a triggering mechanism that is predictable and perhaps automatic in certain circumstances. The need for criteria is widely understood.

4. An equally important majority identify weaknesses in the current IA RTE model. These weaknesses are present throughout the process: IA RTE timing and predictability; concept, approach and methods; and dissemination, communication and follow-up. Weaknesses in the IA RTE process have impacted negatively on direct utilization of the pilot IA RTE's. Addressing weaknesses in an integral and comprehensive manner can lead to an effective IA RTE mechanism. Failing to address them comprehensively will probably result in a continued lack of satisfactory results and return on investment.

5. Costs and approaches of the IA RTE’s are more in line with ‘heavier’, non-real time evaluations than some single agency RTE’s. Interviewees were of mixed views regarding the potential for cost reductions, arguing it depends largely on how useful they may ultimately be. Clarity around the concept and process is a prerequisite for firm conclusions. Possible avenues for cost-reduction are described in the report.

6. Alternatives to an IA RTE mechanism were briefly examined during the review . Options which emerged include:
a. Dropping the ‘real time’ objective and doing just ‘IA emergency evaluations’;
b. Offering Country Teams (CT’s) the possibility of choosing between real time or non-real time emergency evaluations based on criteria for an ‘automatic’ emergency evaluation mechanism; and
c. Developing pilot ‘impact evaluations’ for emergency operations.

7. As noted above, most responses favour some type of mechanism to facilitate real time feedback and learning for on-going operations and/or for accountability reasons. The second option (to allow CTs choose the type of evaluation) emerged late in the review and could be examined in more detail. Finally, regarding ‘impact-evaluations’, given the complexity and early development stage of the concept, views were clearly mixed among IG members. The Geneva workshop requested that OCHA refine the concept for further discussion.