Breaking the evaluation rule book: how do we learn in the midst of a global pandemic?

16 December 2021

The COVID-19 Pandemic propelled humanitarian organisations to improve the way they learn while also adapting and responding to rapidly evolving crises. The need for timely learning resulted in a renewed interest in Real-Time Evaluations (RTE) and other rapid learning exercises. But how do you carry out an RTE in multiple countries simultaneously, in rapidly evolving global and national contexts, in the midst of a crisis that affects every aspect of how your organisation functions?
 

There is surprisingly little written about how to conduct rapid learning exercises at a global scale or how to plan multi-country RTEs. Although there was no evaluation rule book and little guidance on how to design these exercises or where to start, ALNAP members set out to document learning and quickly identify what was and was not working, using a wide range of innovative and timely learning exercises. Of the many obstacles and questions along the way, a common internal debate was what to call such exercises. Would they be evaluations or rapid reviews? Would they fit the formal definition of RTEs or were they a new form of evaluative learning? 
 

In practice, many were both, being Real-Time in nature, but using new formats and definitional frameworks. Utilisation-focused learning approaches during COVID-19 have ranged from more traditional RTEs with a global scale (or multi-country RTEs) to lighter-touch exercises such as Real-Time Assessments and adaptive management reviews; to more developmental-style evaluations – all of which are proven to be suitable evaluation exercises in evolving contexts. While many of these exercises share similar features, there is ‘definitional anarchy’ as to how to label them.  To end this anarchy, ALNAP’s latest  paper ‘From Real-Time Evaluation to Real-Time Learning: exploring new approaches from the COVID-19 response’ coins a new term –Real-Time Learning (RTL) – which we define as:

a range of evaluative approaches, reviews and assessments with the purpose of understanding and articulating issues that need to be addressed in an ongoing development or humanitarian response, that can be fed back immediately into programming, decision-making and management processes with the overall aim of improving the response.


How does Real-Time Learning relate to the tradition of Real-Time Evaluations in the humanitarian sector?

In 2009, ALNAP published the most widely used guidance on RTEs, defining them as: ‘an evaluation in which the primary objective is to provide feedback in a participatory way in real time (i.e. during the evaluation process) to those executing and managing the humanitarian response’. RTE’s are unique because they are not only retrospective, but also prospective (future-looking). They tend to be more rigorous and formal than Real-Time Reviews and Real-Time Assessments and are generally less internal than Adaptive Management Reviews and After Action Reviews. However, there are similarities and overlaps in some of these exercises. Indeed, for ALNAP, Real-Time Evaluations, Real-Time Assessments and other forms of rapid learning reviews with short feedback loops all fit under this new broader category of Real-Time Learning.
 

RTL can fill the gap between monitoring and more traditional evaluation approaches. RTL exercises can be used to share implicit individual and team knowledge, and to document and share learning across an organisation (capturing experiences at country, regional and HQ level) via interactive workshops and sense-making sessions.

In addition to providing a new definition, our paper reviews over a dozen RTL exercises conducted by ALNAP members since the start of the pandemic and identifies 12 key success factors for Real-Time Learning (see image at the end).

By highlighting the approaches and methods used in recent Real-Time Learning exercises by UNICEF, WFP, FAO, IFRC, Oxfam, World Vision and others, the paper also provides tips and examples of how to design, plan and implement effective RTL exercises both for COVID and for future complex multi-country crises.

If you are considering conducting a Real-Time Learning exercise and you would like to learn more, you can now read the paper ‘From Real-Time Evaluation to Real-Time Learning’. To share your reflections and experiences of Real-Time Learning, or to contribute a blog on RTL, contact us at alnap [at] alnap [dot] org.


'Real-time success factors' figure from ALNAP's paper 'From Real-Time Evaluation to Real-Time Learning'.