Halfway there! EHA Guide pilot so far: who is taking part and what we’ve heard

ALNAP has long been developing and piloting guides for humanitarian practitioners. There was the Participation Handbook for humanitarian field workers (ALNAP and URD, 2004), the guide on Evaluating Humanitarian Action using the OECD-DAC Criteria (Beck, 2006), and the Real-time Evaluations of Humanitarian Action - an ALNAP Guide (Cosgrave, Ramalingam and Beck, 2009), for example. However, with our Evaluation for Humanitarian Action Guide (EHA) is the first time that ALNAP takes on an active pilot for this type of guidance material.

The EHA pilot Guide, co-authored by John Cosgrave and Margie Buchanan-Smith, benefitted from the inputs and steer of an inter-agency advisory group that brought together evaluators and programme staff from donor agencies, INGOs, the Red Cross Movement, UN agencies, and academics.

However, we also realised that in order for the Guide to be more practical and utilisation-focused, we also needed to incorporate inputs from users – starting with evaluators and M&E staff from within the ALNAP Network.

But why an active pilot? And what does that even mean?

When I was first asked to help with the coordination and monitoring of this process, I was unsure of the purpose of this adjective. However, after speaking to some of the authors of the guides mentioned above as well as other partners, it became clear that pilots can be very passive (some may even say to the point of dormancy). Some guides are labelled as pilot versions, with an invitation to share feedback via email or to fill out a survey, yet, little energy is put into going out and gathering users’ experiences.

So, what did we do to actually capture this richness so that it may be later incorporated into the final guide?

Don’t get me wrong, we received an enormous amount of positive feedback on the Guide (great resource, what a pretty colour scheme, cool interactive PDF) and are very thankful for these, but we wanted more. We sought to know what users liked, disliked, what we missed and what could be elaborated. This is why we put in place a slew of ways to submit feedback, from (my favourite) confidential feedback buttons at the bottom of every page of the guide through brief interviews with users to (the most interesting) ALNAP shadowing a member’s evaluation process.

Now at the half-way mark of the pilot, we have 24 organisations, 16 of which are from the ALNAP membership, actively providing feedback. Excluding positive, but non-constructive comments, we have received over 250 pieces of feedback — in terms of content specific contributions: 122 were directly shared by piloters and 62 were gathered through discussions with piloters.