7 questions to consider when switching to phone surveys with crisis-affected people

07 October 2020

Covid-19 has upended the way humanitarian assistance is provided to affected people around the globe and communicating with those in need in the midst of a pandemic is no exception. The focus on social distancing and travel restrictions has so curbed access to vulnerable populations that nearly all humanitarian contexts now face similar challenges as hard-to-reach conflict settings. Remote monitoring of aid in such situations has always been fraught with problems, particularly if the goal is to go beyond simply validating aid delivery to actually hearing from the intended beneficiaries.

In remote management mode, it would be tempting to switch to phone surveys from face-to-face interviews. Phone surveys reduce the need for complicated travel logistics and field access and can be cheaper and easier to scale across multiple locations. Instead of worrying about enumerator teams working in refugee camps or remote villages, questions can be sent to a phone survey provider to collect and deliver a clean dataset. After all, aren’t people in places from Afghanistan to Zimbabwe increasingly connected to the mobile network? In our modern world, why even do face-to-face surveys when you can digitise and innovate?

Ground Truth Solutions has been championing perception surveys in the humanitarian sector for close to a decade, so it would be far easier to follow this narrative and pivot our approach to phones. In our experience, however, phone surveys are generally a poor substitute for face-to-face encounters in humanitarian settings. Identifying respondents and understanding their point of view is harder when you can’t meet them in person. Attention spans evaporate when people are asked questions on the phone rather than face-to-face. Facial expressions and hand gestures add important nuance to information that cannot be conveyed over the phone. Without these silent clues there’s also the potential for greater misunderstanding.

Such constraints require a fundamental rethink of how to best communicate with affected people instead of resorting to phone surveys. While useful in some instances, we still put a lot of effort into determining whether it makes sense to survey remotely. To continue learning from people affected by crisis while maintaining a distance in these uncertain times, the following questions may be useful for you, too:

1. Who is your target population and can you reach them by phone?

Are you interested in the view of the general population? If so, reaching them via phone is probably fine, depending on network coverage and phone ownership in your country of interest. But the same does not necessarily apply if you are interested in the views of displaced people, refugees or recipients of aid. For these populations, you will have to disregard phone ownership data, because national ownership rates are inflated by owners of multiple phones among the middle class. Access to a cell phone is drastically lower in vulnerable communities.

2. What share of the overall population does the group you are interested in constitute?

If it is small, it does not make sense to use a national phone registry or randomly generating phone numbers from a particular provider. You will need a list of numbers for only this sub-group.

3. How many people would you need to call to get one person of interest in your sample, and can you manage the bias arising from this?

In a typical polling survey in the United States, 94 percent of calls go unanswered or get rejected. Data on response rates in humanitarian settings is scarce, but they are unlikely to be much higher in settings where there is poor phone reception, people are busy with daily chores and where robocall spam is on the rise too.

4. Where will you get phone numbers from? Do you have a sampling frame or phone numbers to randomly choose from?

Your survey provider of choice may have their own database of respondents. If so, make sure it is not a black box. Do you know exactly who is on the list and why? Has the list been used for other purposes?

People may receive a lot of calls because survey firms use their coordinates for multiple clients. Is there any sort of remuneration for being on this list, or for responding to a call? If so, what are the incentives and could they skew your results? Can you manage potential biases inherent in the list? No matter how randomly respondents are selected, any such biases will be replicated in your sample.

5. If you determine that a phone survey is the right way to go, have you redesigned your survey instrument accordingly?

Are your questions short and easy to understand, ideally all using the same scale or simply all formulated as yes/no questions? Can the survey (including the introduction) be completed in 10-15 minutes? If not, shorten it. If that’s impossible, consider splitting your survey in two.

Are the questions clear without the context of a field survey? For example, if you ask someone what type of aid they received or what type of house they live in, they may simply show you. Think about how you can compensate for this and avoid misunderstandings on the phone.

6. Have you vetted your phone survey provider?

At a minimum, ask the provider about their data quality checks. These should include documentation on training of phone operators; testing of the survey; ongoing interview monitoring; verification of interviews after they have been conducted; and regular supervisor meetings to provide feedback to phone operators on what they can improve.

Once you have started to work with a phone survey provider, make sure the data includes interview start time, end time, and date; enumerator name or ID; and data on all calls, successful or not, consented to or not. Ideally, this information should be provided while the survey is underway so problems can be identified and addressed in real time.

7. Did you debrief your phone survey provider?

After you have received the data, make sure to debrief with your phone survey provider to hear any observations from the survey process that may impact your analysis.

Phone surveys still have an important role, but not in all circumstances

 

The long-term implications of the pandemic on the humanitarian sector are still unknown, but what is clear is that it is challenging everyone to find creative solutions, including those who want to hear from affected people. However, it is important to be aware of the beguiling simplicity of digital solutions. Phone surveys still have an important role, but not in all circumstances – and certainly not without asking some hard questions as to whether they are fit for purpose.

Do you have experiences with surveying people affected by crisis that you think others could learn from? Write us at info [at] groundtruthsolutions [dot] org.