“The world cannot be understood without numbers. But the world cannot be understood with numbers alone.”

– Hans Rosling, Factfulness

In today’s world, we hear phrases such as big data, algorithms and machine learning everywhere. They follow us in our private life as much as in our work. And more and more, people assume that the more data we have, the better informed we are and the more we know.

When asked, people assume that data is just quantifiable and numerical data, like that collected from survey responses. But data can be so much more, including interview transcripts, photos, videos, audio recordings or other multimedia.

We at Arqaam cannot say it better than Hans Rosling did in his latest book: we need numbers to explain the world, but numbers are not enough. Numbers cannot provide us with comprehensive context or depth. They give a broad overview or specific details, but quantitative data always has limitations. For instance, quantitative data usually comes from questions with prescribed answers that we either (1) provide directly while conducting the survey or (2) group into given categories based on our perspective. This means that quantitative data can miss useful outlier information or important small differences between responses. This is where qualitative data has a crucial role. While qualitative data cannot be representative, it provides critical insights and fills in the gaps left by quantitative research.

For an example of the role of qualitative data in action, take a project Arqaam was commissioned to conduct in 2018. This project, an evaluation in Eastern Afghanistan, was intended to assess to what extent local and national conflict resolution and peace processes have been built or strengthened through the promotion of women’s rights, participation and decision-making power at the community level and reducing gender-based violence.

Eastern Afghanistan is one of the most conservative areas in an overall conservative and restrictive country, and therefore women are not part of many decision-making processes. This project aimed to implement directives from the UN Security Council resolution 1325 on women, peace and security, within which for the first time the UN acknowledged that women have to play a major role in peacekeeping operations. One way to achieve this is to support their participation in all forms of decision making and governing throughout all levels of society, which is what this project supported.

If Arqaam only looked at the quantitative numbers to measure success in this evaluation, we would have determined that the project had a medium impact according to project indicators. However, women were still not really participating in local decision making, and gender-based violence continued to be a major issue that women could not talk to anyone about. But Arqaam went a step further and used the Most Significant Change research tool to get more in-depth qualitative data.

With the help of ImpactMapper, Arqaam analysed 20 interviews conducted with women from the targeted communities and learnt that the project had a significant impact on an individual level to each one of the interviewees.

Half of the success stories were related to marriage in some form and indicated that some women were no longer as oppressed in these relationships.

For example, a 12-year old girl could continue her education instead of marrying someone three times her age in order to settle her parents’ debt. Likewise, there were two women who were not exchanged for marriage between two fighting families, as would have been the norm in this situation. Other women could continue their education after primary school or stopped being beaten by their husbands. While the project failed at some of the key goals on a larger scale, it impacted individual lives significantly.

Through ImpactMapper, Arqaam was able to showcase the successful results in these 20 stories. Using the tagging function, we created broad categories, such as marriage, gender-based violence, inheritance, and education. We further created tags to link the stories to project activities referred to by the women. Lastly, we used the quote function of ImpactMapper to highlight the relevant comments from the women.

The combination of these powerful tools let us see connections and trends in the interviews we conducted and create a compelling narrative on the impact of the project. Using ImpactMapper allowed Arqaam to rigorously review qualitative data and use it to provide supporting context and depth to this evaluation. It provided a level of understanding that we would never have achieved through statistical analysis of quantitative data alone.

This brings us back to the quotation at the beginning of this post: numbers are needed, but they should not be relied on exclusively. Without the qualitative data we gathered, we would not have uncovered these positive but unexpected benefits. Conducting quantitative research alone may miss key insights, thus resulting in providing findings that don’t capture that realities on the ground. In order to serve the communities and international development sector better, researchers and consultants need to prioritise both quantitative and qualitative data working in tandem. With this, the full stories of the communities become clearer.

–

This guest blog contribution was written by Ann-Katrina Bregovic, a Director and Founder of Arqaam, a social enterprise that helps improve the impact of humanitarian and development actors through better use of data.

‍