Monitoring and evaluation (M&E) and impact measurement frameworks often rely on predefined quantitative metrics. These can be helpful to track and report on social change in a simple, standardized approach. However, change trajectories are far from simple, standard or predefined. As a result, standardized quantitative indicators alone can often miss out on the nuances of an organization’s impact.
In many cases, overly focusing on quantitative changes also comes at the expense of leaving aside learnings and insights that stem directly from people’s experiences. We're often not hearing from the voices of people and their own experiences and using that to create and track indicators for success.
Organizations usually have a lot of stories but these are not systematically leveraged for M&E and reporting. Used properly, these stories can help us understand the complex realities and impact trajectories through human voices. Stories can provide us with meaningful information for decision making.
Our Founder, Alexandra Pittman, gave a webinar in February sharing a diverse set of examples, from Azerbaijan, the Balkans, to the US on how to leverage storytelling and impact stories as a dataset to understand social impact. Sign up here to receive the video recording of the event and read further for some high level tips that were shared.
We have 3 practical tips to start using story data more effectively:
- Define the sample and data collection strategy. Collect stories of impact, grounded in people's realities.
Rigour, feasibility and practicality need to underpin qualitative approaches to M&E. Before collecting stories, it is important for organizations to step back and plan. Organizations need to design a sampling and data collection strategy that aligns with their needs and resources to ensure a diverse and representative set of voices are heard. There is no one-size-fits-all approach to sampling or data collection, but it should be mindful of vulnerable groups that are often left out and silenced and ensure safe spaces are created for sharing, e.g., people with less access to resources, people living in rural settings, children, illiterate groups etc.
To obtain meaningful narratives, be mindful of the questions asked. Ensure that they are specific to the role of the intervention being evaluated or the contribution of different groups to a change process. Choosing the right approach can turn data collection into a meaningful personal reflection process, as well as support data collection. For instance, during participatory storytelling workshops, creating a safe space and selecting the right questions can ensure everyone feels safe, engaged, invited and compelled to speak up and be heard. If managed correctly, these workshops can offer a safe space for everyone to share, listen and reflect, empowering participants and organizations.
Sometimes it may also be the case that organizations do not even need to collect stories right away. It is also important to stories that your organization may already be sitting on and not analyzing--these can be hidden in grantee or company reports, emails or even meeting notes!
- Create a story and outcome database and transform voices into indicators to help show aggregate level impact trends
Creating an outcome dataset with the stories that you have collected requires bringing an analytical lens to code stories, connect the dots, and unearth trends.
Organizations need to decide whether they want to follow a deductive, inductive or mixed approach to indicator and metric development. In an inductive approach to coding text or stories, participants create outcome indicators based on changes identified in the stories that are shared. It is a bottom-up approach to indicator development, relying on people's voices to define indicators of success. This approach provides an interesting shift in power dynamics in the field of monitoring and evaluation and impact management. A deductive approach to coding means that you take pre-existing indicators and metrics from your Theory of Change, logframe, your KPIs, or a standardised framework like the SDGs or IRIS metrics. A mixed approach uses both approaches combining deductive and inductive techniques, such as coding any unexpected outcomes that arise from the data and indicators from your theory of change.
The coding process begins by analysing the stories with a focus on identifying the positive or negative changes, shifts, or instances of maintaining past gains described within, as well as as strategies that led to success. To unearth trends, these shifts are then coded and aggregated across stories. This gives the frequency of outcome trends across a range of different stories - providing an actionable dataset that can be analyzed in a variety of ways: over time, across regions within a grant, impact investment, project, etc. You will need to decide how to keep track of this data, in excel or a software tool dedicated to this purpose in the tracking diverse datasets in the philanthropic, corporate social responsibility, and impact investing spaces. Most efficient is to use an impact tracking software tool that allows for mixed-method analysis, like ImpactMapper, where you can ensure rigorous tracking of indicator frequency and access data visualization along with other financial and quantitative metrics.
- Communicate impact trends, and share exemplary stories and more nuanced case studies
Once you have coded the data, you will have a set of trends and changes that are the most frequent across the stories. Story databases provide rich material to produce to communicate social impact and move targeted audiences to action. These datasets can be leveraged to highlight aggregate trends, while illustrating the nuance of particular experiences through exemplary stories and case studies. The key is thinking about who will be reading the report and to communicate that accordingly.
Who are your audiences? Who do you want to move and how? What are some of the most compelling ways to share this data? To ensure stories are heard and acted on it is important to be strategic about communication and reporting.
We hope these tips are useful - if you want to hear more about how you actually use stories as datasets for impact measurement and why this really matters, get in touch! Impact Mapper organizes participatory workshops that shows your organization how use report and storytelling data more effectively and can help you to analyse your story or report data.