Impact measurement has become a relevant challenge for changeworkers all over the world. Measuring economic success is easy: you account for income and costs, and calculate your profit from there. But how to calculate social and environmental success, or lack thereof? This is a question that many changeworkers are asking themselves: How to know whether, and eventually “prove” they are creating the change they are striving for?

Impact measurement has deep roots, since the emergence of the Measurement and Evaluation (M&E) of social projects and the rise of public policy evaluation in the middle of the 20th century. Later on, as the nonprofit and philanthropic sectors evolved, there was an increased demand that charitable giving was more than just “good intentions”. Funders started requesting more rigor in the evidence of results that social organizations produced thanks to the grants received. 

From “Impact Measurement” to “Impact Management”

As impact investing, venture philanthropy and corporate social responsibility have risen in the last decades, impact measurement is becoming a more common practice among social entrepreneurs, NGOs and impact funders alike. Gathering impact data helps changeworkers understand the impact that a social project/program is creating, improve its design and execution, report and engage with key stakeholders, and support communication and advocacy efforts.

In fact, the term is evolving from “Impact Measurement” to “Impact Measurement and Management” (IMM). According to Impact Frontiers, “Impact management is the process by which an entity understands, acts on and communicates its impacts on people and the natural environment, in order to reduce negative impacts, increase positive impacts, and ultimately to achieve sustainability and increase well-being.” That is, beyond gathering evidence of change, how do you use this data to make strategic and operational decisions that ultimately maximize impact and change systems?

However, measuring and managing impact effectively is easier said than done. What most organizations end up doing is measure outputs (number of people supported, number of hours of training, € invested…) but aren’t really able to measure long-term impact or systems-level change. Why? 

Why is it so hard to measure impact at a systems level?

While traditional evaluation comes with its own challenges (need for upskilling, lack of consensus on tools, comparability issues, etc.), measuring systems change is harder because systems behave very differently from linear programs:

• Complexity & nonlinearity.

  Mainstream impact measurement relies on linear cause-effect chains (activities → outputs → outcomes → impact), while systems change is nonlinear, emergent and adaptive. The reality of working with systems and systems change means that we have to navigate complexity. Furthermore, there are no pre-set boundaries, or they can change over time, so classic evaluation approaches may not work in these settings. For example, in a systems change health project, boundaries may not be limited to a specific disease-treating program in a hospital, but become broader to include more actors, deeper relationships, and interconnected issues (e.g. the role the education system plays in disease prevention, or also housing, social services, family structures… which may have been excluded from the scope in a “typical”, more linear evaluation). 
   

• Long horizons and multiple levels of change.

  Established evaluation methods focus on short term, program-level outputs rather than long term systemic shifts in policy, norms, relationships or field architecture. Measuring systems change takes more time, because shifts in the system usually take longer to happen. It is unlikely funders will assign budgets to evaluations that go beyond project funding cycles or may be reluctant to invest larger budgets into complex evaluations. A bold, more patient approach is needed from funders that really want to support systems-level change.
   

• Attribution vs contribution.

  The mainstream approach to measuring impact overemphasizes attribution: to prove that someone created a certain impact, instead of contribution, whereas in complex systems it is multiple stakeholders who influence systemic shifts through a portfolio of initiatives, rather than one and only “hero” saving the day.
   

• Power dynamics and structures.

  Impact measurement is deeply embedded in economic, financial frameworks and investment logic (cost-effectiveness, return on investment, accountability), and “often predominantly designed to serve those who pay (financially) for a project or programme”. They focus too much on performance management, which can lead to data corruption (results are distorted to show what the funder wants to see). Instead, evaluation systems would be more effective if they put in place the kind of structures that would support the systems changeworkers in the field, both in terms of process and insights, and involve key stakeholders in meaningful ways, so that evaluations benefit them instead of only “using” them to extract the data.
   

• Measurement of intangible changes.

  To be able to maximize efficiency, assess impact, and compare results, mainstream impact measurement relies on standardized quantitative indicators that can miss critical evidence of systems change (changes in narratives, power, network structure, institutional practice…). Qualitative metrics are harder to measure: intangible things like trust, empowerment, relationships… are more complex than straightforward things like money, time, kilograms, etc.; but provide invaluable insights to systems change.

Then, how can we measure systems change?

“In a world where systems change is often reduced to scale, speed, and metrics, a different story is unfolding—one that centers relational work, inner development, cultural shifts, and collective learning as the true foundations for transformation.”

(Scaling Deep: Shifting Power and Redefining Success by Tatiana Fraser, The Systems Sanctuary)

Systems change is uniquely hard to measure because systems are “living” and are constantly changing. So if we want to measure this kind of work, the way we measure success needs to change as well. 

The way we evaluate a direct social service may not tell the full story when aiming for systems change. For example, counting the number of women who attended a training on financial literacy doesn’t tell you whether this has actually improved gender equity in that community. Measuring systems change requires a different mindset and a new role for evaluators: from external judgement to really support learning in an ongoing way.

Here are some approaches to take when measuring systems change:

1. Co-create what “success” means with stakeholders.

Try to improve things from the perspective of communities, not from the perspective of those trying to help. What does “impact” mean for the people involved in the context? And also, who has a seat at the table to define not only what “impact” means, but also how to evaluate it and define strategies to create more of it? Whose values matter? Value-based, equity-oriented and participatory methods help create inclusive processes rooted in the contexts we work in.

2. Focus on learning. 

Toby Lowe suggests an alternative approach to strategy that starts asking: “how is impact in the world created?”.  With that question in mind, Learning Cycles become the process to answering that question in a constantly changing environment. Instead of focusing on reporting and demonstrating “our” impact to others, we seek to understand the impact of our actions as a necessary part of our practice. Through action research and iterative experiments/explorations, “we can enable data to do its job – to help people to get better at creating genuine impact”.
 

 

3. Don’t look for attribution, but contribution.

Systems change is “a process of collective engagement, learning, and adaptation, and the transformation that emerges from that journey”. Acknowledge that, and work collaboratively towards a shared purpose, instead of trying to prove “your” impact.

4. Combine quantitative with qualitative data. 

According to Ann Murray Brown, “Quantitative data tells us what happened, but not why. Without stories, context, and lived experiences, numbers become hollow.” That’s why combining quantitative scales with complexity-aware methods and proxy indicators can uncover unexpected insights and also increase credibility, as there is triangulation between different sources. Marina Apgar and Emily Gates suggest “Methodological Bricolage” as an alternative approach to systems change evaluation. By recombining relevant parts of different methods, through iterative co-design, creatively and intentionally, we can develop evaluations that are truly fit for purpose.

Tools to measure systemic change

The field of measuring systems change is emerging and changeworkers, as well as systemic measurement practitioners, are experimenting with many different tools to measure systems change work. Some of them are:

• Developmental evaluation / adaptive learning
• Contribution Analysis
• Ripple Effects Mapping
• Outcome Harvesting
• Most Significant Change
• Process Tracing
• Narrative Assessment
• Bellwether Method
• SenseMaker®
• Social Network Analysis
• General Elimination Methodology
• Evaluation Rubrics
• Qualitative Impact Protocol
• Labs

In a second article, we will focus on different tools and approaches and how to integrate them in learning-driven evaluation systems. Sign up to our newsletter to receive the link in your inbox when it’s out!

Here’s a couple more ways we can be of service to you: 

• Are you new to the field of systems change? We have a free Systems Change 101 workshop that provides a helpful starting point (possibly also useful for systems change leaders to help ground this conversation with your funders).
• Are you working in this field for a while? We welcome contributions to our blog! You can also join our #MyPieceForChange campaign for your work to be featured, and check out our support offerings, like the Fellowship.

About the author

Andrea is a senior program and impact strategy professional with deep experience supporting changeworkers to amplify their impact (foundations, impact investors, NGOs, purpose-driven companies and public administrations). With a background in International Economics and Design for Change, since 2017 she has supported 200+ organizations to define their impact strategy, lead impact-driven programs and social innovation labs, implement impact measurement and management (IMM) systems, and facilitate effective multistakeholder collaborations. Skilled at turning complexity into clear operational plans and measurable outcomes with a systems-informed approach, Andrea works towards just & regenerative economics, by redefining success and how we measure it, and putting communities at the center.