Making Impact Measurement Meaningful (For Even The Smallest Organizations!)

Impact measurement (IM) is a contentious topic in the philanthropic sector.

To some, the growing interest in transparency, accountability and quantifiable impact is a leap towards scalable social change. To others, IM “represents all that is going wrong with philanthropy…the rise of the ivory-tower theorists and technocrats whose logic models and fixation with metrics blind them to real-world knowledge and common sense.”

What is certain is that small philanthropic organizations are facing pressures to produce meaningful measurements with limited resources. Is there a middle path to balance IM’s efforts and rewards?

Acknowledging the human, technical, and financial constraints of small charities, I propose a proportional approach to IM. A meaningful IM strategy is about finding the right fit for your organization; what Gugerty & Karlan call ‘The Goldilocks Challenge.’ By exploring free, lean, and ready-made resources, I will outline why investing in IM is worthwhile for small philanthropic organizations, and recommend various tactics to make IM strategic, inexpensive and effective. Even the smallest of organizations can enhance their capacity and impact through measurement.

Measuring Matters

Progress lies not in enhancing what is, but in advancing toward what will be. —Khalil Gibran

Why we measure speaks to the heart of philanthropy: We want to make a difference. Research shows that quality IM improves a charity’s overall effectiveness and impact over the long term. Meaningful measurement “can help small charities focus their limited resources on activities they know work, can boost morale where work is shown to make a difference, and help attract funding.” Evaluation explores foundational questions like “Are we making a difference? How can we improve? Are we reaching the right people? How does our programme lead to change?”

Traditionally, grant-making organizations have found these questions particularly challenging to quantify across a portfolio. It is one reason why charity reports typically focus on counting outputs rather than understanding the outcomes their activities generate. “Many foundations believe that they cannot assess the impact of their responsive grants in any meaningful way and therefore do not attempt to do so” report Ranghelli, Mott & Banwell. Fortunately, the rise of impact investing and cross-sector capacity-building efforts have generated a wealth of resources applicable to even the smallest foundations.

Start Small and BUILD Up

Authors Gugerty and Karlan suggest starting small. Only collect data critical to helping the organization learn and iterate. “When data is collected and then not used internally” observe Gugerty and Karlan, “monitoring is wasted overhead that doesn’t contribute to organizational goals” (p. 12). Instead, Acumen’s Lean Data methodology provides us with the acronym BUILD to match our IM to our capacity and needs.

Proportional IM is:

• Bottom-up. It nurtures the habit of listening [to stakeholders] in order to provide actionable insight on their needs and interests.

• Useful. It yields data of sufficient quality to support decision-making.

• Iterative. It allows for learning, adaptation, and replication.

• Light-touch. It uses low-cost tools and technologies that require a minimal investment of time and money.

• Dynamic. It enables rapid data collection within a fast-changing environment.

BUILD remedies three flaws in the traditional approach to IM. (1) BUILD realigns the focus from upwards reporting for funding and investment towards multilateral accountability. This shift puts your mission, not compliance, at the center of IM. (2) It suggests that IM strategies need to be alive in organizational culture and embedded in management practices. (3) It promotes flexible IM that can evolve with organizational needs.

Do Not Reinvent the Wheel

The labyrinth is thoroughly known, we have only to follow the thread... —Joseph Campbell

Despite the name, using a BUILD framework does not require us to build from scratch. It is quite the opposite. While quality IM should reflect your unique contexts, BUILD's lean methodology recommends relying on existing resources for design and implementation. What is more, using recognizable standards helps you more clearly communicate your impact to stakeholders, allows you to engage in peer learning communities easily, and contributes to a data-rich social sector.

I find the most useful IM guides come from the United Kingdom. A cross-sector initiative of making “high-quality impact measurement the norm for charities and social enterprises by 2022” (Inspiring Impact, n.d., p. 1) has produced a treasury of navigable resources. New Philanthropy Capital (NPC) (thinknpc.org) and Nesta (nesta.org.uk) both produce exceptional guidance documents with practical tool portals called Inspiring Impact and Development Impact & You (DIY) that provide resources for philanthropy and social innovation respectively (inspiringimpact.org, diytoolkit.org).

The amount of IM resources online can be overwhelming, which is why I recommend efficiency-seeking foundations look exclusively at NPC and Inspiring Impact. Their resources are free, peer-reviewed, with comprehensive support like self-assessment tools, case studies and practitioner forums. Inspiring Impact’s easy-to-use Resource Library also curates external resources by sector, type, ease of use, and cost.

Theory of Change

Remember, always, that everything you know, and everything everyone knows, is only a model. Get your model out there where it can be viewed. Invite others to challenge your assumptions and add their own. ―Donella H. Meadows

All aforementioned resources recommend starting with a theory of change (e.g. NPA's four-pillar evaluative framework in Figure 1). A theory of change is a clear, often visual, demonstration of "what you want to achieve and how you plan to achieve it, setting out the causal links between your activities and your end goal."

Figure 1. NPC’s four pillar approach to building a measurement framework (Kazimirski & Pritchard, 2014).

Theories of change have multiple levels of utility. In addition to mapping your IM, they can help investigate partner charities and choose grantees. The Oxford Impact Measurement Programme, NPC and other leading evaluators all score theory of change when analysing social outcomes across a portfolio. “While there are no ‘silver bullets’ or panaceas in the practice of evaluation,” explains Jackson, “the concept and tool of theory of change can and should be an integral and explicit element in the evaluation enterprise in the impact investing industry." Moreover, having and refining a theory of change is becoming a prerequisite for more and more funding, investment and grant opportunities.

Embedding Evaluations

At the heart of IM is the principle of learning. A learning organization, according to systems theorist Peter Senge, “is an organization that is continually expanding its capacity to create its future." Embedding evaluation is a whole-organizational commitment to listening to your stakeholders’ needs through routinely collecting data, assessing, learning from and improving your work.

Embedding evaluation takes engagement, enthusiasm and effort, but it is essential to reap the benefits of meaningful measurement. Fortunately, Inspiring Impact recommends a practical guide for making IM stick. It includes workshop tools, self-assessments, a diagnostic wheel and advice on linking measurement to strategy (see Figure 2). Additionally, when evaluative practice is embedded, IM is routine. So there’s no need to scramble for evidence come reporting time.

Figure 2. Evaluation Support Scotland’s embedded evaluation diagnostic wheel, where 1 = poor and 5 = excellent.

A whole-organization approach can feel daunting, but the agility of small organizations provides an advantage. Besides, the clarity and quality of available resources through NPC and Inspiring Impact mitigates the effort. Thinking back to the BUILD framework, it is vital that we apply the same lean principles to the whole life-cycle of IM. Embedding efforts still need to be flexible and proportional to your needs.

Paddling Together

We won’t change systems if we paddle our own canoe.—Al Etmanski

It is worth noting that none of my recommended resources are Canadian. While Canadian content exists (e.g. Tamarack Institute, CanadaHelps), our resources centers are not as developed as those across the pond. That said, even in the absence of a national peer-network, it is worth seeking out partnerships for sharing IM findings, failures, and learnings within your environment. Tapping into the growing body of collective knowledge is not only efficient and cost-effective but a high-leveraging tactic that produces tangential positive results. As Kramer observes, “if you bring [change-makers] together, they start stealing from each others’ work in really productive ways.”

To conclude, building an organization that understands and intrinsically values impact takes time. However, the outcomes for your charity’s operations and aspirations make it well worth the effort. IM is proven to further your organization’s focus and productivity towards achieving your goals. With BUILD principals based on lean methodology and free tools from NPA and Inspiring Impact to guide you through each step of the process, even the most resource-constrained organizations can model meaningful, proportional IM.