Measuring our success: Which voices count when the numbers are in? And why?

One Girl international programs director Erica Berthelsen considers radically participatory programming and how it’s challenging the status quo of how they work in-country and achieving better results/relationships.

How do we know we’re really making a difference? Whose impact are we measuring? To what end – and how? And are we measuring the right change? 

Limiting our “is-it-working” barometers to purely traditional quantitative and qualitative measurements can really reduce complex social and cultural change to mere numbers – and we are far too nuanced for this to be humane or realistic. 

For example, let’s say that this year, your New Year’s resolution is to get fit and have more meaningful relationships.

You decide how to measure your goals and make the following basic assumptions:

X number of gym visits = fit

X number of coffee dates = good relationships

Come July 2020, you’ve been going to the gym and have rekindled some relationships you’ve been neglecting lately by scheduling in one coffee date and two gym sessions a week. Hurrah! 

But in reality, how do we know you didn’t just drive to the gym, get on the treadmill for 10 minutes and give up? How do we know most of those coffee dates weren’t really painful tinder-style dates filled with awkward silences and repeated life stories? 

Yes, yes, because we also collect qualitative responses. But…does this really give us the full picture? It completely depends on whether you answered honestly. 

What about your relationships and the potential consequences of your responses? Or imagine the person asking you is your trainer and offers you a free pass to the gym for a year if you succeed? 

I think you might fudge them… just a tad.

In any case, it certainly casts a bit of doubt on the effectiveness of evaluating our success through set questions or simply counting participation numbers. Because it doesn’t reflect the  true story, it doesn’t see the full picture, the individual journeys, worlds and intentions. 

We must question our own intentions and ask: who owns this change?

Working with culturally and linguistically diverse groups, we have to respect distinct ways of being, knowing and believing – and be prepared to work outside our own traditional systems if we are to see the full picture. Because these intersect in all aspects of life, and mean different things to different groups of people. 

Typically, evaluation and research methods perpetuate the notion of them versus us. There is a clear power differential – one person asking, and the other responding. It doesn’t leave a lot of room for collective knowledge and belief systems. And by using this method, we’re potentially measuring change in a way that those involved aren’t able to gain or see the value in it. And isn’t that the whole point?

At One Girl, we challenge the status quo and use radically participatory methods – alongside the numbers of course. We take time and we listen actively. We speak to those who are participating in the programs to find out how they see change occurring. “What is the impact you want to see in your own community? How do you define and measure when it is achieved?”

For example, in many indigenous and collectivist cultures, messages are often conveyed through storytelling and art. So this is often how we design the “is-it-working-o-meter”. By using activities such as storytelling or group mural painting, power dynamics are dismantled and people are able to express themselves in a way they feel comfortable. 

It’s a pretty enlightening journey and allows for spontaneous and organic findings, that can be celebrated by all involved. We’ve also used a method called PhotoVoice, where participants take photos related to a central theme. In one evaluation, many girls’ responses to what their education means to them, linked this with nature and kinship. 

“I will grow [to be tall, strong, flourishing] just like the palm tree. I will have many branches, they represent my family, and I will help my family.”

For many people, the world is seen through a collectivist lens. The sometimes overused Zulu word Ubuntu, often translated as “I am because we are” encapsulates the essence of interpreting the world through others. It calls into question the colonial and individualistic lens we use to conduct research or evaluations. 

This is contrasted with individualistic methods (eg asker versus responder), which although useful in some ways, can fail to highlight the nuances of how education and notions of success are interrelated with the natural and social environment.

But perhaps the most significant impact of radically participatory methods is that individuals and communities no longer have to play the role of passive subjects – they are actively valued, respected and included. 

At One Girl, the girls we work with have to present in front of parents, elders and peers which contributes to their confidence, leadership and public-speaking skills. We find that using these culturally appropriate and participatory methods gives power and voice and the people themselves define the impact, which in turn, leads to stronger programs that are owned and celebrated by community members. 

And importantly, we still gain a deep understanding of what we’re doing, if it’s working and how we can do it better. In new year’s resolution terms, co-creating something that describes how it feels to be rekindling those relationships or getting to the crux about whether the gym is really the best environment for you to get fit in, might be a more effective way to measure your success in a way that you can really understand and celebrate it.

Using these kinds of uplifting and inclusive approaches is committing to the creation of a more conscious and connected world. 

A world where we don’t just aim for self-improvement but we recognise Ubuntu and commit to learning, listening, and acting together with first peoples and those close to the natural world. 

About the author: Erica Berthelsen is the international programs director of One Girl, an organisation that harnesses the power of education to drive change for girls and their communities. She specialises in gender-transformative programming, indigenous cultural revitalisation and asset-based community development.