Our commitment to using evidence to affect outcomes must be unflinching

Building capability to measure outcomes can only help in creating change, writes The Smith Family’s Doug Taylor, in his final piece in a series marking the organisation’s centenary

I have worked my whole professional life for not for profits (NFPs). What keeps me engaged in this work is the opportunity to create positive and lasting change in the lives of people who need it most. I must confess it’s sometimes hard to see progress, but in those moments I try to remember Dr Martin Luther King’s words “that the arc of history is towards justice”. 

Working with people and communities to create change is the last not-for-profit superpower I wanted to write about in this short series. I want to write about it because it’s core to who we are, but also enormously challenging to substantiate.

The key question is: How can NFPs know they are delivering on their purpose when it’s hard to quantify the change they are making in the communities they serve?

This was best espoused by the famous business writer Jim Collins (best known for Good to Great). What many people don’t know is that he translated this seminal book for the NFP sector, as Good to Great and the Social Sectors: Why Business Thinking Is Not the Answer (published in 2005). In this book, he grapples with the unique role and challenge for NFPs.  “There is no one underlying economic driver that applies across all social sector organisations,” he writes. “The whole purpose of the social sector is to meet social objectives, human needs and national priorities, that cannot be priced at a profit.”

When grappling with this challenge, many NFP boards and executive teams do one of two things. Firstly, ignore the problem altogether. But, as Collins writes, “to throw our hands up and say we cannot measure performance in the social sectors, the way you can in a business, is simply a lack of discipline… What matters is that you rigorously assemble evidence – quantitative or qualitative – to track your progress.”

Another common trap for NFPs is to rely on metrics that fail to reflect the true degree of progress in making change for people. Too often, we rely on citing outputs (e.g. numbers of people supported) or we focus disproportionately on financial performance. Important as these metrics are, they don’t really tell us much about our progress in improving people’s lives.

Our sector has a responsibility to demonstrate effectiveness – both to the individuals and communities we support and the wider community who enable us to do our work. We must be sure our work has a benefit, and we can’t know this unless we meaningfully track and measure the outcomes of our work.

Gathering data also helps us learn as we go, to inform our own work and that of others. It gives us the tools to understand what’s working, what’s not, and where new strategies are needed.

Delving into The Smith Family’s history, I’ve learned that our organisation has been intentionally “assembling evidence” (to use Collins’ words) since the 1970s, to try and understand the changing nature of people’s experience of poverty. For example, there were two landmark research investigations into poverty by Professor Ronald Henderson in 1970 (People in Poverty: A Melbourne Survey), who identified a threshold point for impoverishment, which he later developed in a Commission of Inquiry into Poverty (1972), which he chaired.

Professor Henderson charged The Smith Family with the responsibility of providing high-level insight, and the submission of a formal report on the dietary challenges faced by poor families. The findings of the Henderson inquiry changed the landscape of the poverty debate in Australia, because for the first time policy makers and practitioners could reflect on hard data that sought to offer a definitive measure of poverty in Australia.

This capability was further developed in the late 1990s when The Smith Family’s first research and evaluation function was created to further develop our understanding of people’s experience of poverty. It was also to review our practice and progress in supporting improved educational outcomes for children and young people.

And over the last decade, we have implemented our current methodology to track the outcomes of the students we support in terms of their school attendance, school completion, post-school engagement (in employment, education, and training) and tertiary completion. This approach draws on the widely used results-based-accountability methodology, and is an internationally accepted approach used by many NFPs and across all levels of government.

We report on these outcomes annually and transparently. We have chosen them because international research shows their relationship to educational and employment success, as well as a range of other life outcomes such as income, health and wellbeing. They were also chosen because they are policy relevant – the national/state/territory education agreements identify them as key student progress measures.

The scale and long-term nature of The Smith Family’s work with students and families creates a significant opportunity to measure program effectiveness, and in turn contribute to the Australian educational evidence base, with the longitudinal nature of the data being particularly important.

It’s great to see so many more organisations building their own capability in outcomes measurement. This is one of the sector’s developing superpowers: as the evidence base grows, we can be more courageous in focusing our efforts on, and diverting resources to, the work that makes the biggest difference. I’m also heartened to see a number of NFP organisations working together on data capability and outcomes measurement. This should be encouraged – ultimately, it will strengthen the sector, and make us all more effective at our core purpose. 

Read other articles in this series by The Smith Family’s Doug Taylor:

Rediscovering not for profit ‘superpowers’ for the challenges ahead

Is mobilising and listening to community a diminishing superpower?

NFPs must undergo their own change to meet society’s challenges