Measuring Impact in a Complex Organisation

Eleri Morgan-Thomas has been with Mission Australia for seven years and has held a number of positions in community services, housing, advocacy and public affairs. She recently took up the new position of Director Service Impact in the Office of the CEO. This article is from the CSI Blog.

OPINION: Imagine you are a new Director on the Board of a large Not for Profit and passionate about wanting to make a difference. You were attracted to this Not for profit in the first place because you believed it was doing good things.

But you might be sitting there thinking? Do I really know whether we are achieving the mission? Are we investing in the right programs? Are we really changing lives? What reports should I be requesting from management that can help me understand?

That’s the challenge confronting Directors of most Not for Profits and perhaps more so for large organisations providing multiple programs. There is often solid financial information but the indicators of service effectiveness just aren’t there.

I’m in the newly created role of Director of Service Impact at Mission Australia. We work towards creating a fairer Australia by advocating for people in need and helping them to get back on their feet. We strengthen families, empower youth, strive to solve homelessness and provide employment solutions. Recently we have fostered the growth of two new service arms in early learning /childcare and social housing.

Led by our Chairman and the CEO, we have just embarked on a new initiative to measure and report on impact across the whole organisation.

We have some of the building blocks already in place. For instance, we have mapped the outcomes and impacts we expect to achieve in four out of five ‘pathways’. And we have been investing in data collection systems.

And at an individual program level we have done some great evaluations. However we still have a long way to go before we can measure and report on impact. Not least because we currently collect data to meet our various internal and external reporting requirements and most of that is on inputs and outputs. For all the talk of outcome measurement in Australia, most funders, particularly government, can’t see beyond outputs and don’t want to fund outcome measurement.

It is important we are realistic about what we can collect and report on now and in the future and to be clear on why we are reporting and to whom. Unless we are crystal clear about the purpose of data and reporting, it will be a wasted exercise. The rigour of an evaluation exercise that will justify scaling up a program or getting government to invest is at a higher level to the performance information required to make adjustments to an existing program to increase the effectiveness.

The Michael Project is a good example of a rigorous evaluation. It was funded by a very generous philanthropist who wanted to test a new model of ending men’s homelessness. The resulting evaluation showed what worked – and what didn’t – and a standard cost benefit analysis resulted in demonstrable net savings of $3,600 per year from reduced hospital and justice costs.

While I was writing this, I was fortunate enough to come across an interview by Bridgespan’s Matthew Forti with Jodi Nelson Head of Measurement and Evaluation for the Bill and Melinda Gates Foundation. They face similar issues to Mission Australia of size, diversity, complexity and different uses.

Jodi Nelson says: “The biggest learning for me is that my job is more about organisational change than it is about being an evaluation expert.” This resonates with me a great deal.

To truly measure impact we need data and we need each of our service delivery divisions to collect that data. We want them to use it to improve their service delivery, to make decisions about what to scale up and what we need less. We also want them to report to the Board so that the Directors and senior management can understand better what we do and where resources should be directed.

But that won’t happen without quite significant change in the organisation. And I’m just recognising how big that challenge will be.

Fortunately, this is something Mission Australia’s Board of Directors, the CEO and I are all passionate about. From everything I read leadership from the very top of the organisation must drive that change.