Brave Leaders Hold and Measure Clear Goals

Leaders need to commit to their goals and not be swayed by funders, according to New Philanthropy Capital’s Tris Lumley in an interview with Social Ventures Australia (SVA) Consulting’s Executive Director, Duncan Peppercorn.

This interview is published in SVA’s latest issue of the SVA Consulting Quarterly which provides social change and Not for Profit leaders, entrepreneurs, and investors with SVA Consulting’s insights, practices and new methodologies as well as opinion pieces from SVA colleagues in the sector.

Tris Lumley says he would die happy if every Not for Profit organisation set clear goals, measured them, held to them and stopped making excuses.

DP: When talking about measuring impact, what constitutes impact?

TL: I use a general definition: the difference you make through the work that you do. Impact is a broad church term to describe a range of things that others might define as output, outcomes and impact.

So for an organisation that’s trying to raise educational attainment in a particular area, the impact would be the eventual increase in educational outcome that you see. But it would also include the things that lead up to that: students more engaged in work, parents contributing more to school work, and better curricula.

DP: You used the word “eventual”. Many organisations struggle with measurement as many of the impacts happen so far down the track. What’s your advice to people, for example in the environmental space, where the impact may not happen for 20-30 years?

TL: This is something that everyone struggles with.

To look at the long term and intended impact that you are trying to achieve, you have to have the theoretical model that says how your piece fits into the long-term picture and what it makes sense for you to focus on now.

Theory of change or program logic provides the model that connects your activities to the ultimate aim, through the outcomes that you expect to see along the way.

You can’t wait for 25 years, and then find out that what you did didn’t work.

So you must have the [program logic] model and know how your work fits into the big picture.

David Pritchard, my colleague at New Philanthropy Capital (NPC) who heads up Measurement and Evaluation, tells the story of when he was working in Northern Ireland in peace and reconciliation work 20-30 years ago. He ran football projects with young people – inclusion work to bring them together from different sides of the divide. But the eventual aim was peace and reconciliation.

Back then, they didn’t have a theory of change.

Now he says, one of the outcomes that they could have measured was increased quality of relationships across the divide. The model would show that this would eventually lead (with other work, by other organisations) to peace and reconciliation.

So you must have the model and know how your work fits into the big picture.

If you know what you are trying to achieve, that will help you to get there. If you understand the route map that connects where you are now with where you want to get to, that will probably help you to get there. It’s pretty straightforward.

DP: Many of the changes that result from organisations’ work are soft outcomes such as stronger relationships which can be difficult to spot or to measure.

TL: I don’t think they are so difficult to measure. There are ways of finding these things out by asking the right questions.

Over the last six to seven years, NPC has developed a well being measure for 11-16-year-olds. It’s based on a straightforward questionnaire that assesses a number of different domains of their well-being including their sense of self, their self-confidence and self-esteem, and their relationships with family, friends and school. It’s rigorously tested, based on the best academic research about how to measure these well.

I’m not sure that it’s difficult to measure soft outcomes. I don’t think we’re trying hard enough.

The questions themselves are meaningful to those young people and it’s possible to find out, for example, if someone is happy.

To know that the results are comparable across individuals, you need to develop good questions and draw on the available academic research – something we’re not very good at in the social sector.

So I’m not sure that it’s difficult to measure soft outcomes. I don’t think we’re trying hard enough.

DP: One challenge is the time and commitment required to administer tools of this kind. How do you respond to someone who says, I’ve got a dysfunctional kid that I’m working with: getting them to focus for three minutes on the educational outcome is hard enough, let alone getting them to respond to a 15-minute survey?

TL: There are two aspects to this.

Firstly, the front line staff are the people providing the service and interacting with the beneficiaries. Their perspective is paramount and any approach to measuring impact won’t really stick, won’t get embedded in an organisation, unless it fits with practice.

One of the things that comes out of our research into impact measurement is the field workers’ concern about the burden of data collection and the interference in the relationships with clients and service users.

Data provides a voice to those people in vulnerable situations who are under represented…

Listening to frontline staff and working with them to develop approaches that do work should be a high priority.

Second, the power dynamic is important.

One of the main reasons that non-profits need to measure impact is not so that they can report how well they’re doing to others, but so that they can represent the service user or the beneficiary (on behalf of whom they are acting).

Data provides a voice to those people in vulnerable situations who are under-represented, who these resources are supposed to be deployed to help.

So you have to find ways of representing their experience and the change in their experience as accurately and robustly as possible. Yet, you still have to fit that in with practice, so you don’t terrify everyone who comes to get help from you with a half-hour questionnaire before you’ll work with them.

It reinforces the importance of getting this right and doing this properly.

This can make it more difficult to provide services but it’s important. We are not providing services to vulnerable people just in case it makes a bit of a difference. We have a greater responsibility than that.

DP: Do you think that there is a case for different tools for different ends? Should we use different measures and mechanisms depending on if you are telling the story and representing the beneficiary, or whether you are trying to prove that what you’ve done has had the impact that your funders want?

TL: Different techniques do lend themselves to different communication styles and to different audiences. The government may be more interested in statistics. Your fundraising communications may be better to tell the human stories about individuals.

Your theory of change or program logic model underpins all of your data collection and evaluation, so that you’ll have a golden thread of data that you draw on for different communications.

However, I don’t think you should be measuring completely different aspects of programs and playing them back to different audiences. It happens a lot but there are problems with it.

If you say to some audiences, ‘we provide the cheapest service you can possibly buy’, to another audience, ‘we take the most holistic, person-centered approach and that’s why our service is best for the beneficiary’, and to another, ‘for every 1 pound in, you get 27 pound out’, then there’s a disconnect and a lack of real meaning within it.

Hopefully, your theory of change or program logic model underpins all of your data collection and evaluation, so that you’ll have a golden thread of data that you draw on for different communications.

Co-ordinated Action Against Domestic Abuse (CAADA), for example, says that at the heart of all of its impact measurement is a single set of data that needs to be collected by every domestic violence advisor.[i] It includes certain disclosures, elements of their concepts, and key areas that they want to address. That data is then processed, analysed and presented in different ways, but there is a single data set that everyone works with.

DP: Many organisations recognize that the range of outcomes that result from their programs are truly immense and they want to monitor every single one of them. How do you advise them to get focused?

TL: Firstly, it comes back to how strong your theory of change or program logic model is in the first place. If you haven’t got that, you can’t relate data collection to outcomes nor know what is most important to measure. So you’ll just measure a bunch of stuff.

This is the hard challenge for leadership teams and boards to address.

This may not be straightforward. It’s chastening for us at NPC (itself a charity) to look at how long it took us to get our theory of change straight and work out the right data to capture.

Secondly how focused is your work? Often funding takes the work in a direction that can be funded, rather than you doing the work that is focused.

I was helping an NGO with its planning. The organisation works in scores of countries through dozens of programs and it wanted to come up with 10 key strategic indicators. However, it didn’t have 10 outcomes that it was trying to achieve – it had hundreds. So it wasn’t a surprise that it couldn’t focus its measurement; its programs weren’t coherent.

This is the hard challenge for leadership teams and boards to address.

DP: There is a critical linkage between measurement and evaluation and planning – making decisions about what you should do. There doesn’t seem to be enough of a focus on this. What are your thoughts?

TL: I’d die happy if the majority of organisations in the social sector had clear goals and the majority of funders supporting them had clear goals. Not enough of a spotlight is shone on planning as there should be.

NPC has worked hard to get impact measurement on the agenda, however in some ways we went wrong in focusing on the word ‘measurement’. Measurement is something that follows on from goals. Measurement is critically important but you need to have the goals in the first place before you know what to measure.

Our research last year into the UK charitable sector reported in Making an impact, demonstrated that measurement is driven by funders. Organisations are measuring so that they can meet funders’ requirements. While the results are beneficial and often result in learning and improving performance, that’s not the reason organisations are doing it.

It is up to the chief executive or the boards to say, ‘We need to do this stuff because it’s right, because we’re here to do the best job we possibly can for the people we are trying to help…’

Structurally, the sector is fundamentally broken; the funding market place has a transactional focus. In the UK, foundations are so protective of their ability to make independent decisions; that there seems to be next to no real responsibility for thinking about the strength of the organisations that they fund and whether or not they have clear goals and plans to achieve those goals.

Impact measurement is a game being played on a transactional basis to satisfy each funder, to get each chunk of money through the door. So it’s not a surprise that it’s not focused on strategic planning, learning and improvement.

This is where leadership is so important.

It is up to the chief executive or the boards to say, ‘We need to do this stuff because it’s right, because we’re here to do the best job we possibly can for the people we are trying to help. And we need to do better job next year than we’re doing this year. And we need to do this for ourselves. We’ll get to communicate this to others when we get it down for ourselves. But it starts here.’

There needs to be more focus on the brave leaders who are going to push this forward, who will communicate failures and what they’ve learnt from them, or turn down funding because it takes them away from their theory of change.

We can produce all the tools and approaches in the world, but it’s up to the leadership to use them.

DP: Would clearer goals help organisations to work more effectively together?

TL: It is important to develop shared theories of change within a field or sector. For example, around youth employability, we have developed the JET framework – a consensus from the field about the key outcome domains that young people need to address to get into and sustain constructive employment.

I’d die ecstatic if there were clear, shared goals within fields, and organisations were able to locate their work in the context of the work of others…

Shared frameworks are fundamentally important because they allow you to compare performance and the effectiveness of different interventions.

However, there’s something more fundamental which is understanding how different interventions and organisations fit together and work towards shared goals.

While I said I’d die happy if everyone had clear goals, I’d die ecstatic if there were clear, shared goals within fields, and organisations were able to locate their work in the context of the work of others and able to pitch shared plans.

Wouldn’t it be amazing to have organisations getting together developing a plan that consisted of their different interventions and working towards a single set of shared goals? And they pitched to funders saying 'You fund it or you don’t, but this is what we’re going to do'.

There’s too little of this, again because of the way that the funding eco-system works.

You don’t see organisations looking at how they could best work with each other. If I work in educational intervention, why aren’t I looking for the different organisations who can provide the outreach work, the job readiness support, and the employment support? There’s not enough of that, or of the funding that incentivises that.

DP: One of the challenges with this, is that many smaller organisations are driven by the determination and passion of a social entrepreneur; those people almost need to believe, that they are unique, and uniquely capable to give them the energy to keep going against all odds. Is there a risk that reducing this to shared goals, 65 consistent outcomes that everyone buys into, that the passion for innovation that social entrepreneurs bring to the sector might be lost?

TL: In the private sector, the innovators and entrepreneurs are often competing with the ways things are done; they’re up against well-known models. They’re out to change those models and potentially overturn practice in a whole field through their innovation.

Then an entrepreneur might be transforming a whole field, instead of just getting their organisation to grow.

In the social sector, the entrepreneur often has real vision and insight into the problems a group of people face and ways of addressing those problems. But even if they can do it for that group of people, how are they going to scale it to work across a nation, to reach 100 per cent of the people that they need to serve?

There aren’t many examples of successful scaling like that – completely overturning practice in the area.

We need a clearer understanding of what the important outcomes and goals are across the field, so that with more sharing of practice, it would be possible to scale effective interventions across the whole domain.

Then an entrepreneur might be transforming a whole field, instead of just getting their organisation to grow.

Endnotes

[i] Making an impact: Impact measurement among charities and social enterprises in the UK, pg 49, NPC, Oct 2012

About the author: Tris Lumley leads NPC’s Development Team, creating partnerships and strategic initiatives across the charity sector. Through relationships with charities, grant-makers, businesses and government, the Development Team explores and initiates projects to help the charity sector increase its impact. This includes the development of the Inspiring Impact program which aims to embed impact measurement across the charity sector by 2022.

Lumley will be a keynote speaker at the conference: Measuring Social Outcomes in Melbourne, October 15-16.