Case Study: Assessing the impact of an impact investment
10 July 2019 at 5:24 pm
Can tutors help children improve their maths? The Nesta Impact Investment team supported Third Space Learning to undertake a randomised controlled trial and process evaluation to assess whether supporting children’s learning with a tutor made a difference, writes Nesta CEO Geoff Mulgan as part of a series of case studies highlighting different approaches to capturing impact.
In 2015, Nesta Impact Investment (NII) invested in Third Space Learning (TSL), a company that helps children struggling with maths by connecting them via an online platform to tutors based in India and Sri Lanka.
Over half of pupils eligible for free school meals, a key measure of poverty in the UK, did not achieve expected levels in maths in 2017. That’s one in two young people from low-income households who were denied access to secure and well-paid careers, and to further study. Over the last five years, a core part of NII’s education strategy has been to invest in companies that provide high quality products and services that reduce the gap in attainment between high and low-income households.
The method
Wherever possible, NII is committed to investing where there is strong evidence of impact. We know from work by the Education Endowment Foundation (EEF) that one-to-one tutoring is one of the most effective ways of improving learning, however, it is also very expensive and often not affordable for parents or schools. By using its innovative model, TSL is able to reduce the cost of tutoring and make it accessible to the most disadvantaged students. For us, TSL was a clear fit.
When making the decision to invest, NII first looks at three things on the impact side:
- Commitment to impact. Is the company focused on positive social impact for those who need it most? We liked TSL because the founder specifically set out to address this problem of low attainment for poorer students.
- Alignment between the social impact and the commercial case. Creating impact must be core to the commercial success of the business. For Third Space, this is very clear – they sell into schools and if they can’t demonstrate the change they make for students then schools simply won’t buy the product.
- Inclusion. We need to be sure that businesses are reaching the people who need their products and services the most. Fifty-five per cent of Third Space’s students receive free school meals compared to the national average of 14 per cent.
We then go on to make a deeper analysis of the amount of impact we expect an investment to achieve (so-called “impact returns”): How many people will it reach? What is the level of need in those people? How much of a difference will this product or service make to their lives?
One big lesson we’ve learned over the years is the need to look at a host of risks to impact as well as impact “returns”. These risks take into account the evidence base behind a particular approach, the willingness of a company to measure impact and learn from the results and the dependence of the model on other people, such as teachers, changing their behaviour.
It is these impact risk and return metrics that we track throughout the life of an investment. In addition, we look for quarterly indicators that a product or service is of sufficient quality to achieve impact. For example, for TSL, in line with the research from the EEF, we track the number of students receiving at least half an hour of tuition per week for eight weeks.
To understand more about our approach, take a look at the Impact Strategy NII published last year.
The impact
We work with all of our investees to increase the rigour of their evaluation and therefore reduce their impact risk. Over the course of the 2014/2015 academic year, TSL undertook a randomised controlled trial, funded by the EEF and Nesta, and undertaken by the York Trials Unit. A process evaluation was also undertaken to understand the perceptions of teachers and pupils, assess whether the intervention was delivered as intended, and inform any future development of the intervention.
The trial found no significant effect of TSL on maths test scores. The process evaluation suggested several potential explanations for the lack of impact observed, including technical problems and communication difficulties between students and tutors. In contrast, teachers were largely positive about the online tuition, and reported that it appeared to improve pupils’ comprehension, verbal fluency, and confidence in maths.
What we learnt
Despite the fact that the evaluation found no significant impact, neither NII nor TSL lost faith in the potential of the model. Improving maths scores is extremely difficult and it is unsurprising that this wasn’t immediately successful. Indeed, fewer than one quarter of EEF trials demonstrate a positive effect. TSL was at an early stage of its development during the trial and both during and after the trial made significant changes to the stability of its technology platform and its training of tutors.
Over the last two years, TSL has strengthened the elements of its model identified as weak in the evaluation. It has implemented a state of the art approach to tracking the progress of children through the curriculum, giving before and after data on each pupil, as well as more rigorously managing tutor performance.
We supported TSL on this journey, as we do with all of our companies, by helping to track data, interpret it and take action. Much as a traditional venture capital investor does on the commercial side, we also take an active approach to managing for impact, meeting with portfolio companies quarterly and providing technical assistance wherever necessary.
This slow reduction in risk since the evaluation result has also been matched by a growth in returns as we have seen their scale grow five-fold, from less than 1,500 students three years ago to about 7,000 today.
Much as a traditional venture capital investor we take an active approach to managing for impact.
What’s next?
TSL has assessed the future of the market for tutoring and taken the decision to develop a new product, Matr, which is used directly by students in their homes. On this model, parents pay for the tutoring rather than it being arranged through the school. TSL provides this tutoring at a lower cost than its competitors, making it more accessible to students from disadvantaged backgrounds. NII is working with TSL to adjust the theory of change and quarterly metrics to reflect the new delivery model. Once this model has stabilised, NII will look to work with TSL to undertake an evaluation that would meet a Level 3 on our standards of evidence by establishing a causal link between TSL and improvements in maths scores. This may be another RCT or it might use administrative data from schools to carry out a matched analysis. We look forward to continuing to support TSL in its journey, we have learned much from its unrelenting commitment to impact for those children that need it the most.
About the author: Geoff Mulgan has been chief executive of Nesta since 2011. Nesta is the UK’s innovation foundation and runs a wide range of activities in investment, practical innovation and research.
This article was first published by Nesta.
It is one of a series of eight examples provided by Nesta to highlight different approaches to capturing impact they have used in recent years.
Read Geoff Mulgan’s article here about why it’s essential to try and track what is being achieved.
See also:
Case study: Capturing the impact of a challenge prize
Case study: Using matched-control groups to evaluate a volunteering scheme
Case Study: Randomised control trial of SME business support vouchers
Case Study: Embedding standards of evidence into grant-making
