Case Study: Randomised control trial of SME business support vouchers
6 June 2019 at 8:14 am
Are vouchers that give SMEs access to creative sector support effective? The Creative Credits programme used a randomised control group method to evaluate a voucher scheme, writes Geoff Mulgan, as part of a series of case studies highlighting different approaches to capturing impact.
Previous research had shown that businesses that transacted with suppliers in creative sectors like advertising, software and design were more likely to introduce product innovations.
Our programme, therefore, intended to boost innovation by connecting small and medium sized enterprises (SMEs) in other sectors with creative suppliers. Specifically, 150 Creative Credits vouchers with a value of £4,000 (A$7,260) were awarded to eligible SMEs in the Manchester City Region to develop collaborative innovation projects with creative businesses. The SMEs were required to contribute a further £1,000 (A$1,810) of their own funds towards their project.
Creative Credits created genuinely new relationships between SMEs and creative businesses. The award of a Creative Credit increased the likelihood by at least 84 per cent that firms would undertake their innovation project with a creative business they had not previously worked with.
Creative Credits created genuinely new relationships between SMEs and creative businesses.
In the six months following completion of their creative projects, SMEs were significantly more likely than those that were not assigned Creative Credits to have introduced product and process innovations. The use of creative services also had a statistically significant positive effect on the sales growth of SMEs over the same period.
However, these benefits proved temporary: 12 months after the completion of the project there was no longer a significant difference between firms participating in the scheme and the control groups in the proportion of firms innovating – nor in sales growth.
The method
We evaluated the scheme using a novel three-stage methodology centred around a randomised controlled trial (RCT). It included:
- a randomised allocation of Creative Credits to firms in the “treatment” group, comparing innovation and business performance with a “control” group of firms made up of non–recipients;
- data collection over time, allowing longer-term as well as short–term impacts to be assessed; and
- a “mixed–methods” approach, combining qualitative in-depth interviews with quantitative surveys.
The six-month additional benefits and 12-month decrease in impacts would have remained hidden with normal evaluation methods used by government, and indicated that RCTs should be used more widely in the evaluation of business support policies.
By comparing the experience of SMEs where benefits were sustained with those that were not, the qualitative element of the evaluation revealed that in the latter the creative projects had been too “transactional” in nature, and for others there had been communication difficulties with their creative partners. This suggests that future versions of the scheme, which seek to sustain longer-term benefits, should explore opportunities for targeted brokerage to support the relationship between SMEs and their creative suppliers.
Innovation support schemes are notoriously difficult to evaluate because of the possibility of selection biases, namely that SMEs that opt to participate in such schemes may in any case be more likely than the average firm to innovate.
The RCT methodology was possible because voucher schemes lend themselves to random assignment. Longitudinal data collection enabled longer-term impacts to be assessed as well as the more usual short-term impacts. And the programme’s logic model pointed to a number of confounding factors shaping the relationship between creative engagement and firm innovation that only a qualitative evaluation could satisfactorily unpick.
What we learnt
The three-part evaluation approach in the Creative Credits project – randomised allocation of vouchers, longitudinal data collection, and the use of mixed methods – proved to be effective, and should be used much more widely by the government and other agencies in developing new innovation support policies.
We recommended that innovation vouchers and other innovation support schemes should, wherever possible, be subjected to an RCT evaluation before they are rolled out – i.e. businesses are randomly allocated to the participating group or to a non-participating control group. This would enable a sound assessment of the programme’s effectiveness before wider roll-out, and is a cheaper and simpler allocation method than subjective assessments.
Creative Credits stimulated several creative innovation voucher initiatives across Europe.
Although the benefits for SMEs in the case of Creative Credits were temporary, we found they were large enough to offset the costs of running the programme. As a result of the programme, Creative Credits stimulated several creative innovation voucher initiatives across Europe, such as the European Commission-funded VINCI pilot in Salzburg, Austria, which was then scaled up to a national programme called Kreativwirtschaftssheck.
In a report presented at the European Creative Industries Alliance conference in 2014 the Creative Credit scheme was referenced as one of the first “to address the innovation needs of traditional SMEs through the provision of services by creative supplier firms”.
The impact
In the UK, Creative Credits inspired a number of local innovation voucher schemes funded by the European Regional Development Fund, such as the Innovation Vouchers scheme at Aston University, as well as the Technology Strategy Board (later Innovate UK)’s national innovation voucher scheme. It was widely credited as having given the UK’s business department the confidence to use RCTs in designing and evaluating innovation schemes, including the Growth Vouchers.
Creative Credits’ international influence was cemented by it being profiled in the OECD’s 2015 review of best practice in industrial policy evaluation.
About the author: Geoff Mulgan has been chief executive of Nesta since 2011. Nesta is the UK’s innovation foundation and runs a wide range of activities in investment, practical innovation and research.
This article was first published by Nesta.
It is one of a series of eight examples provided by Nesta to highlight different approaches to capturing impact they have used in recent years.
Read Geoff Mulgan’s article here about why it’s essential to try and track what is being achieved.
See also:
Case study: Capturing the impact of a challenge prize
Case study: Using matched-control groups to evaluate a volunteering scheme
