UK Report Highlights Limitations of SROI
13 January 2011 at 3:51 pm
A report from the UK’s Third Sector Research Centre has raised concerns about the danger of funders using SROI (social return on investment) as a way of choosing between Not for Profit organisations.
The ambitions and challenges of SROI working paper says while comparisons of SROI ratios within the same organisation, over time, may be useful for tracking progress, it warns against using SROI ratios to compare organisations.
Social Return on Investment is defined in the report as a form of adjusted cost-benefit analysis that takes into account, in a more holistic way, the various types of impact that programs have.
The SROI framework creates a ratio of how effective a NFP or program is by putting a financial figure on work performed and results achieved.
The report says the SROI ratio, and the way by which it has been produced, is specific for each organisation and does not lend itself to cross-organisational comparison. The methods used and judgements made in formulating a SROI ratio are rarely identical between organisations, therefore use of the ratio as an indicator of comparative program effectiveness is flawed.
The report expresses the concern that funders, whether they be philanthropists, grant-making organisations or governments, will use SROI in decision making when deciding between organisations.
The report acknowledges there is growing interest in measuring the impact of Not for Profit sector activities, saying funders and philanthropists are becoming increasingly focused on outcome, impact and ‘value for money’.
The Social Return on Investment framework is seen as a way of providing effective measurement in these areas.
The report notes that, while aiming for rigour, the SROI method leaves a great deal of space for personal judgement. This makes it possible to inflate the value created and may also lead to misunderstandings about how to interpret the SROI ratio. Although auditing tools and procedures to help standardise the way SROI is calculated have been developed, attention needs to be paid to how the results of SROI are used.
Malin Arvidson from TSRC says that SROI is likely to become an increasingly dominant approach to measuring the impact of voluntary organisations due to support from various parts of the public sector. But analysis of SROI raises important questions about why and how we measure impact.
He says SROI arguably provides a powerful tool to help organisations illustrate the value they create in a language that those outside the sector understand. But attention needs to be payed to how results are used, especially as there is a tendency to adopt it as a comparative tool.
The report’s authors, Dr Malvin Arvidson, Professor Fergus Lyon, Professor Stephen McKay and Dr Domenico Moro, note that SROI is still at an evolutionary stage, and identify emerging challenges in a bid to push for further research and to enable readers to properly interpret SROI reports.
The emerging challenges for SROI framework identified in the report include:
The need for evidence and monitoring systems
The report says SROI will not reach its full potential until there is more investment in improving the evidence base of the sector. Not for Profit organisations need effective monitoring systems for an SROI approach to work.
Focus on impact at the expense of understanding process
SROI claims to use a mix of narrative, qualitative and financial measures to tell a story of change, however in the reporting of SROI this is often a secondary element behind the most cost-benefit focused overall SROI ratio.
Competing goals and principles
The focus on collecting data on quantitative measure presents a risk of affecting what actual activities are carried out. The report says there is a need for research which examines the extent to which the maxim ‘if it cannot be measured it cannot be managed’ is permeating the sector. There may exist less tangible benefits in an NFPs actions that cannot be so easily recorded. The report also raises concerns that the mission of an organisation may change to cover the goals that are more easily quantified.
Quantifying the value of benefits
The report says there are obvious challenges in attributing monetary values on outcomes and impacts. The focus of SROI is in expressing social benefits in monetary terms.
Deadweight, displacement and attribution
Where change is caused by a range of factors including the activities of the organisations being evaluated, it can be very difficult to know the amount of benefit that can be attributed to the activity being measured. The report uses the example of programs aiming to support unemployed people, saying it is very difficult to distinguish the impact of a specific program from that of other Not for Profit organisations, the public sector, or changes in the wider economy resulting in increased or reduced employment.
High Cost
The report also raises concerns about the high cost of carrying out SROI analysis.
