Pro Bono Economics

An update by Sue Holloway

For the RES Newsletter, January 2012, Online Issue 1

In the 146th issue of the Newsletter, July 2009, we highlighted an initiative by Martin Brookes and Andrew Haldane to launch this service whereby professional economists could make their skills available to charities on a ‘pro bono’ basis. Sue Holloway, Managing Director of Pro Bono Economics has sent us this update.

Pro Bono Economics was launched in September 2009. It is a charity whose aim is to broker volunteer economists to work with other charities, mainly helping them to measure their impact and value for money.

The charities

Since then we have received proposals from over 70 charities, mostly asking us to look at their impact, but in some cases to produce a literature review or piece of analysis to support their advocacy efforts. Some are already well along the road of evaluating their interventions and collecting data on outcomes, while others are at the start of that journey. A number have not been able to continue the conversation with us because of the pressures of the current funding environment, which takes up all the time and energy of a few dedicated members of staff. Others have not had sufficient data to allow a piece of economic analysis, but our volunteers have advised them on what they should be collecting in order to be able to do this in the future. And some have been able to share rich datasets which have allowed the volunteers to look at the value they are creating across a number of dimensions. To date we have completed 10 projects, details of which can be found on our website: www.probonoeconomics.com/projects, with another 21 in various stages from scoping to near completion, and 13 at the early stage of discussing feasibility.

The charities we have worked with are immensely grateful for the help they have received ...

‘We are proud to have had our work evaluated by economists from a leading consultancy. We hope this is a first step in defining terms and creating measures that others can adopt, ultimately producing transparent reporting across all organisations in our sector.’ — Baroness Stedman-Scott, Chief Executive, Tomorrow's People

... but there are still many charities grappling with the need to demonstrate their impact, who have yet to understand or appreciate the value of economic analysis to their business and so do not seek it.

Bank of England economists looking at Barnardo’s work with young people at risk of sexual exploitation were able to use a difference in treatment age between two local authority areas and econometric modelling to create a synthetic control group (full details of the methodology can be found here: http://www.probonoeconomics.com/projects). Their innovative approach showed that outcomes were likely to deteriorate without Barnardo’s intervention, and that the costs avoided were greater than a simple before and after analysis suggested.

‘The moral argument for helping girls and boys who are exploited for sex is plain — now we have tangible, economic evidence of the necessity for specialist help too.’ Anne Marie Carrie, Chief Executive, Barnardo’s

The volunteers

To date over 160 economists have registered as volunteers. The majority come from the private sector — either freelance, retired or working with economics consultancy firms such as Charles Rivers Associates, Europe Economics, Frontier Economics, FTI Consulting, Oxera and NERA. We have volunteers from the Bank of England and Standard Chartered Bank, as well as fifty from the Government Economic Service and twelve from academia. The benefits of getting involved with PBE do not accrue only to the charities. PBE contributes to professional development as economists get to apply their toolkit to areas they previously may not have worked on, and to be at the forefront of developing methodologies to measure social impact in a more robust and rigorous way.

The challenges facing PBE volunteers

Data is often a challenge — the charity may not be collecting the information needed to inform an economic analysis, or may not be collecting it on a consistent basis. Many charities are working with very small groups, and volunteers may need to look at interactions between sample size and the magnitude of the change effected, to understand both significance and power, and whether it’s possible to draw any statistically significant conclusions from the data available. While official administrative data may be necessary to track longer term outcomes for the client group, the charity may not be able to get hold of it easily or at all.

Some areas of intervention are well researched and there is a wealth of academic literature and evaluation material to back up assumptions about the longer term impacts of interventions. In other areas, there is very little to go on and heroic assumptions have to be made. In all cases, transparency about the assumptions made and the strength of the evidence-base is key, to allow discussion and challenge, and continuous improvement.

Most charities do not have the resources or, in some cases, the mandate to conduct a randomised control trial. Often, identifying an appropriate counterfactual can be problematic. Potential ways to overcome this include comparisons with an appropriate national average — assuming the charity collects sufficient data to be confident that their client group is representative of the national profile — self selection bias is a big issue in some types of intervention. Where the charity or the volunteer can access a large administrative dataset, it may be possible to identify a control group through propensity score matching and compare outcomes.

There is still considerable debate about what should and should not be included in a wider social cost benefit analysis. If we want to focus narrowly on savings to the Exchequer, we need information on the marginal rather than average cost of savings in public expenditure, unless we are dealing with an intervention that is operating at such a scale that success would lead to the closure of whole institutions or substantial parts of them. Unfortunately most of the published cost information is based on average costs, which leads to inflated estimates of savings. The important thing here is to be careful about how the benefit from reduced public expenditure is described and to make a distinction between what is cashable (reduced benefit payments to someone who is now in work) and what is not (the average cost of preventing re-offending).

Ideally we want to include improvements in individual well-being brought about by the charity’s intervention. These are particularly difficult to measure and value, but with the collection of national well-being data by the ONS, and developments in valuation methodologies using the life satisfaction approach (see a discussion paper by HM Treasury and DWP here http://www.hm-treasury.gov.uk/data_greenbook_news.htm), it may not be long before we can begin to include these important benefits in our economic analysis.

‘Working with St Basils through PBE has been a great experience. We have had the opportunity to broaden our own professional skills across a complex and fascinating set of issues. We would encourage other economists to get involved with PBE..’ — Patrick Curry and Ben Gill, HM Treasury

Lessons learnt

We are grateful to the 80+ economist volunteers who have contributed or are contributing to a project in some way. Many are doing so entirely in their spare time, and we have learnt the importance of using teams wherever possible, so that the inevitable peaks and troughs of working life can be smoothed across a number of team members.

The small central team (currently two full-time staff) has a key role in managing the charity/volunteer relationship and ensuring a high standard of economic analysis. We do the latter through sharing resources and best practice with the volunteers, commenting on work at appropriate stages and arranging for an independent economist (i.e. who has not been involved in the project) to peer review the final analysis. We also work with the charity on how the results are publicised, helping them to communicate clearly what the analysis is and isn't saying — avoiding the simplistic interpretation and taking the caveats seriously.

What next

There is still more to do to spread the word that economists are willing and able to help charities demonstrate the value of the outcomes they produce. There are many charities who would like this kind of support, but cannot afford to pay for in-house or outsourced economists, and would, in any event, be unlikely to pay for the amount of analysis that its public value would suggest is optimal. PBE can continue to fill this missing market for many years to come.

The economics profession, unlike lawyer and accountancy colleagues, does not have a history of volunteering using their professional skills. We hope to continue to provide these opportunities to make a valuable contribution to the third sector while at the same time enhancing the professional development of the economists who volunteer with us. We want to develop guidance on methods and share information on the latest research and published costs, in order to support volunteers and ensure they can hit the ground running when they embark on a project.

To do all this we need three things:

  • more discussions with charities to build demand and ensure a pipeline of feasible projects - if you are involved with a charity who might benefit from PBE’s services, start the conversation and put them in touch with us here http://www.probonoeconomics.com/charities-register;
  • continuing support from economist volunteers - get in touch here http://www.probonoeconomics.com/economists - register if you are willing and able to spend a couple of days scoping a project, or half a day peer reviewing a piece of analysis, or if you would like to be part of a team who can work on producing a piece of analysis over the course of many months - the input will vary according to the complexity of the analysis and the size of the team;
  • continued funding for the central team, including a modest expansion - if you would like to donate, see our website http://www.probonoeconomics.com/donate-now or contact us at info@probonoeconomics.com
From issue no. 156, January 2012, pp.18-19

Page Options