Media Briefings

Debating The Impact Of Public Policy: Top Us Economist Calls For Us All To Recognise

  • Published Date: August 2011

Both politicians and journalists should be deeply sceptical of analysts in academia, thinktanks and government agencies who seem unwavering in their policy predictions. That is the central message of Professor Charles Manski of Northwestern University in a detailed critique of what he calls ‘incredible certitude’ in the analysis of public policy and the likely consequences of alternative policy choices.

Writing in the August 2011 issue of the Economic Journal, Professor Manski urges policy analysts to improve the credibility of their studies and inform policy formation better by being upfront about the uncertainty of their predictions.

For example, predictions about the consequences of alternative policies are regularly expressed as point estimates – how much money a programme will cost or how many people will participate. These numbers are often extremely unreliable, resting on unsupported assumptions or leaps of logic, yet policy-makers, the media and the public tend to take them at face value.

Professor Manski exposes various practices of incredible certitude in the hope that their identification will lead to more useful policy analysis. He starts with the problem of ‘conventional certitude’ – the prediction equivalent of conventional wisdom or a conclusion that is generally accepted as true though it might not be.

An example is the ‘scores’ or point predictions produced by the US Congressional Budget Office (CBO) to assess the budgetary impact of legislation 10 years out. Eagerly awaited by both Democratic and Republican members of Congress, CBO scores are often used to stake out legislative positions even though they are based on strong and unsubstantiated assumptions.

Scoring the recent US healthcare reform legislation, CBO director Douglas Elmendorf reported to Congress that its enactment ‘would produce a net reduction of changes in federal deficits of $138 billion over the 2010-19 period as a result of changes in direct spending and revenue’.

Professor Manski points out that anyone seriously contemplating the many changes to federal law embodied in this legislation should recognise that the prediction of $138 billion deficit reduction is nothing more than a very rough estimate. Yet Elmendorf’s 25-page letter expressed no uncertainty about the figure, subsequently reported by the media without questioning the reliability of the number.

In an instance of ‘duelling certitudes’, Douglas Holtz-Eakin, a former CBO director, made a very different prediction. In an op-ed in the New York Times, he predicted that the legislation would increase deficits by $562 billion rather than decrease them by $138 billion. Thus, the former and present directors of the CBO made predictions that differed by $700 billion, yet each presented his prediction as certain.

While policy analysts in the United States seem to abhor uncertainty, the UK government has started to recognise the sensitivity of forecasts to varying assumptions and adjust its practices accordingly.

UK agencies are now required to state upper and lower bounds on their estimates of the economic impact of proposed legislation when submitting an impact assessment to Parliament. In addition, agencies are instructed to highlight key assumptions and assess how changes in assumptions might affect policy delivery. This transparency opens the door to more credible policy analysis, Professor Manski concludes.

The economist also takes the media to task for stretching the conclusions of scientific studies – ‘overreaching’. In other cases, journalists seeking the latest scoop sometimes report research prematurely – especially before it has been peer-reviewed and published.

He suggests that since journalists cannot be experts in every field that they cover, they should do more to seek out reputable researchers who are not closely associated with the authors of studies.

In the end, Professor Manski suggests that policy-makers, the media and the public should be deeply sceptical of those who appear unwavering in their policy predictions. He recommends recognising ambiguity rather than ignoring it.

He also proposes a possible solution for policy analysis: using a ‘layered’ analysis, researchers could move from weak, highly credible assumptions to stronger, less credible ones, determining the conclusions that follow in each case. This would help resolve the tension between the credibility and power of assumptions and also improve the transparency of policy discussions.

ENDS

Notes for editors: ‘Policy Analysis with Incredible Certitude’ by Charles Manski is published in the August 2011 issue of the Economic Journal.

Charles Manski is at the Department of Economics and Institute for Policy Research, Northwestern University.

For further information: contact Charles Manski on +1-847-491-8223 (email: cfmanski@northwestern.edu); or Romesh Vaitilingam on +44-7768-661095 (email: romesh@vaitilingam.com).