Media Briefings

DEBATING THE IMPACT OF PUBLIC POLICY: Top US economist calls for us all to recognise the extent of our uncertainty

  • Published Date: August 2011

Both politicians and journalists should be deeply sceptical of analysts in academia,
thinktanks and government agencies who seem unwavering in their policy predictions. That
is the central message of Professor Charles Manski of Northwestern University in a
detailed critique of what he calls ‘incredible certitude’ in the analysis of public policy and the
likely consequences of alternative policy choices.
Writing in the August 2011 issue of the Economic Journal, Professor Manski urges policy
analysts to improve the credibility of their studies and inform policy formation better by
being upfront about the uncertainty of their predictions.
For example, predictions about the consequences of alternative policies are regularly
expressed as point estimates – how much money a programme will cost or how many
people will participate. These numbers are often extremely unreliable, resting on
unsupported assumptions or leaps of logic, yet policy-makers, the media and the public
tend to take them at face value.
Professor Manski exposes various practices of incredible certitude in the hope that their
identification will lead to more useful policy analysis. He starts with the problem of
‘conventional certitude’ – the prediction equivalent of conventional wisdom or a conclusion
that is generally accepted as true though it might not be.
An example is the ‘scores’ or point predictions produced by the US Congressional Budget
Office (CBO) to assess the budgetary impact of legislation 10 years out. Eagerly awaited by
both Democratic and Republican members of Congress, CBO scores are often used to
stake out legislative positions even though they are based on strong and unsubstantiated
Scoring the recent US healthcare reform legislation, CBO director Douglas Elmendorf
reported to Congress that its enactment ‘would produce a net reduction of changes in
federal deficits of $138 billion over the 2010-19 period as a result of changes in direct
spending and revenue’.
Professor Manski points out that anyone seriously contemplating the many changes to
federal law embodied in this legislation should recognise that the prediction of $138 billion
deficit reduction is nothing more than a very rough estimate. Yet Elmendorf’s 25-page letter
expressed no uncertainty about the figure, subsequently reported by the media without
questioning the reliability of the number.
In an instance of ‘duelling certitudes’, Douglas Holtz-Eakin, a former CBO director, made a
very different prediction. In an op-ed in the New York Times, he predicted that the
legislation would increase deficits by $562 billion rather than decrease them by $138 billion.
Thus, the former and present directors of the CBO made predictions that differed by $700
billion, yet each presented his prediction as certain.
While policy analysts in the United States seem to abhor uncertainty, the UK government
has started to recognise the sensitivity of forecasts to varying assumptions and adjust its
practices accordingly.
UK agencies are now required to state upper and lower bounds on their estimates of the
economic impact of proposed legislation when submitting an impact assessment to
Parliament. In addition, agencies are instructed to highlight key assumptions and assess
how changes in assumptions might affect policy delivery. This transparency opens the door
to more credible policy analysis, Professor Manski concludes.
The economist also takes the media to task for stretching the conclusions of scientific
studies – ‘overreaching’. In other cases, journalists seeking the latest scoop sometimes
report research prematurely – especially before it has been peer-reviewed and published.
He suggests that since journalists cannot be experts in every field that they cover, they
should do more to seek out reputable researchers who are not closely associated with the
authors of studies.
In the end, Professor Manski suggests that policy-makers, the media and the public should
be deeply sceptical of those who appear unwavering in their policy predictions. He
recommends recognising ambiguity rather than ignoring it.
He also proposes a possible solution for policy analysis: using a ‘layered’ analysis,
researchers could move from weak, highly credible assumptions to stronger, less credible
ones, determining the conclusions that follow in each case. This would help resolve the
tension between the credibility and power of assumptions and also improve the
transparency of policy discussions.
Notes for editors: ‘Policy Analysis with Incredible Certitude’ by Charles Manski is
published in the August 2011 issue of the Economic Journal.
Charles Manski is at the Department of Economics and Institute for Policy Research,
Northwestern University.
For further information: contact Charles Manski on +1-847-491-8223 (email:; or Romesh Vaitilingam on +44-7768-661095 (email: