Media Briefings

The Questionable Reliability Of International Statistics: Evidence From The Human Development Index

  • Published Date: June 2011

Small revisions to the data that go into the widely cited Human Development Index (HDI) can substantially alter a country’s position in the global rankings produced by United Nations Development Programme (UNDP). That is one of the findings of research by Professor Hendrik Wolff and colleagues, published in the June 2011 issue of the Economic Journal.

Their study, the first comprehensively to measure data error in international statistics, reveals how such errors can lead to inappropriate policy conclusions, particularly as a result of the UNDP’s classification of each country into one of three ‘development bins’.

The researchers urge the UNDP to discontinue this practice because the cut-off values seem arbitrary, the classifications can provide incentives for strategic behaviour and they have potential to misguide politicians, investors, charity donators and the public.

The HDI, published annually by the UNDP, has become the most widely used measure for communicating a country’s development status. Compared with the traditional measure of GDP, the HDI is a broader measure of development, since it captures not only the level of income but also incorporates measures of health (life expectancy) and education (school enrolment and literacy rate).

Importantly, the UNDP ranks all countries into a global league table and then classifies each country into one of three development bins:

  • low human development for HDI scores between 0.0 and 0.5,
  • Medium human development for HDI scores between 0.5 and 0.8
  • High human development for HDI scores between 0.8 and 1.0.

Each year when the new HDI statistics are published, much public attention focuses on the ranking of a country relative to its ranking in previous years and to those of competing countries.

For example, in 2001, when Canada lost the top position, The National Post wrote: ‘We’re not No. 1! Canada drops in UN rankings… Prime Minister Jean Chretien often refers to the report in public statements and speeches’. Or in 1998, when Pakistan (ranked 138 and 119 the previous year) bypassed India (ranked 139 and 118 the previous year), The Tribune noted: ‘Pak beat India, both lose!

Hendrik Wolff first gathered data for the HDI when he was working in Ghana in 1997. In later years when he wanted to compare the rankings, he noticed that the original ranks jumped around substantially, which led to him and his colleagues to scrutinise the statistics used in the HDI.

Their study addresses these questions by exploiting (1) the originally published HDI time series; (2) the variables used to construct the HDI; (3) changes to the HDI formula; and (4) documented data revisions. It identifies three sources of data error: measurement error due to data revisions; data error due to formula updating; and misclassification due to inconsistent cut-off values.

The research finds that:

  • The HDI contains data error standard deviations ranging from 0.03 (the United States) to 0.11 (Niger), which is significant given the 0 to 1 scale of the HDI.
  • The magnitude of the error variances is greater the lower the HDI rank. This is consistent with the quality of the statistical agencies improving with higher development.
  • 11%, 21% and 34% of all countries can be interpreted as misclassified in the three development bins due to the three sources of data error, respectively.
  • The expected deviation in rank in the world HDI league table is nine rank positions. These calculations show that statements such as ‘Pak beat India’ are to be interpreted with great care.
  • By replicating prior academic studies, the research shows that key parameters vary by up to 100% due to data error.

The authors suggest that the United Nations should discontinue the practice of classifying countries into the three development bins. Politically sensitive uses of the HDI might potentially provide perverse incentives for a country to manipulate the sub-indicator variables, if it has realised the comparative advantage of a 0.49 HDI score versus a 0.51 score.

Some of the criticism raised by this research were noted in The Economist in January 2011: http://www.economist.com/node/17849159?story_id=17849159&fsrc=rss). The UNDP responded that it undertook a systematic revision of the methods used for the calculation of the HDI and that the new methodology and classifications of countries are now robust: https://www.economist.com/user/4291658/comments

ENDS

Notes for editors: ‘Classification, Detection and Consequences of Data Error: Evidence from the Human Development Index’ by Hendrik Wolff, Howard Chong and Maximilian Auffhammer is published in the June 2011 issue of the Economic Journal.

Hendrik Wolff is at the University of Washington. Howard Chong is at Cornell University. Maximilian Auffhammer is at the University of California at Berkeley.

For further information: contact Hendrik Wolff on +1-510-220-7961 (email: hgwolff@u.washington.edu); Howard Chong on +1-510-333-539 (email: hgchong@berkeley.edu); Maximilian Auffhammer on +1-510-463-1546 (email: auffhammer@berkeley.edu); or Romesh Vaitilingam on +44-7768-661095 (email: romesh@vaitilingam.com).