Media Briefings

‘TRUTH SERUMS’: New research on how to get people to respond truthfully to economic surveys

  • Published Date: December 2015

‘TRUTH SERUMS’: New research on how to get people to respond truthfully to economic surveys

When collecting survey data on consumers’ beliefs about the economy, rewarding them based on the accuracy of their reported expectations improves the quality of the answers they give. That is the central finding of an experimental study by Stefan Trautmann and Gijs van de Kuilen, published in the December 2015 issue of the Economic Journal.

Consumers’ expectations about economic variables such as inflation, demand and future income play an important role in the economy, and are therefore often measured in surveys. But if consumers are not financially motivated to respond truthfully, their reported expectations might become meaningless. The researchers explore the effectiveness of different methods of administering what they call ‘truth serums’ – financial incentives to encourage accurate responses.

Expectations about uncertain events play an important role not only in consumer choice: they are measured in many other fields in social science, including accounting, education, finance, medicine, meteorology and politics. In these studies, respondents are typically asked to report their own subjective likelihood assessments about the uncertain event of interest, which in turn, can be used to inform policy.

A sceptical commentator might argue that basing policy on data on these expectations is questionable because respondents lack any incentives to reveal their expectations truthfully. They may be tempted to misrepresent their expectations because of social desirability or to influence what they think is the goal of the survey.

To overcome this potential problem, several clever methods have been developed that extract expectations from respondents’ revealed preferences between uncertain lotteries. By linking monetary payoffs with the accuracy of the reported expectations, these methods work very much like a ‘truth serum’: it is in the best interest of respondents to reveal their expectations truthfully.

Potential drawbacks of these truth serums are that they are highly complex for respondents, and that they often make assumptions about respondents’ preferences and attitudes that may not hold true in practice.

To weigh the costs (in terms of complexity, survey length, etc.) and benefits (in terms of high quality expectations data) of different elicitation methods, Trautmann and van de Kuilen compare the performance of several methods in a controlled experiment, using a sample of university students.

The results show that respondents are more inclined to base their decisions on reported expectations when they are rewarded for the accuracy of these expectations, compared with a situation where they report their expectations without any incentives for truthful reporting. But the researchers find no differences between the different truth serum methods, which suggests that the least complex of these methods score best in terms of cost-benefit efficiency.

On balance, there are some benefits of using complex methods, but these have to be weighed against the costs of implementing a method in a specific research setting (type of respondent, online versus in person interviews, etc.).

ENDS


Notes for editors: ‘Belief Elicitation: A Horse Race Among Truth Serums’ by Stefan Trautmann and Gijs van de Kuilen is published in the December 2015 issue of the Economic Journal.

Stefan Trautmann is at the Alfred-Weber-Institute for Economics, University of Heidelberg. Gijs van de Kuilen is at the Department of Economics & CentER, Tilburg University.

For further information: contact Stefan Trautmann on +49 6221 54 2952 (email: trautmann@uni-hd.de); Gijs van de Kuilen on +31 13 466 2039; email: g.v.d.kuilen@uvt.nl); or Romesh Vaitilingam on +44-7768-661095 (email: romesh@vaitilingam.com; Twitter: @econromesh).