When oracles fail-A comparison of four procedures for aggregating subjective probability forecasts
When a panel of experts is assembled to make predictions about some aspect of the future, they invariably disagree. The possible strategies for dealing with such disagreement include (1) taking the statistical average of the individual forecasts, (2) face-to-face discussion until consensus is achieved, (3) the Delphi procedure, and (4) the Estimate-Talk-Estimate procedure proposed by D. H. Gustafson, R. K. Shukla, A. Delbecq, and G. W. Walster (Organizational Behavior and Performance, 1973). This paper does two things. First, it very briefly reviews the literature relevant to opinion aggregation when forecasts are expressed as subjective probability distributions. Second, it describes an experimental comparison of the four procedures listed above using a subjective probability forecasting task. Together, the review and the experiment lead to two conclusions. First, subjective probability forecasts can be substantially improved by aggregating the opinions of a group of experts rather than relying on a single expert. Second, from a practical standpoint, there is no evidence to suggest that the method used to aggregate opinions will have a substantial impact on the quality of the resulting forecast. © 1981.
