Prediction Markets vs. Expert Surveys: Comparing Forecast Accuracy for Scientific Replication Outcomes

Abstract

Fewer than half of published findings in psychology and social science have been shown to replicate, yet researchers, journals, and funders must constantly decide which results to trust and build upon. Knowing which forecasting method best predicts replication outcomes could help the scientific community identify unreliable findings before resources are wasted on follow-up work. We ask whether prediction markets or expert surveys produce more accurate forecasts of scientific replication outcomes. We use the dataset of expert surveys, which collect predictions directly from researchers, and prediction markets, where participants trade based on how likely they believe a finding is to replicate. We compare the forecasting accuracy of prediction markets and expert surveys using 103 findings from four scientific replication projects. We average expert survey responses to create a survey forecast for each finding and the final transaction price was used as the prediction market forecast. We measure accuracy by comparing predicted probabilities and actual findings. Our results show that across 206 forecasts, prediction markets have a lower average absolute error of 0.384 compared with 0.423 for expert surveys. After accounting for differences between the four projects, prediction markets had an estimated 0.039 lower error, although this difference was not statistically significant (p = 0.135). A second analysis using Brier scores also found no statistically significant difference (p = 0.591). Our findings suggest that prediction markets are more accurate than expert surveys, but the results are not strong enough to conclude that prediction markets consistently outperform expert surveys in forecasting scientific replication outcomes.