Australasian Mathematical Psychology Conference 2019

Using crowd meta-knowledge to identify expertise in the single-question domain

Marcellin Martinie
Melbourne School of Psychological Sciences, The University of Melbourne
Piers Howe
Melbourne School of Psychological Sciences, The University of Melbourne
Tom Wilkening
Department of Economics, The University of Melbourne

When trying to predict whether a future event will occur, one will often solicit predictions from multiple individuals. Contemporary forecast aggregation algorithms typically use external information such as individuals’ past performance in order to select and weight individuals. In practice, such information may not be available and can be cost-prohibitive to obtain. The SP algorithm (Prelec, Seung, & McCoy, 2017, Nature, 541, 532) has been proposed as a solution to this problem. The SP algorithm generates predictions using forecasters’ actual predictions as well as forecasters’ meta-predictions about the percentage of other people endorsing each response. We present the results of an experiment investigating the SP mechanism using a novel dataset of US grade school questions, with questions varying systemically in difficulty according to their grade. We find that the algorithm’s performance relative to other forecast-aggregation algorithms is largely dependent on question difficulty. We show that the SP algorithm mechanism exploits the differences between each person’s predictions and their meta-predictions in order to identify and leverage expertise in the crowd. The SP algorithm outperforms other algorithms for questions of moderate difficulty, where differences in individual expertise exist and can be leveraged effectively. However, when questions are very difficult, little or no expertise is present in the crowd, and the SP algorithm does not outperform other algorithms. Our findings provide guidance for those seeking to maximize forecast accuracy under conditions where the individuals’ past performance is unknown.