Australasian Mathematical Psychology Conference 2019

Unrepresentative samples and the quest for generality: Ideas from survey statistics

Lauren Kennedy
School of Social Work, Columbia University
Andrew Gelman
Columbia University

Psychology has long been reliant on a variety of convenience samples. Although the source has changed with tools like Amazon Mechanical Turk, there is still no guarantee that AMT samples are representative of the general population on all relevant demographics. Convenience samples can be a threat to the generality of a study if the effect size differs between demographic subsets and the sample differs in proportion to those subsets. Tools from survey statistics, a field which specializes in generalization from non-representative samples to a wider population, have not previously been applicable due to the reliance on probability samples. In this talk we consider multilevel regression and poststratification (MRP), a technique that has proved useful for non-probability samples. We discuss the potential costs and benefits of incorporating this technique with psychological datasets.