When we think of social science studies that use surveys to look at features of the general population, we tend to assume that they go out and find a randomized sample of people, either by mail or phone or in person, and that each such study finds its own sample, so that any person in the general population has a chance of being surveyed. But there was something about the research methods used in the study of how Christians reconcile their political beliefs with the Bible that I posted about recently that surprised me. The data for this research was based on a survey of 1,256 participants who were obtained from something called the ‘SurveyMonkey Audience‘.

Some of you may know about SurveyMonkey, an online survey tool that is easy to use and free and hence quite popular. It appears that this site also has created a database of many thousands of individuals who are willing to fill out surveys on practically anything in exchange for a small donation ($0.50) to a charity of their choice as well as a chance to win $100. This group thus provides researchers with a pre-existing pool of willing participants. Researchers can purchase the required number of randomized respondents from this panel based on the sample size they need, paying $3.00 for each completed survey. This is much cheaper for researchers than finding their own random sample of respondents, which can be a major headache.

One of the problems of research in the social sciences is obtaining a large enough random sample in order to minimize the risks of spurious results due to small numbers. This can be very expensive, causing researchers to take short cuts. The benefits of having access to a cheap source of a large number of respondents may be enough that they outweigh that of depending on a pre-selected and self-selected population to sample from.

Of course, the immediate question then is what kinds of systematic biases exist in such a population? The paper’s authors acknowledge that this self-selection aspect is problematic, saying that “the use of this relatively inexpensive service in research is still quite new, and the particular biases it introduces have yet to be systematically studied. However, there is no obvious reason why our sample would be particularly prone to show the types of projection and dissonance-reducing expressions of religious priorities documented in the present study.”

I had not been aware of the existence of people who had agreed to serve on a panel to be surveyed repeatedly by researchers on various topics, and do not know how widespread the use of such panels is. The study of any biases would be interesting. One obvious thing to look at is that serving on such a panel may appeal only to those people who enjoy doing surveys and/or think that their time spent on them is worth it to help their favorite charity, and how this particular view correlates with other views.

Of course, this lack of true randomness is not unprecedented. A lot of psychological and sociological research is done on populations of students who are enrolled in psychology courses in the larger universities because they are the handiest group for the professors of psychology who usually conduct such studies to work with. There has always been some concern as to the generalizability of the results since it is clear that this is not a random sample of the population of a whole. Over time we have come to not question too closely the validity of the results thus obtained and maybe that will become true of panels of survey respondents too.